View More View Less
  • 1 Corvinus University of Budapest Budapest Hungary
  • 2 Economic Research Institute Budapest Hungary
Restricted access

In economic and social sciences it is crucial to test theoretical models against large and reliable databases. The general research challenge is to build up a well-structured database that suits the given research question well and that is cost efficient at the same time. In this paper we focus on crawler programs that proved to be an effective tool for data base building in very different problem settings. We present three structurally different research models where crawler programs can be applied successfully: exploration, classification and time series analysis. In the case of exploration we present findings about the Hungarian web agency industry where no previous statistical data was available about their operations. For classification we show how the top visited Hungarian web domains can be divided into predefined categories of e-business models. In the third research we used a crawler to gather the values of specific pre-defined records containing ticket prices of low cost airlines from one single site. Based on the experiences we highlight some conceptual conclusions and opportunities for crawler based research in e-business.

  • Bolisani, E. — Biagi Di, M. — Scarso, E. (2003): Knowledge Intermediation: New Business Models in the Digital Economy. Presented at the 16 th International Bled e-Conference, Bled, Slovenia, 9–11 June.

  • Brin, S. — Page, L. (1998): The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems 30(1–7): 107–117.

    Page L. , 'The anatomy of a large-scale hypertextual Web search engine ' (1998 ) 30 Computer Networks and ISDN Systems : 107 -117.

    • Search Google Scholar
  • Chakrabarti, S. (2003): Mining the Web: Analysis of Hypertext and Semi Structured Data. New York: Morgan Kaufmann.

    Chakrabarti S. , '', in Mining the Web: Analysis of Hypertext and Semi Structured Data , (2003 ) -.

    • Search Google Scholar
  • Ganz, J. — Reinsel, D. (2009): As the Economy Contracts, the Digital Universe Expands. IDC-Multimedia White Paper.

  • GKI (2009): Report on the internet economy — Online stores. GKI, http://www.gkienet.hu/content/view/104/38/lang.en/ , accessed 27 May 2011

  • Hine, C. (2005): Virtual Methods: Issues in Social Research on the Internet. Berg: London.

    Hine C. , '', in Virtual Methods: Issues in Social Research on the Internet , (2005 ) -.

  • Koster, M. (1993): Guidelines for robot writers, http://www.robotstxt.org/wc/guidelines.html , accessed 23 February 2005

  • Menczer, F. — Pant, G. — Srinivasan, P. (2004): Topical Web Crawlers: Evaluating Adaptive Algorithms. ACM Transactions on Internet Technology 4(4): 378–419.

    Srinivasan P. , 'Topical Web Crawlers: Evaluating Adaptive Algorithms ' (2004 ) 4 ACM Transactions on Internet Technology : 378 -419.

    • Search Google Scholar
  • O’Boyle, P. — Acton, T. — Campion, M. — Conboy, K. — Scott, M. (2009): Towards a Toolset for Intranet Evaluation. Presented at the 22 nd International Bled e-Conference, Bled, Slovenia, 14–17 June.

  • Osterwalder, A. — Pigneur, Y. (2002): An e-Business Model Ontology for modelling e-Business. Presented at the 15 th Bled Electronic Commerce Conference, e-Reality: Constructing the e-Economy, Bled, Slovenia, 1–11 June.

  • Polanski, P. (2006): Intellectual Property Law versus Customs and Values of the Internet Community. Presented at the 19 th International Bled e-Conference, Bled, Slovenia, 5–7 June.

  • Rappa, M. (2002): Business Models on the Web, http://digitalenterprise.org/models/models.html , accessed 27 May 2011

  • Riemer, K. — Brüggemann, F. (2006): Personalisation of eSearch Services — Concepts, Techniques, and Market Overview. Presented at the 19 th PitInternational Bled e-Conference, Bled, Slovenia, 5–7 June.

  • Risvik, K. M. — Michelsen, R. (2002): Search Engines and Web Dynamics. Computer Networks, 39: 289–302.

    Michelsen R. , 'Search Engines and Web Dynamics ' (2002 ) 39 Computer Networks : 289 -302.

  • Shkapenyuk, V. — Suel, T. (2002): Design and implementation of a high performance distributed web crawler. In: Proceedings of the 18th International Conference on Data Engineering (ICDE), San Jose, CA: IEEE CS Press, pp. 357–368.

    Suel T. , '', in Proceedings of the 18th International Conference on Data Engineering (ICDE) , (2002 ) -.

    • Search Google Scholar
  • Stepanchuk, T. (2008): An Empirical Examination of the Relation between Bids and Positions of Ads in Sponsored Search. Presented at the 21 st International Bled eConference, Bled, Slovenia, 15–18 June.

  • Thelwall, M. — Stuart D. (2006): Web Crawling Ethics Revisited: Cost, Privacy and Denial of Service. Journal of the American Society for Information Science and Technology 57(13): 1771–1779.

    Stuart D. , 'Web Crawling Ethics Revisited: Cost, Privacy and Denial of Service ' (2006 ) 57 Journal of the American Society for Information Science and Technology : 1771 -1779.

    • Search Google Scholar