In economic and social sciences it is crucial to test theoretical models against large and reliable databases. The general research challenge is to build up a well-structured database that suits the given research question well and that is cost efficient at the same time. In this paper we focus on crawler programs that proved to be an effective tool for data base building in very different problem settings. We present three structurally different research models where crawler programs can be applied successfully: exploration, classification and time series analysis. In the case of exploration we present findings about the Hungarian web agency industry where no previous statistical data was available about their operations. For classification we show how the top visited Hungarian web domains can be divided into predefined categories of e-business models. In the third research we used a crawler to gather the values of specific pre-defined records containing ticket prices of low cost airlines from one single site. Based on the experiences we highlight some conceptual conclusions and opportunities for crawler based research in e-business.
Bolisani, E. — Biagi Di, M. — Scarso, E. (2003): Knowledge Intermediation: New Business Models in the Digital Economy. Presented at the 16th International Bled e-Conference, Bled, Slovenia, 9–11 June.
Brin, S. — Page, L. (1998): The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems 30(1–7): 107–117.
Page L. , 'The anatomy of a large-scale hypertextual Web search engine ' (1998 ) 30 Computer Networks and ISDN Systems : 107 -117 .
Chakrabarti, S. (2003): Mining the Web: Analysis of Hypertext and Semi Structured Data. New York: Morgan Kaufmann.
Chakrabarti S. , '', in Mining the Web: Analysis of Hypertext and Semi Structured Data , (2003 ) -.
Ganz, J. — Reinsel, D. (2009): As the Economy Contracts, the Digital Universe Expands. IDC-Multimedia White Paper.
GKI (2009): Report on the internet economy — Online stores. GKI, http://www.gkienet.hu/content/view/104/38/lang.en/ , accessed 27 May 2011
Hine, C. (2005): Virtual Methods: Issues in Social Research on the Internet. Berg: London.
Hine C. , '', in Virtual Methods: Issues in Social Research on the Internet , (2005 ) -.
Koster, M. (1993): Guidelines for robot writers, http://www.robotstxt.org/wc/guidelines.html , accessed 23 February 2005
Menczer, F. — Pant, G. — Srinivasan, P. (2004): Topical Web Crawlers: Evaluating Adaptive Algorithms. ACM Transactions on Internet Technology 4(4): 378–419.
Srinivasan P. , 'Topical Web Crawlers: Evaluating Adaptive Algorithms ' (2004 ) 4 ACM Transactions on Internet Technology : 378 -419 .
O’Boyle, P. — Acton, T. — Campion, M. — Conboy, K. — Scott, M. (2009): Towards a Toolset for Intranet Evaluation. Presented at the 22nd International Bled e-Conference, Bled, Slovenia, 14–17 June.
Osterwalder, A. — Pigneur, Y. (2002): An e-Business Model Ontology for modelling e-Business. Presented at the 15th Bled Electronic Commerce Conference, e-Reality: Constructing the e-Economy, Bled, Slovenia, 1–11 June.
Polanski, P. (2006): Intellectual Property Law versus Customs and Values of the Internet Community. Presented at the 19th International Bled e-Conference, Bled, Slovenia, 5–7 June.
Rappa, M. (2002): Business Models on the Web, http://digitalenterprise.org/models/models.html , accessed 27 May 2011
Riemer, K. — Brüggemann, F. (2006): Personalisation of eSearch Services — Concepts, Techniques, and Market Overview. Presented at the 19th PitInternational Bled e-Conference, Bled, Slovenia, 5–7 June.
Risvik, K. M. — Michelsen, R. (2002): Search Engines and Web Dynamics. Computer Networks, 39: 289–302.
Michelsen R. , 'Search Engines and Web Dynamics ' (2002 ) 39 Computer Networks : 289 -302 .
Shkapenyuk, V. — Suel, T. (2002): Design and implementation of a high performance distributed web crawler. In: Proceedings of the 18th International Conference on Data Engineering (ICDE), San Jose, CA: IEEE CS Press, pp. 357–368.
Suel T. , '', in Proceedings of the 18th International Conference on Data Engineering (ICDE) , (2002 ) -.
Stepanchuk, T. (2008): An Empirical Examination of the Relation between Bids and Positions of Ads in Sponsored Search. Presented at the 21st International Bled eConference, Bled, Slovenia, 15–18 June.
Thelwall, M. — Stuart D. (2006): Web Crawling Ethics Revisited: Cost, Privacy and Denial of Service. Journal of the American Society for Information Science and Technology 57(13): 1771–1779.
Stuart D. , 'Web Crawling Ethics Revisited: Cost, Privacy and Denial of Service ' (2006 ) 57 Journal of the American Society for Information Science and Technology : 1771 -1779 .