Building Domain-Specific Search Engines with Machine Learning Techniques

March 1999

Building Domain-Specific Search Engines with Machine Learning Techniques

Authors:

Andrew McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore

Abstract:

Domain-specific search engines are growing in popularity because they offer increased accuracy and extra functionality not possible with the general, Web-wide search engines. For example, www.campsearch.com allows complex queries by age-group, size, location and cost over summer camps. Unfortunately these domain-specific search engines are difficult and time-consuming to maintain. This paper proposes the use of machine learning techniques to greatly automate the creation and maintenance of domain-specific search engines. We describe new research in reinforcement learning, information extraction and text classification that enables efficient spidering, identifying informative text segments, and populating topic hierarchies. Using these techniques, we have built a demonstration system: a search engine for computer science research papers. It already contains over 50,000 papers and is publicly available at www.cora.justresearch.com.
@conference{McCallum-1999-16662,
author = {Andrew McCallum And Kamal Nigam And Jason Rennie And Kristie Seymore},
title = {Building Domain-Specific Search Engines with Machine Learning Techniques},
booktitle = {Proceedings of AAAI '99 Spring Symposium on Intelligent Agents in Cyberspace},
year = {1999},
month = {March},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.