BUILDING HIGHLY SPECIALISED WEB CRAWLER USING B-TREE SEARCH AND HTML PARSER

Abstract

Web crawlers are Internet bot that automatically traverse the hyper-link structure of the world wide web in order to locate and retrieve information. This paper describes a web crawling approach based on B-tree search and HTML Parser. As the goal of crawler is to selectively seek out pages that are relevant to given keywords. Rather than collecting and indexing all available web documents to be able to answer all possible queries, a crawler analyze its crawl boundary to hit upon the links that are likely to be most relevant for the crawl, and avoids irrelevant links of the document.