A comparative analysis of search engine indexing time

Abstract

Internet usage is increasing dramatically daily, as is Web development which is enhanced by the emergence of new Web technologies. Social networks are continuously dominating our Web interactions through Facebook, Twitter, LinkedIn and MySpace, drawing a considerable number of Internet users. However, the dominance of the Web as the epicentre of both important and useless information has affected competition in industries like e-business. It takes some time for a website to be indexed and appear on the search engine result page. The indexing process involves the reading and recording of the weight-carrying words in a search format to an index file by a crawler. This results in the webpage being discovered by users on a search engine results page. A user can also submit a website manually for indexing. Crawlers should then visit this website, record all the words on the pages, and note links to other sites. The index file is updated regularly, either by human editors or by these crawlers. After an extensive literature survey, empirical evidence indicated that the indexing period for webpages is not fixed. In a recent two phase experiment, five websites were monitored to determine their indexing time for Google, Yahoo! and Bing. The experiment proved that there is a relationship between keyword density and indexing time. Yahoo! and Bing seem to favour sites with high keyword density when indexing. During the first experiment the shortest indexing waiting time was five days and the longest, 33 days. For the second one, the waiting time varied between 19 and 29 days. However, according to this study, a period of approximately 19 days is a reasonable average waiting time. It is possible that the Google sandbox effect plays a role in these experiments, and this was also investigated.