In order to avoid unwanted content within the search indexes, website owners can instruct spiders not to crawl particular information or directories throughout the normal robots.txt file in the basis Listing of the domain. Also, a page might be explicitly excluded from the search engine's databases through the use of a meta tag specific to robots (