crawler

A crawleri is a type of program used to search the internet for new resources that are publicly accessible. Once it has accumulated this information it adds it to a database that can be searched by internet users via a search engine. Crawlers (also known as spiders) are used by many of the popular search sites such as Google or Lycos.