Crawling. Search engines like yahoo count on a staff of web crawlers, also referred to as spiderbots or simply bots, to discover up to date articles on the internet. Website crawlers help explore and collect information required to index a webpage. One thing that I actually like about this is the