Home > SEO Services > Search Engine Optimisation Glossary > Search Engine Spider
A search engine that is crawler based such as Google, Yahoo, Alta Vista etc uses web spiders (robots - bots) to crawl the internet from website to website.
There job is to mainly copy or take a snap shots of each web page then bring the copies back to the main search engine for processing.
Example:
Google has two main spiders/crawlers or search robots. These
are called:
Freshbot - Regularly crawls the World Wide Web looking for new websites and updated page content.
Deepbot - Exactly how it sounds Deepbot does a deep crawl of web content
once a month.
