Noun | 1. | spider - predatory arachnid that usually has silk-spinning organs at the back end of the body; they spin silk to make cocoons for eggs or traps for prey |
2. | spider - a computer program that prowls the internet looking for publicly accessible resources that can be added to a database; the database can then be searched with a search engine Synonyms: wanderer | |
3. | spider - a skillet made of cast iron |
(World-Wide Web) | spider - (Or "robot", "crawler") A program that
automatically explores the World-Wide Web by retrieving a
document and recursively retrieving some or all the documents
that are referenced in it. This is in contrast with a normal
web browser operated by a human that doesn't automatically
follow links other than inline images and URL redirection. The algorithm used to pick which references to follow strongly depends on the program's purpose. Index-building spiders usually retrieve a significant proportion of the references. The other extreme is spiders that try to validate the references in a set of documents; these usually do not retrieve any of the links apart from redirections. The standard for robot exclusion is designed to avoid some problems with spiders. Early examples were Lycos and WebCrawler. http://info.webcrawler.com/mak/projects/robots/robots.html. |