What's this?

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.

Other terms for Web crawlers are ants, automatic indexers, bots, or Web spiders, Web robots, or especially in the FOAF community - Web scutters. This process is called Web crawling or spidering.

A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.

The bot will not pick up the changes in robots.txt immediately, usually it may take up to one week. If you want the changes to be active immediately - please contact our kliu@seplinkbot.com.

How to turn it off?

The quick way to prevent Seplinkbot from visiting your site is to put these two lines into the /robots.txt file on your server: