What is Swiftbot?
What websites does Swiftbot crawl?
Most web crawlers automatically crawl all websites, because they are trying to build a search index of the entire web. Swiftbot, by comparison, only crawls sites that our customers have asked us to crawl.
Can I change how Swiftbot crawls my website?
You can control the behavior of Swiftbot by using a
robots.txt file on your web site.
More information about this file and how to use it can be found at The Web
Robots Pages, Wikipedia’s entry on the Robots
Exclusion Standard, and Google’s
Webmaster Tools article. Swiftbot will obey all restrictions it finds in a
and will even refuse to crawl your entire site, if you tell it to.
Swiftbot looks for the
Swiftbot when examining a
robots.txt file. (It will also, of course, obey restrictions set for
I have more questions about Swiftbot.
If you have further questions about Swiftbot, you can email us at email@example.com. We’ll answer your email as soon as we can.