About Swiftbot

What is Swiftbot?

Swiftbot is Swiftype’s web crawler — a piece of software that retrieves information from (or “crawls”) web sites in order to make them searchable. It’s similar to Googlebot or msnbot/bingbot.

What websites does Swiftbot crawl?

Most web crawlers automatically crawl all websites, because they are trying to build a search index of the entire web. Swiftbot, by comparison, only crawls sites that our customers have asked us to crawl.

Can I change how Swiftbot crawls my website?

You can control the behavior of Swiftbot by using a robots.txt file on your web site. More information about this file and how to use it can be found at The Web Robots Pages, Wikipedia’s entry on the Robots Exclusion Standard, and Google’s Webmaster Tools article. Swiftbot will obey all restrictions it finds in a robots.txt file, and will even refuse to crawl your entire site, if you tell it to.

Swiftbot looks for the User-agent string Swiftbot when examining a robots.txt file. (It will also, of course, obey restrictions set for User-agent: *.)

I have more questions about Swiftbot.

If you have further questions about Swiftbot, you can email us at support@swiftype.com. We’ll answer your email as soon as we can.