A web crawler is a piece of software that visits a website and indexes all the content on a webpage, then follows all the links on that page and does the same to the each page it can find. Web crawlers will also follow sitemaps to discover and index all the content on a website. Web crawlers are primarily famous because of Google’s web crawler Googlebot, which is perpetually crawling the web to index new content for Google’s global search index.
While Googlebot crawls the entire web to create a global search index (a process which can take weeks to fully complete), Swiftype has its own web crawler (Swiftbot) that is specifically designed to index and monitor changes to individual websites. This provides two key advantages: (i) our web crawler can quickly create an index for a new website by simply crawling a web site url or sitemap, and (ii) Swiftbot will immediately index new changes to your website, ensuring search results always reflect the live offerings of your website. Site owners can also place custom Swiftype meta tags in the <head> element of their website to add specific information to their search engine index that Swiftbot does not index by default.