search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection
Search Concepts / Web Crawler

Web Crawler

A web crawler is a piece of software that visits a website and indexes all the content on a webpage, then follows all the links on that page and does the same to the each page it can find. Web crawlers will also follow sitemaps to discover and index all the content on a website. Web crawlers are primarily famous because of Google’s web crawler Googlebot, which is perpetually crawling the web to index new content for Google’s global search index.

In Swiftype

While Googlebot crawls the entire web to create a global search index (a process which can take weeks to fully complete), Swiftype has its own web crawler (Swiftbot) that is specifically designed to index and monitor changes to individual websites. This provides two key advantages: (i) our web crawler can quickly create an index for a new website by simply crawling a web site url or sitemap, and (ii) Swiftbot will immediately index new changes to your website, ensuring search results always reflect the live offerings of your website. Site owners can also place custom Swiftype meta tags in the <head> element of their website to add specific information to their search engine index that Swiftbot does not index by default.