search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso
Search Concepts / Web Crawler

What is Web Crawler?

A web crawler is a piece of software that visits a website and indexes all the content on a webpage. Once it lands on a page, it follows all the links on that page and then does the same to each page it can find. Web crawlers will also follow sitemaps to discover and index all the content on a website. Web crawlers are primarily famous because of Google’s web crawler, Googlebot, which is perpetually crawling the web to index new content for Google’s global search index.

In Swiftype

While Googlebot crawls the entire web to create a global search index (a process which can take weeks to fully complete), Swiftype Site Search has its own web crawler (Swiftbot) that is specifically designed to index and monitor changes to individual websites. This provides two key advantages: (i) our web crawler can quickly create an index for a new website by simply crawling a web site url or sitemap, and (ii) Swiftbot will immediately index new changes to your website, ensuring search results always reflect the live offerings of your website. Site owners can also place custom Swiftype meta tags in the <head> element of their website to add specific information to their Engine index that Swiftbot does not index by default.

App Search is an API-based solution that does not leverage a crawler. Instead, users populate their Engine with documents through a robust set of API endpoints.

Swiftype is the easiest way to add great search to your website