What is Web Crawler?
A web crawler is a piece of software that visits a website and indexes all the content on a webpage. Once it lands on a page, it follows all the links on that page and then does the same to each page it can find. Web crawlers will also follow sitemaps to discover and index all the content on a website. Web crawlers are primarily famous because of Google’s web crawler, Googlebot, which is perpetually crawling the web to index new content for Google’s global search index.
In Swiftype
While Googlebot crawls the entire web to create a global search index (a process which can take weeks to fully complete), Swiftype Site Search has its own web crawler (Swiftbot) that is specifically designed to index and monitor changes to individual websites. This provides two key advantages: (i) our web crawler can quickly create an index for a new website by simply crawling a web site url or sitemap, and (ii) Swiftbot will immediately index new changes to your website, ensuring search results always reflect the live offerings of your website. Site owners can also place custom Swiftype meta tags in the <head> element of their website to add specific information to their Engine index that Swiftbot does not index by default.
App Search is an API-based solution that does not leverage a crawler. Instead, users populate their Engine with documents through a robust set of API endpoints.
Sign up with email