search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection
Swiftype Documentation / tutorials: E-Commerce Engine Maintenance

E-commerce Engine Maintenance

The following provides strategies for keeping your search engine index up to date with your product inventory.

Ensuring your index is complete

Sitemaps

Creating and maintaining a current sitemap will help ensure your engine index is complete. Our crawler supports the Sitemap XML format. If the sitemap cannot be found under the /sitemap.xml path of your site, please direct your sitemap to our crawler using your robots.txt file.

Ensuring new products get crawled

Two popular methods for adding products are utilizing the crawler or installing the Constant Crawl tool.

Partial and Full Recrawls

Our crawler performs several automatic partial and full recrawls of your site. Daily partial recrawls focus on your homepage and new URLs. Full recrawls run weekly, and will visit every discoverable page.

In addition to the automatic crawls, you have the option to trigger a full recrawl through your Swiftype dashboard. This can be found under the Manage > Domains tab. The frequency at which you can request a recrawl is limited by your plan.

Constant Crawl

If you are finding that the automatic and manual crawls do not provide you with enough opportunities for adding content, you may want to look into our Constant Crawl tool. The Constant Crawl detects and indexes new or changed pages on your site in near real time.

Use our API to crawl a single URL

If you have a dedicated developer working on your Swiftype implementation, you might want to use our Crawler API endpoint. You can trigger the crawl of a single URL by using the domain's crawl_url endpoint with the URL as the parameter.

It would be most valuable if you can incorporate it into your content management system to be triggered when a product is edited, added or deleted.

Removing 'out of stock' products

Removing products from your search results varies based on your index and filtering rules. When products become 'out of stock', you can either mark them as such and filter them out of your search results or you can remove them from the search index until they become 'in stock'.

Removing the product's page from your site

If the product pages are removed entirely from the site and return a 404 error, when the crawler attempts to recrawl that page, it will be removed from your search index.

Robots Meta Tags

One commonly used method to remove documents from your engine is to use robots meta tags. You can use the robots noindex meta tag to indicate to our crawler not to index that specific document. These tags will be recognized when the crawler next visits the specific page.

Using Swiftype meta tags to create a filtered search

Some of our e-commerce users keep track of product availability using a swiftype specific meta tag. This meta tag could either keep track of a string value, 'in stock' or 'out of stock' or an integer number representing the quantity of available products. Either method you choose, you will then be able to create your filtered search.