E-commerce Engine Maintenance
The following provides strategies for keeping your search engine index up to date with your product inventory.
Ensuring your index is complete
Creating and maintaining a current sitemap will help ensure your engine index is complete. Our crawler supports the Sitemap XML format. If the sitemap cannot be found under the
/sitemap.xml path of your site, please direct your sitemap to our crawler using your robots.txt file.
Ensuring new products get crawled
Two popular methods for adding products are utilizing the crawler or installing the Constant Crawl tool.
Partial and Full Recrawls
Our crawler performs several automatic partial and full recrawls of your site. Daily partial recrawls focus on your homepage and new URLs. Full recrawls run weekly, and will visit every discoverable page.
In addition to the automatic crawls, you have the option to trigger a full recrawl through your Swiftype dashboard. This can be found under the Manage > Domains tab. The frequency at which you can request a recrawl is limited by your plan.
If you are finding that the automatic and manual crawls do not provide you with enough opportunities for adding content, you may want to look into our Constant Crawl tool. The Constant Crawl detects and indexes new or changed pages on your site in near real time.
Use our API to crawl a single URL
If you have a dedicated developer working on your Swiftype implementation, you might want to use our Crawler API endpoint. You can trigger the crawl of a single URL by using the domain's crawl_url endpoint with the URL as the parameter.
It would be most valuable if you can incorporate it into your content management system to be triggered when a product is edited, added or deleted.
Removing 'out of stock' products
Removing products from your search results varies based on your index and filtering rules. When products become 'out of stock', you can either mark them as such and filter them out of your search results or you can remove them from the search index until they become 'in stock'.
Removing the product's page from your site
If the product pages are removed entirely from the site and return a 404 error, when the crawler attempts to recrawl that page, it will be removed from your search index.
Robots Meta Tags
One commonly used method to remove documents from your engine is to use robots meta tags. You can use the robots
noindex meta tag to indicate to our crawler not to index that specific document. These tags will be recognized when the crawler next visits the specific page.
Using Swiftype meta tags to create a filtered search
Some of our e-commerce users keep track of product availability using a swiftype specific meta tag. This meta tag could either keep track of a string value, 'in stock' or 'out of stock' or an integer number representing the quantity of available products. Either method you choose, you will then be able to create your filtered search.