Crawler Quick Start
The Swiftype crawler is the easiest way to add a search engine to your site. It is flexible, intuitive and powerful. This guide will get you started in just a couple of minutes.
1. Create an account
Before getting started, you'll need to create a Swiftype account if you haven't already. Be sure to confirm your account, otherwise we will only crawl part of your website.
2. Create a search engine
Once you have an account, create a new Engine. All your crawled content will be stored in the Engine, making it available for searching.
When you create an engine, you have the option of a crawler-based engine, using our WordPress plugin, or using the Swiftype API. Choose the Crawler-based engine and enter your site's URL into the validation field. Once it passes verification checks, you'll be able to enter a name for your search engine (to help you remember it later).
After you create the engine, Swiftype's web crawler Swiftbot will crawl the URLs you provided and index your pages for searching.
3. Install your search box
Click the Install tab to get the install code for your search engine. You need to copy this into your site's template so it appears on every page that you want the search box to appear.
4. Customizing your crawl
The Swiftype crawler produces good results on most web sites, but it also allows you to customize how data is indexed on your site.
Swiftype meta tags give you an easy way to override how title or section data is extracted from a page, as well as add powerful new features like associating an image with a page, or filtering pages by type.
For example, if a page has a title like "MySite.com -- Books -- The Master and Margarita by Mikhail Bulgakov" and you would like the title in search results to be "The Master and Margarita", you can use Swiftype meta tags to custom define the value of the
<html> <head> <title>MySite.com -- Books -- The Master and Margarita by Mikhail Bulgakov</title> <meta class="swiftype" name="title" data-type="string" content="The Master and Margarita" /> </head> ... </html>
Meta tags can be combined and extracted as an array. Building on the previous example, you can also associate an image with a page. The image will be displayed in the search results.
<html> <head> <title>MySite.com -- Books -- The Master and Margarita by Mikhail Bulgakov</title> <meta class="swiftype" name="title" data-type="string" content="The Master and Margarita" /> <meta class="swiftype" name="image" data-type="enum" content="http://mysite.com/images/the_master_and_margarita.jpg" /> </head> ... </html>
Read more: Meta Tags.
Excluding content by URL
Swiftype allows you to limit the crawl to specific areas of your website using path inclusions and exclusions.
Whitelists only index URLs matching a pattern.
Blacklists index URLs that do not match a pattern.
Whitelists and blacklists can be combined for precise control over what will be indexed on your site. Consider this example:
This will make Swiftype only index URLs starting with
/documentation and not ending with
Read more: Path Whitelist and Blacklist Rules
Excluding pages with robots.txt
Swiftype supports the Robots.txt standard and you can use that file to exclude pages you don't want indexed. You might want other search engines to index those pages, however, so it's best to specify the Swiftbot User-agent in your robots.txt.
# Don't let Swiftbot index the pages under /archives/ User-agent: Swiftbot Disallow: /archives/ # Allow other agents to index the entire site User-agent: * Disallow:
Read more: Robots.txt support.
5. Advanced implementations
The Swiftype Result Designer works well for customizing the basic styling of your search results and autocomplete menu, but more advanced implementations are also possible by using our our jQuery search and jQuery autocomplete plugins.
Stuck? Looking for help? Contact Support!