search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection
Swiftype Documentation / crawler: Crawler Quick Start

Crawler Quick Start

The Swiftype crawler is the easiest way to add a search engine to your site. However, it is also very flexible and powerful. This guide will get you started in just a couple of minutes.

1. Create an account

Before getting started, you'll need to create a Swiftype account if you haven't already. Be sure to confirm your account, otherwise we will only crawl part of your website.

2. Create a search engine

Once you have an account, create a new Engine. All your crawled content will be stored in the Engine, making it available for searching.

When you create an engine, you have the option of a crawler-based engine, using our WordPress plugin, or using the Swiftype API. Choose the Crawler-based engine and enter your site's URL into the validation field. Once it passes verification checks, you'll be able to enter a name for your search engine (to help you remember it later).

Creating an engine


After you create the engine, Swiftype's web crawler Swiftbot will crawl the URLs you provided and index your pages for searching.

After Swiftbot crawls your website and creates a search index, you will use the Swiftype Result Designer to customize the look and feel of your search experience and grab the javascript embed code to paste on your site. The following video walks you through the process.

Click the Install tab to get the install code for your search engine. You need to copy this into your site's template so it appears on every page that you want the search box to appear.

4. Customizing your crawl

The Swiftype crawler produces good results on most web sites, but it also allows you to customize how data is indexed on your site.

Meta Tags

Swiftype meta tags give you an easy way to override how title or section data is extracted from a page, as well as add powerful new features like associating an image with a page, or filtering pages by type.

For example, if a page has a title like "MySite.com -- Books -- The Master and Margarita by Mikhail Bulgakov" and you would like the title in search results to be "The Master and Margarita", you can use Swiftype meta tags to custom define the value of the title field.

Example - Using title
<html>
  <head>
    <title>MySite.com -- Books -- The Master and Margarita by Mikhail Bulgakov</title>
    <meta class="swiftype" name="title" data-type="string" content="The Master and Margarita" />
  </head>
  ...
</html>
  

Meta tags can be combined and extracted as an array. Building on the previous example, you can also associate an image with a page. The image will be displayed in the search results.

Example - Using image
<html>
  <head>
    <title>MySite.com -- Books -- The Master and Margarita by Mikhail Bulgakov</title>
    <meta class="swiftype" name="title" data-type="string" content="The Master and Margarita" />
    <meta class="swiftype" name="image" data-type="enum" content="http://mysite.com/images/the_master_and_margarita.jpg" />
  </head>
  ...
</html>
  

Read more: Meta Tags.

Excluding content by URL

Swiftype allows you to limit the crawl to specific areas of your website using path inclusions and exclusions.

Whitelists only index URLs matching a pattern.

Blacklists index URLs that do not match a pattern.

Whitelists and blacklists can be combined for precise control over what will be indexed on your site. Consider this example:

Path rule example

This will make Swiftype only index URLs starting with /documentation and not ending with /print. This could be used to only index a site's documentation section, but exclude all the printer-friendly pages.

Read more: Path Whitelist and Blacklist Rules

Excluding pages with robots.txt

Swiftype supports the Robots.txt standard and you can use that file to exclude pages you don't want indexed. You might want other search engines to index those pages, however, so it's best to specify the Swiftbot User-agent in your robots.txt.

Example - Robots.txt file restricting Swiftbot while allowing all other User-agents.
# Don't let Swiftbot index the pages under /archives/
User-agent: Swiftbot
Disallow: /archives/

# Allow other agents to index the entire site
User-agent: *
Disallow:
  

Read more: Robots.txt support.

5. Advanced implementations

The Swiftype Result Designer works well for customizing the basic styling of your search results and autocomplete menu, but more advanced implementations are also possible by using our our jQuery search and jQuery autocomplete plugins.

For additional information about these plugins, visit our jQuery tutorial or contact support@swiftype.com with any questions.