search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso

Site Search Meta Tags

Your pages must be re-crawled before any code level changes will be received by Site Search!

See Crawler Troubleshooting if your documents seem out-of-sync with your live content.

The Site Search Crawler supports a flexible set of meta tags to control how you ingest your website content.

When the crawler visits your webpage, by default, we extract a standard set of fields (e.g. title, body) that are then indexed.

With these meta tags, you can augment - or completely alter - the set of fields our crawler extracts to better fit the data you wish to be indexed into your Engine.

Meta Tags

Each field must define specific name, type, and content values.

The field type, which is specified in the data-type attribute, must be a valid, Site Search supported field type.

The template for a Site Search meta tag is as follows:

  <meta class="swiftype" name="[field name]" data-type="[field type]" content="[field content]" />
Choose your field's data-type carefully! Once a new meta tag has been indexed, the custom fields are created. Once created, the data-type cannot be changed.

The next example is more complex, showing the creation of multiple fields.

As you can see, the tag field is repeated, and as a result our crawler extracts an array of tags for this URL. All field types can be extracted as arrays.

  <title>page title | website name</title>
  <meta class="swiftype" name="title" data-type="string" content="page title" />
  <meta class="swiftype" name="body" data-type="text" content="this is the body content" />
  <meta class="swiftype" name="url" data-type="enum" content="" />
  <meta class="swiftype" name="price" data-type="float" content="3.99" />
  <meta class="swiftype" name="quantity" data-type="integer" content="12" />
  <meta class="swiftype" name="published_at" data-type="date" content="2013-10-31" />
  <meta class="swiftype" name="store_location" data-type="location" content="20,-10" />
  <meta class="swiftype" name="tags" data-type="string" content="tag1" />
  <meta class="swiftype" name="tags" data-type="string" content="tag2" />

An important note is that the crawler will not capture default SEO meta tags, like these:

  <meta name="description" content="A descriptive descriptor.">
  <meta name="keywords" content="helpful, documentation">

To be indexed by the crawler, they would need to be adapted:

  <meta class="swiftype" name="description" data-type="string" content="A descriptive descriptor.">
  <meta class="swiftype" name="keywords" data-type="string" content="helpful, documentation">

Body-embedded Data Attribute Tags

If you do not want to repeat tons of text in the <head> of your page then you can just add data attributes to existing elements:

  <h1 data-swiftype-name="title" data-swiftype-type="string">title here</h1>
  <div data-swiftype-name="body" data-swiftype-type="text">
    Lots of body content goes here...
    Other content goes here too, and can be of any type, like a price:
    $<span data-swiftype-name="price" data-swiftype-type="float">3.99</span>

Thumbnail Image Tags

If you want to index images from your website and serve them as thumbnails to users in your search results, add a Swiftype image tag to the <head> section of your website template that indicates where images are located on your various page types, like this:

<meta class="swiftype" name="image" data-type="enum" content="" />

Robots Meta Tag Support

The Site Search Crawler supports the robots meta-tag standard. This allows you to control how the Site Search Crawler indexes the pages on your site.

Using the "robots" meta-tag

Place the robots meta-tag in the <head> section of your page:

Example - Place the robots meta tag in the head section
<!doctype html>
    <meta name="robots" content="noindex, nofollow">
    Page content here

Robots meta tag content values

Site Search supports the NOFOLLOW, NOINDEX, and NONE values for the robots tag. FOLLOW and INDEX are the defaults and are not necessary unless you are overriding a robots meta tag for Swiftype, see below. Other values - such as NOARCHIVE - are ignored.

To tell the Site Search Crawler not to index a page, use NOINDEX:

<meta name="robots" content="noindex">

Links from an unindexed page will still be followed.

To tell the Site Search Crawler not to follow links from a page, use NOFOLLOW.

<meta name="robots" content="nofollow">

Content from a page that has NOFOLLOW will still be indexed.

To not follow links and not index content from a page, use NOINDEX, NOFOLLOW or NONE.

<meta name="robots" content="noindex, nofollow">

NONE is a synonym for the above:

<meta name="robots" content="none">

We recommend specifying the robots directives in a single tag, but multiple tags will be combined if present.

Directing instructions at the Site Search Crawler only

Using the meta name="robots" will apply your instructions to all web crawlers, including Swiftbot, the Site Search Crawler. If you would like to direct special instructions at the Site Search Crawler, use st:robots as the name instead of robots.

Example - st:robots overrides robots for the Site Search Crawler
<meta name="robots" content="noindex, nofollow">
<meta name="st:robots" content="follow, index">

This example tells other crawlers not to index or follow links from the page, but allows the Site Search Crawler to index and follow links.

When any meta name of st:robots is present on the page, all other robots meta rules will be ignored in favor of the st:robots rule.

Repeated content values

If robots directives are repeated, the Site Search Crawler will use the most restrictive.

<meta name="robots" content="noindex">
<meta name="robots" content="index">

The above is equivalent to NOINDEX.

Casing, spacing, and ordering

Tags, attribute names, and attribute values are all case-insensitive.

Multiple attribute values must be separated by a comma, but whitespace is ignored. Order is not important (NOINDEX, NOFOLLOW is the same as NOFOLLOW, NOINDEX). The following are considered the same:

<meta name="robots" content="noindex, nofollow">
  <META name="rOBOTs" content="     noIndex    ,     NOfollow   ">

Stuck? Looking for help? Contact support or check out the Site Search community forum!