search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection
Swiftype Documentation / crawler: Robots Meta Tag Support

Robots Meta Tag Support

The Swiftype web crawler supports the robots meta tag standard. This allows you to control how Swiftype indexes pages on your site.

Using the "robots" meta tag

Place the robots meta tag in the <head> section of your page:

Example - Place the robots meta tag in the head section
<!doctype html>
    <meta name="robots" content="noindex, nofollow">
    Page content here

Robots meta tag content values

Swiftype supports the NOFOLLOW, NOINDEX, and NONE values for the robots tag. FOLLOW and INDEX are the defaults and are not necessary (unless you are overriding a robots meta tag for Swiftype, see below. Other values (such as NOARCHIVE) are ignored.

To tell Swiftype not to index a page, use NOINDEX:

<meta name="robots" content="noindex">

Links from an unindexed page will still be followed.

To tell Swiftype not to follow links from a page, use NOFOLLOW.

<meta name="robots" content="nofollow">

Content from a page that has NOFOLLOW will still be indexed.

To not follow links and not index content from a page, use NOINDEX, NOFOLLOW or NONE.

<meta name="robots" content="noindex, nofollow">

NONE is a synonym for the above:

<meta name="robots" content="none">

We recommend specifying the robots directives in a single tag, but multiple tags will be combined if present.

Directing instructions at Swiftype only

Using the meta name="robots" will apply your instructions to all web crawlers, including Swiftype's. If you would like to direct special instructions at Swiftype's crawler, use st:robots as the name instead ofrobots.

Example - st:robots overrides robots for the Swiftype crawler
<meta name="robots" content="noindex, nofollow">
<meta name="st:robots" content="follow, index">

This example tells other crawlers not to index or follow links from the page, but allows Swiftype's crawler to index and follow links.

When any meta name of st:robots is present on the page, all other robots meta rules will be ignored in favor of the st:robots rule.

Repeated content values

If robots directives are repeated, Swiftype will use the most restrictive.

<meta name="robots" content="noindex">
<meta name="robots" content="index">

The above is equivalent to NOINDEX.

Casing, spacing, and ordering

Tags, attribute names, and attribute values are all case-insensitive.

Multiple attribute values must be separated by a comma, but whitespace is ignored. Order is not important (NOINDEX, NOFOLLOW is the same as NOFOLLOW, NOINDEX). The following are considered the same:

<meta name="robots" content="noindex, nofollow">
  <META name="rOBOTs" content="     noIndex    ,     NOfollow   ">