search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection
Swiftype Documentation / crawler: Content Inclusion & Exclusion

Include or Exclude content from your Search Engine

Content Inclusion

In some cases you may have many items you want excluded from the page but only a few you would like included. In these scenarios you can use the content inclusion feature to whitelist elements of your pages for indexing. For example, if you have a single content section of every page you can very easily set data-swiftype-index=true on this element and our crawler will only extract text from there.

<body>

  This is content that will not be indexed by the Swiftype crawler.

  <div data-swiftype-index='true'>
    <p>
      All content under the above div tag will be indexed.
    </p>
    <p>
      Content in this paragraph tag will be included from the search index!
    </p>
  </div>

  This content will not be indexed, since it isn't surrounded by an include tag.

</body>

Content Exclusion

You may want to keep certain content from being indexed in your site search engine -- for example, your site header, footer, or menu bar. You can tell the Swiftype crawler to exclude these elements by adding the data-swiftype-index=false attribute to any element, as illustrated below.

<body>

  This is your page content, which will be indexed by the Swiftype crawler.

  <p data-swiftype-index='false'>
    Content in this paragraph tag will be excluded from the search index!
  </p>

  This content will be indexed, since it isn't surrounded by an excluded tag.

  <div id='footer' data-swiftype-index='false'>
    This footer content will be excluded as well.
  </div>

</body>

Nested Rules

Nesting the above rules will work as you might expect. If there are multiple rules present on the page, all text will inherit its behavior from the nearest parent element with a rule. Thus you can very granularly include and exclude elements within each other. If there is text outside of any rules on the page, it will be treated the opposite of the first-appearing rule on the page. For example, if the first rule is data-swiftype-index=false, any text without a rule will be indexed in the search engine.

<body>

  This is content that will not be indexed since the first rule is true.

  <div data-swiftype-index='true'>
    <p>
      All content under the above div tag will be indexed.
    </p>
    <p>
      Content in this paragraph tag will be included from the search index!
    </p>
    <p data-swiftype-index='false'>
      Content in this paragraph will be excluded because of the nested rule.
    </p>
  </div>

</body>