Adding Swiftype-powered Search to any CMS

Content management systems are great for creating websites, but most have terrible search. Once you've added Swiftype to your CMS, you'll be able to take advantage of our powerful search servers to remove load from your database and provide modern features like fast typeahead autocomplete and relevance-based full-text search.

Content management systems have some nice properties that make adding Swiftype search simple. With a CMS, your content is stored in a database of some kind and you can control how it is output, or receive an event when a new item is added.

There are two ways to add Swiftype-powered search to a CMS: you can use the Swiftype web crawler, or you can use the Swiftype API.

Using the Swiftype web crawler

Using Swiftype's web crawler is the easiest way to get started with Swiftype. It works well for most web sites, so most CMS-based sites will be easy to integrate with Swiftype. There are a few things you can do that will make the crawler work better with your site.

Give the Swiftype crawler an easy way to find content

Swiftype supports Sitemap.xml files and RSS/Atom feeds as a starting point for crawling your site. If your site provides these feeds it will greatly speed up crawling your site and it will make incremental updates much more useful.

Your CMS probably has plugins that will create these from its database of your content.

Tell Swiftype what parts of your site to index

Swiftype will index your entire site by default (up to the page limit of your plan). To index only parts of your site, restrict the Swiftype crawler with Domain Path Exclusions or robots.txt rules.

Add data with Swiftype meta tags

Swiftype offers meta tags that allow you to add data to your search engine's index.

A common case is to improve how titles are displayed. For example, for SEO purposes you may want your title to be something like "Page title - [more information about topic] - Your Site Name". But this makes little sense for internal search results.

Swiftype removes most boilerplate text from titles to improve display, but you can customize the behavior with a title meta tag.

Another useful addition to the search index is a thumbnail image. If your CMS associates an image with a page, you can give Swiftype the URL of that image by using an image meta tag.

Finally, most CMSes know when a page was originally published. Using the published_at meta tag, you can tell Swiftype that specific information as well. This is useful for sorting search results.

Here's an example using the Jekyll static site generator to set some Swiftype meta tags in a layout:

Example - Swiftype meta tags using Jekyll's Liquid template syntax and variables

<head>
    <title>{{ page.title }} | Swiftype Crawler Demo Site</title>
    <meta class="swiftype" name="title" data-type="string" content="{{ page.title }}" >
    <meta class="swiftype" name="published_at" data-type="date" content="{{ page.date | date_to_xmlschema }}">
</head>

Optimize content extraction with data attributes

Swiftype attempts to extract the most relevant text from pages. However, if extraneous text from navigation menus or footers is interfering with your search results, you can exclude it from your index with the data-swiftype-index attribute.

To include only content inside a div use data-swiftype-index="true":

Example - Using data-swiftype-index="true" to index only part of a page's content

<html>
  <body>
    <div>
      This is navigation: <a href="/">home</a>
    </div>

    <div data-swiftype-index="true">
      This is the page's content. It will be indexed.
    </div>

    <div>
      This is a footer. &copy; 2013 YourSite.com
    </div>
  </body>
</html>

Sometimes it may be easier to exclude certain parts of the page instead. You can do that with data-swiftype-index="false":

Example - Using data-swiftype-index="false" to index only part of a page's content

<html>
  <body>
    <div data-swiftype-index="false">
      This is navigation: <a href="/">home</a>
    </div>

    <div>
      This is the page's content. It will be indexed.
    </div>

    <div data-swiftype-index="false">
      This is a footer. &copy; 2013 YourSite.com
    </div>
  </body>
</html>

The Swiftype data attribute is easy to add to your CMS's template, and will help Swiftype extract the most meaningful content from your site.

Use the Swiftype embed code or jQuery libraries to implement search

Since you are using the web crawler, you can use the Swiftype install code provided in the Swiftype dashboard. It will look something like this:

<script type="text/javascript">
  (function(w,d,t,u,n,s,e){w['SwiftypeObject']=n;w[n]=w[n]||function(){
  (w[n].q=w[n].q||[]).push(arguments);};s=d.createElement(t);
  e=d.getElementsByTagName(t)[0];s.async=1;s.src=u;e.parentNode.insertBefore(s,e);
  })(window,document,'script','//s.swiftypecdn.com/install/v2/st.js','_st');

  _st('install','YOUR_INSTALL_KEY','2.0.0');
</script>

You can customize how your search works by adjusting the configuration in the Swiftype Dashboard. You can update the configuration to "take over" your CMS's existing search by by using a CSS selector. You can also change how your search looks by adjusting the CSS.

Another way to implement search is to use the Swiftype search and autocomplete jQuery libraries. This will give you complete control over how autocomplete and search results are rendered.

Ping Swiftype when content is published

Many CMSes offer a callback that can be used to "hook into" saving a piece of content. For example, WordPress has save_post, Drupal has hook_node_insert and hook_node_update, and Ruby on Rails models have after_save callbacks. You can use these callbacks to notify Swiftype of new content to index. This is optional, but will bring your site's search engine up-to-date quickly after publishing content.

Swiftype offers an API endpoint to request the indexing of a URL. If the URL exists in the search engine, Swiftype will update its contents. If the URL does not exist in the search engine, Swiftype will add it.

In order to use this API, you need to know the your Engine Slug, the ID of your domain, and the URL you want to crawl (along with your API key).

Your Engine Slug can be found on your Swiftype dashboard
Your Domain ID can be found by using the GET /api/v1/engines/YOUR_ENGINE_SLUG/domains.json API endpoint

From the command line, the API call looks like this:

curl -X PUT 'https://api.swiftype.com/api/v1/engines/YOUR_ENGINE_SLUG/domains/YOUR_DOMAIN_ID/crawl_url.json' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_AUTH_TOKEN",
      "url": "http://example.com/new-page"
    }'

You can make the equivalent HTTP request using whichever facility your CMS provides.

If possible, it is recommended to make this API call outside of the web request cycle (that is, in a background worker). This will prevent content updates from failing due to timeouts or request failures.

Request a full recrawl when site is re-generated

If you are using a static-file CMS like Jekyll, you can request a full recrawl of your site (this is subject to your plan's limits) when you publish the site. As above, you'll need your Engine Slug, Domain ID, and API Key to issue the recrawl API request.

Here's an example Rake task that publishes a Jekyll site with rsync, then issues a recrawl API request:

Example - Rake task to publish a Jekyll site and request a recrawl

desc 'Build and deploy to web servers then recrawl.'
task :deploy => :jekyll do
  sh "jekyll --no-auto"
  sh "rsync --recursive --times --compress --human-readable --progress --delete _site/ deploy@yoursite.com:/var/www/blog/"
  sh "curl -XPUT -H 'Content-Length: 0' 'https://api.swiftype.com/api/v1/engines/YOUR_ENGINE_SLUG/domains/YOUR_DOMAIN_ID/recrawl.json?auth_token=YOUR_AUTH_TOKEN'"
end

Using the Swiftype API

Using the Swiftype API gives you total control over how your content is indexed and puts you in charge of keeping it up-to-date in your search engine. In general, the approach to using the API will follow this pattern: perform a create_or_update request when content is saved and a delete request when content is removed (or hidden).

You can find examples of using the API in the Swiftype WordPress plugin and Swiftype Ruby on Rails example application.

Create a schema that fits your data model

The first thing you will need to do is create a search engine schema that fits your data model and how you want search to work.

Read the Swiftype Schema Design Tutorial and review the page DocumentType schema for ideas.

Most CMS-based search engines will have fields like: title, author, url, body, and published_at. You may also want to index things like excerpts, tags or category names.

Perform an initial content index

If your CMS already has data in it, you'll need to perform an initial index of the content.

To do this, you can use the bulk_create_or_update API endpoint to send about 100 Documents at a time. (Why use create_or_update instead of create? If you use create_or_update, it will be easier to run multiple times, overwriting previous data instead of causing errors due to duplicate documents.)

Our Rails example includes a Rake task that does this using our Ruby library:

Example - Rake task to index existing content with bulk_create_or_update

task :index_posts => :environment do
  if ENV['SWIFTYPE_API_KEY'].blank?
    abort("SWIFTYPE_API_KEY not set")
  end

  if ENV['SWIFTYPE_ENGINE_SLUG'].blank?
    abort("SWIFTYPE_ENGINE_SLUG not set")
  end

  client = Swiftype::Client.new

  Post.find_in_batches(:batch_size => 100) do |posts|
    documents = posts.map do |post|
      url = Rails.application.routes.url_helpers.post_url(post)
      {:external_id => post.id,
       :fields => [{:name => 'title', :value => post.title, :type => 'string'},
                   {:name => 'body', :value => post.body, :type => 'text'},
                   {:name => 'url', :value => url, :type => 'enum'},
                   {:name => 'created_at', :value => post.created_at.iso8601, :type => 'date'}]}
    end

    results = client.create_or_update_documents(ENV['SWIFTYPE_ENGINE_SLUG'], Post.model_name.downcase, documents)

    results.each_with_index do |result, index|
      puts "Could not create #{posts[index].title} (##{posts[index].id})" if result == false
    end
  end
end

Update Swiftype Documents when changes occur in your CMS

When an item in your CMS changes, you'll need to update it in Swiftype so the search index stays up-to-date.

There are some common cases:

A new item is created: index it with create_or_update
An existing item is updated: index it with create_or_update
An existing item is deleted: delete it with DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{external_id}
An existing item is hidden (such as changed from published to draft): delete it with DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{external_id}

You can see some of these in action in the Rails example's Post model.

Use the Swiftype jQuery plugins to implement search

When you use the Swiftype API, the standard embed install code is not available because it doesn't work with custom DocumentTypes. Therefore, you'll need to implement search using the Swiftype jQuery plugins.

You can find an example in the swiftype-api-example repository.

Typically, you will want to implement a renderFunction for search and autocomplete to display the results as you want, though the plugins come with a basic version that work for most engines.

// searchRenderFunction will be called once for each full-text search result.
// documentType is a String with your Document Type slug (e.g., "page", "post", etc.)
// item is an Object representing a search result. Its fields are determined by your engine schema.
var searchRenderFunction =  function(documentType, item) {
    var out = '<a href="' + item['url'] + '" class="st-search-result-link">' + item['title'] + '</a>';
    out.concat('<p class="genre">' + item[#39;genre'] + '</p>');
    return out;
};

// autocompleteRenderFunction will be called once for each autocomplete result.
// documentType is a String with your Document Type slug (e.g., "page", "post", etc.)
// item is an Object representing a search result. Its fields are determined by your engine schema.
var autocompleteRenderFunction = function(documentType, item) {
  return '<p class="title"><img src="' + item['thumbnail_url'] + ' width="25" height="25">' + item['title'] + '</p>';
};

// swiftypeSearch comes from the jquery.swiftype.search library and implements full-text search
$(function() {
  $('#st-search-input').swiftypeSearch({
    resultContainingElement: '#st-results-container',
    engineKey: 'YOUR_ENGINE_KEY',
    renderFunction: searchResultRenderFunction,
  });
});

// swiftype comes from the jquery.swiftype.autocomplete library and implements autocompletion
$('#st-search-input').swiftype({
  engineKey: 'YOUR_ENGINE_KEY',
  renderFunction: autocompleteRenderFunction,
});

For a full list of options supported by the jQuery plugins, see their documentation.

tutorials