Overview Crawler Developer API Platform API Tutorials Client Libraries

Using Swiftype in a Rails Application

It's easy to get started using Swiftype's API to index content in a Ruby on Rails application. This tutorial will walk through the process step by step. If you're impatient, you can check out the example app right away.

Screencast

Watch a screencast version of this tutorial.

The Starting Point: A Simple Blog Engine

For this tutorial, we will start with a simple Rails application with one model, Post. It has a title, body, and the created_at and updated_at timestamp fields.

class Post < ActiveRecord::Base
  validates_presence_of :title
  validates_presence_of :body

  attr_accessible :title, :body
end
create_table "posts", :force => true do |t|
    t.string   "title"
    t.string   "body"
    t.datetime "created_at", :null => false
    t.datetime "updated_at", :null => false
end

Adding Swiftype to the Application

Add the swiftype gem to the application's Gemfile.

gem 'swiftype'

Indexing content in Swiftype requires making a network call, so it's best to do this outside of the web request cycle. There are a number of background worker libraries suitable for the task of performing this work asynchronously.

For this example, we'll use delayed_job, but the code for the background jobs can be translated to other background libraries easily.

gem 'delayed_job_active_record'

After installing the delayed_job gem, create the delayed_jobs table.

rails generate delayed_job:active_record
rake db:migrate

To make managing the Rails and delayed_job process, we'll use foreman (this also makes moving the application to Heroku easy). The Procfile looks like this:

web: bundle exec rails server thin -p $PORT
worker:  bundle exec rake jobs:work

To start the application, type foreman start

Creating the Engine and Document Type

First, create a Swiftype account if you haven't yet.

Log into your Swiftype account and create an API-based search engine with a Document Type named "post". After you create the engine, you'll see your API key, Engine Slug, and Engine Key.

screenshot showing keys after engine creation

Make note of them, as you will need to configure the Rails application.

Forman supports setting environment variables for its processes with a .env file.

Create a .env file with the Swiftype API Key and your engine slug:

SWIFTYPE_API_KEY=your_api_key
SWIFTYPE_ENGINE_SLUG=your_engine_slug

Add an initializer to configure the Swiftype gem to config/initializers/swiftype.rb:

Swiftype.configure do |config|
  config.api_key = ENV['SWIFTYPE_API_KEY']
end

Creating, Updating, and Removing Documents with Swiftype

ActiveRecord callbacks are a convenient way to let Swiftype know about new posts, changes to existing posts, and deletions. To avoid waiting on a response from Swiftype's API during the web request, the application will enqueue a Delayed::Job to be processed later.

Swiftype documents have an External ID that ties a record in Swiftype to a record in your database. This must be unique within the Document Type. We will use the primary key of the Post model.

Swiftype provides many different field types such as string, text, enum, and date. For this example, a good choice is to map the title attribute to a string field so it can be used for autocompletion; the body attribute maps to a text field because it will be used for full-text search, but not autocompletion; the created_at attribute maps to a date field and can be used for range queries. There is no url attribute on the Post model, but we can construct it using the Rails router.

The URL should be stored as an enum because it must be stored verbatim. (For more details, see the Search Engine Schema Design tutorial.)

For creating and updating Documents in Swiftype's index, we will use the create_or_update API endpoint. This means we do not need to worry if a Post has already been indexed.

class CreateOrUpdateSwiftypeDocumentJob < Struct.new(:post_id)
  def perform
    post = Post.find(post_id)
    url = Rails.application.routes.url_helpers.post_url(post)
    client = Swiftype::Client.new
    client.create_or_update_document(ENV['SWIFTYPE_ENGINE_SLUG'],
                                     Post.model_name.downcase,
                                     {:external_id => post.id,
                                       :fields => [{:name => 'title', :value => post.title, :type => 'string'},
                                                   {:name => 'body', :value => post.body, :type => 'text'},
                                                   {:name => 'url', :value => url, :type => 'enum'},
                                                   {:name => 'created_at', :value => post.created_at.iso8601, :type => 'date'}]})
  end
end

Notice that the url field is set using the router. Since the CreateOrUpdateSwiftypeDocumentJob will not run during the web request, it will not know the host for generating the URL to the post. You must set Rails.application.routes.default_url_options[:host] to your application's host. The example app uses an environment variable and an initializer for this.

Sometimes it's necessary to remove content from the search engine, too. DeleteSwiftypeDocumentJob takes care of that.

class DeleteSwiftypeDocumentJob < Struct.new(:post_id)
  def perform
    client = Swiftype::Client.new
    client.destroy_document(ENV['SWIFTYPE_ENGINE_SLUG'], Post.model_name.downcase, post_id)
  end
end

Now that the background jobs are in place, we can add the callbacks in the Post model to enqueue them. It will now look like this:

class Post < ActiveRecord::Base
  validates_presence_of :title
  validates_presence_of :body

  attr_accessible :title, :body

  after_save :enqueue_create_or_update_document_job
  after_destroy :enqueue_delete_document_job

  private

  def enqueue_create_or_update_document_job
    Delayed::Job.enqueue CreateOrUpdateSwiftypeDocumentJob.new(self.id)
  end

  def enqueue_delete_document_job
    Delayed::Job.enqueue DeleteSwiftypeDocumentJob.new(self.id)
  end
end

Indexing Existing Content

If you're adding Swiftype to an existing application, you'll need to index existing content. You can do that one document at a type, like the CreateSwiftypeDocumentJob, but it will be faster and more efficient to batch the requests.

Swiftype's API supports bulk operations, so we'll write a rake task that creates 100 documents at a time. Again, we'll create or update documents (this time using bulk_create_or_update) so this Rake task can be run multiple times.

Run using:

foreman run rake index_posts

Here is the code for the task:

task :index_posts => :environment do
  if ENV['SWIFTYPE_API_KEY'].blank?
    abort("SWIFTYPE_API_KEY not set")
  end

  if ENV['SWIFTYPE_ENGINE_SLUG'].blank?
    abort("SWIFTYPE_ENGINE_SLUG not set")
  end

  client = Swiftype::Client.new

  Post.find_in_batches(:batch_size => 100) do |posts|
    documents = posts.map do |post|
      url = Rails.application.routes.url_helpers.post_url(post)
      {:external_id => post.id,
       :fields => [{:name => 'title', :value => post.title, :type => 'string'},
                   {:name => 'body', :value => post.body, :type => 'text'},
                   {:name => 'url', :value => url, :type => 'enum'},
                   {:name => 'created_at', :value => post.created_at.iso8601, :type => 'date'}]}
    end

    results = client.create_or_update_documents(ENV['SWIFTYPE_ENGINE_SLUG'], Post.model_name.downcase, documents)

    results.each_with_index do |result, index|
      puts "Could not create #{posts[index].title} (##{posts[index].id})" if result == false
    end
  end
end

Autocomplete

Now that content is getting indexed, it's time to add the search box, starting with autocomplete.

First, download the Swiftype autocomplete jQuery plugin and put it in vendor/assets/javascripts and add //= require jquery.swiftype.autocomplete to your application.js (you'll also need jQuery if you haven't installed it yet).

If you want to use the default Swiftype styling for autocomplete, you'll also need the autocomplete.css file included in that repository. Put it in vendor/assets/stylesheets and add //= require autocomplete to your application.css.

Next, add a search box to the layout. The Swiftype JavaScript targets this by a CSS selector. A DOM ID like st-search-input is namespaced to avoid CSS conflicts as well as nice and expressive.

<form>
  <input type='text' id='st-search-input' class='st-search-input'/>
</form>

The Swiftype API has a private API key which is secret and should only be used for server-side code. For public use like the autocomplete JavaScript, an Engine Key is used. We will need the Engine Key to configure the JavaScript, so add it to the .env file:

SWIFTYPE_ENGINE_KEY=your_engine_key

Now, configure the Swiftype JavaScript after the DOM is ready.

$(function() {
  $('#st-search-input').swiftype({
    engineKey: '<%= ENV['SWIFTYPE_ENGINE_KEY'] %>'
  });
});

By default, Swiftype will perform autocompletion against all the string fields, which in this example is only title. If you would like to restrict the fields matched, you can specify the searchFields option to the swiftype JavaScript function. Read the documentation for more details.

Everything is now in place for autocomplete queries! Add some posts if you haven't already and try out a search. When you type a word that exists in the title of a post, you'll see it as one of the autocompletion options. Select it, and you'll be taken directly to that page.

Now that Autocomplete is working, let's add full-text search. We'll do this two ways: first with the Swiftype JavaScript, and second by using the Swiftype API. Which option you use in your application depends on how you want your app to work. With the Swiftype JavaScript plugin, searches never hit your server, making results generally faster for users. However, the results must be rendered on the page after it loads.

If you want complete control over how Swiftype results are presented, you can call the Swiftype API from your server. This will result in slower search response because two round-trips will be required (user to your server, your server to Swiftype) instead of one (user to Swiftype).

Search with JavaScript

Performing a search with the Swiftype search jQuery plugin is similar to the autocomplete, but you must provide a container element for the displaying the results.

First, download the assets. The Swiftype search plugin requires jQuery (as above) and depends on Ben Alman's hashchange plugin (included in the repository). There is also a search.css file to implement the default formatting of search results. Put the JavaScript files in vendor/assets/javascripts and the CSS file (if you are using it) in vendor/assets/stylesheets

Update your application.js to require the new files:

//= require jquery.ba-hashchange.min
//= require jquery.swiftype.search

If you are using the search.css file, update your application.css to require it:

//= require search

Next, add the container element to the page.

<div id="st-results-container" class="st-result-listing"></div>

This can be put in the layout so search results are show on the same page as the search, or you can redirect the user to a search results page and show the results there. The example app displays the search results on the same page.

Finally, configure the jQuery plugin to attach to the input that we added in the previous step and render its results in the st-results-container div.

$(function() {
  $('#st-search-input').swiftypeSearch({
    resultContainingElement: '#st-results-container',
    engineKey: '<%= ENV['SWIFTYPE_ENGINE_KEY'] %>'
  });
});

With this in place, you can now perform queries powered by Swiftype. Try it by searching for a word in one of your posts. Hit enter, and the results will appear in the st-result-listing div.

The plugin is very customizable with configuration options for what fields to fetch and how to render the results. For details, check out the README file and the Customizing Search Results with the Swiftype jQuery Plugin tutorial.

Search with Server Pass-Through

To search Swiftype on the server, we will create a controller action that performs the search using the Swiftype client.

First, create the route in routes.rb: match '/search' => 'search#search'

Next, we'll add a controller action that performs the search and renders the results.

You perform a search with Swiftype::Client like this:

client = Swiftype::Client.new
results = client.search('your-engine-slug', 'the query', options)

The options hash is optional. Use it to specify additional options like facets, results per page, fields to return (see the search options for details).

The search method returns a Swiftype::ResultSet object. It acts like a Hash with an entry for each Document Type. Each key has a list of documents that matched the search. The text fields that matched the search will be called out in a highlight hash.

The Swiftype::ResultSet also includes information about pagination such as the current page and total number of pages in the results. We'll use this to display pagination controls for the search results view.

class SearchController < ApplicationController
  def search
    if params[:q]
      client = Swiftype::Client.new
      @results = client.search(ENV['SWIFTYPE_ENGINE_SLUG'], params[:q], {:per_page => '10', :page => params[:page] || 1})
      @post_results = @results['post']
    end
  end
end

To display the results, create a view:

<% if @post_results.empty? %>
    <p>Sorry, there were no results for your query.</p>
<% else %>
  <ul class="search_results">
    <% @post_results.each do |post_result| %>
      <li>
        <%# the 'highlight' hash is escaped HTML with unescaped <em> tags around the highlight data, so we call html_safe to render the tags %>
        <h3><%= link_to (post_result['highlight'].try(:[], 'title').try(:html_safe) || post_result['title']), post_path(post_result['external_id']) %></h3>
        <%= post_result['highlight'].try(:[], 'body').try(:html_safe) || post_result['body'] %>
      </li>
    <% end %>
  </ul>

  <% if @results.current_page > 1 %>
    <%= link_to '< previous', search_path(:q => params[:q], :page => @results.current_page - 1) %>
  <% end %>

  <% if @results.num_pages > @results.current_page %>
    <%= link_to 'next >', search_path(:q => params[:q], :page => @results.current_page + 1) %>
  <% end %>
<% end %>

Now you can compare searching via the Swiftype jQuery plugin and passing the query through your application first.

Managing Multiple Environments

Most Rails applications exist in multiple environments (development, test, production) and have multiple team members contributing. Because Swiftype ties documents in the index to documents in your application by external_id, if you have multiple environments or team members writing and deleting to the same index, it will quickly get out of sync. Queries will match documents that don't exist in an application, an application may try to delete a document that doesn't exist, or create one that already does.

To avoid this problem, you should create a separate engine for each environment and developer. With the approach given in this sample application, every team member can specify their own SWIFTYPE_ENGINE_KEY in the .env file. In production, simply set this environment variable to the production engine's Engine Key.

Your test environment can call the Swiftype API, but network calls are slow. For test speed, it is best to use a gem such as webmock or vcr to stub out the network requests.