search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso

How to Build Application Search with Ruby on Rails

You will need an App Search account to follow along. Sign up for a free trial.


When people interact with their computer or phone, often times it looks like this: activate the screen, open the browser, type cryptic strings into an empty search bar, scan the results for a moment, then click on a top result. Search has given tremendous power to Internet users. One can find their desired information or product and they can find it fast. Miraculous, indeed, but with great power comes great expectations.

If you have a website or application, your users will demand similar expediency. App Search is a product that can help you streamline the information or product acquisition phase of the users' web experience. You want that, too, because more rewarding interactions will help you accomplish your business goals. No one wants to wade through pages of results! They want magical boxes that transport them to exactly what they are seeking - or something even better. This tutorial will demonstrate how to start building such an experience.

The completed sample application is live here: gemhunt.swiftype.info. Try it out!

The unfinished code is available on here.

You can access the completed branch to see the finished source code.

Requirements

  • A recent version of Ruby installed on your device. Need help? See Ruby.

  • Half an hour or less.

  • An active App Search account or free trial.

Getting Search-y

We are going to build a simple, engaging search experience on top of Ruby on Rails.

In doing so we shall learn how to...

  • Set-up a sample search application.
  • Create an Engine within App Search.
  • Configure the App Search Ruby Client.
  • Ingest documents.
  • Alter the schema.
  • Fine-tune Search relevance.

In the end, we will have a powerful, slick and intuitive search-based Rails application. As one gains comfort with Search development, they can use their own data and modify the application as they see fit. Perhaps this sample application will become the foundation for something magnificent.

Setup

To get started, clone the tutorial repository and run bin/setup. This will install bundler and the required gems, setup the SQLite database and populate it with seed data. The sample seed data that we will search over is composed of JSON. The JSON contains a set of popular RubyGems. Everyone loves RubyGems! We can examine the raw data within data/rubygems.json.

$ git clone git@github.com:Swiftype/app-search-rails-tutorial.git
$ cd app-search-rails-tutorial

$ bin/setup

To make sure everything is in order, start the app with rails server.

$ rails server
=> Booting Puma
=> Rails 5.2.0 application starting in development
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.11.3 (ruby 2.5.1-p57), codename: Love Song
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://0.0.0.0:3000
Use Ctrl-C to stop

Once the server has started, point your browser at localhost:3000.

It should look something like this:

Gem Hunt - Where your Gems come true
Initial App Screenshot

Looks gooood ~ now, try a query. Hmm. That is not optimal. No matter what we query, a gigantic, linear list of RubyGems comes back. It seems we are just showing the set of data! Although it is paginated and styled in a crisp and tidy way, this is not valuable. It would be better if visitors could... search!

Enter App Search

Begin a free trial of Swiftype App Search by creating an account. Once we have logged in for the first time, we will be prompted to create an Engine.

An Engine is a repository that houses our indexed documents. The App Search platform interacts with the Engine, providing search analytics and tools to help curate results, manage synonyms and much more. An Engine contains documents; documents are often objects, products, profiles, articles -- they can be many things.

Given that we are going to fill our Engine with RubyGems, how about we keep it simple and call it... ruby-gems.

Welcome to App Search - Start your Engine
Create Engine

Install & Configure Swiftype App Search Client

Swiftype provides an official Ruby Client. Through it, we can access the App Search API from within Ruby-based applications. We want to use it!

Open up the Gemfile and add:

gem 'swiftype-app-search', '~> 0.3.0'

Then, run bundle install to install the gems.

$ bundle install

Next, we will need credentials to authorize against the App Search API. We will need the Host Identifier and the Private API Key.

The Host Identifier is a unique value that represents an account. The Private API Key is a standard, all-access key that can manipulate any resource except those dealing with other credentials. Given its powerful nature, we want to keep it secret - and safe.

There are many different ways to keep track of API Keys and other secret information in your development environment. The dotenv gem is a strong choice. However, to keep things nice and clear - albeit, not as secure - we have placed the values within a swiftype.yml file. The swiftype.yml file is included within our .gitgnore. Should you want to host your application somewhere, you will need to bring the credentials with you.

The tutorial's setup script created config/swiftype.yml for us. We should now fill in our Host Identifier and Private API Key.

# config/swiftype.yml

app_search_host_identifier: [HOST_IDENTIFIER] # It should start with "host-"
app_search_api_key: [API_KEY] # It should start with "private-"

Initialize ~

With your new Engine, matching key, and Host Identifier, we can create a new initializer within config/initializers so that we may bring App Search to life:

# config/initializers/swiftype.rb

Rails.application.configure do
  swiftype_config = YAML.load_file(Rails.root.join('config', 'swiftype.yml'))

  config.x.swiftype.app_search_host_identifier = swiftype_config['app_search_host_identifier']
  config.x.swiftype.app_search_api_key = swiftype_config['app_search_api_key']
end

The client will be used in several places. We should wrap it in a small class. To do that, we will craft a new lib directory within app/ for our new Search class.

# app/lib/search.rb

class Search
  ENGINE_NAME = 'ruby-gems'

  def self.client
    @client ||= SwiftypeAppSearch::Client.new(
      host_identifier: Rails.configuration.x.swiftype.app_search_host_identifier,
      api_key: Rails.configuration.x.swiftype.app_search_api_key,
    )
  end
end

We are almost ready to index some documents. Before we do that, we need to restart Spring so that it will pick-up our new app/lib directory...

$ bundle exec spring stop

Bring on the Documents

For now, our documents exist within our local SQLite database. We need to move these documents into App Search, into our Engine. The act of doing so is known as indexing. We want to index the data, so that it may be structured and searched upon.

If we have documents in two places: the Engine and the database, then we need to establish truth. The application database is our "Source of Truth". As users interact with our application, the state of database items will change. Our Engine needs to be aware of those changes.

We can take advantage of Active Record Lifecycle Callbacks to keep the two data sets in sync. To do so, we will add an after_commit callback to notify App Search of any new or updated documents committed to the database and an after_destroy callback for when a document is removed.

# app/models/ruby_gem.rb

class RubyGem < ApplicationRecord
  validates :name, presence: true, uniqueness: true

  after_commit do |record|
    client = Search.client
    document = record.as_json(only: [:id, :name, :authors, :info, :downloads])

    client.index_document(Search::ENGINE_NAME, document)
  end

  after_destroy do |record|
    client = Search.client
    document = record.as_json(only: [:id])

    client.destroy_documents(Search::ENGINE_NAME, [ document[:id] ])
  end

 # ...

end

As this is an example case, we are calling the Swiftype App Search API in a synchronous way. The optimal method when dealing with external services like the App Search API is to use an asynchronous callback to avoid hanging up other application requests. For more information on asynchronous call writing, check out the ActiveJob framework provided by Rails.

Catchy Hooks

Before we apply more code changes to the application, a demonstration of our callbacks.

Open up a rails console from within the project directory.

$ rails console
...
Running via Spring preloader in process 15983
Loading development environment (Rails 5.2.0)
irb(main):001:0>

Within the console, we can explore our documents. Reveal yourself, puma!

irb(main):008:0> puma = RubyGem.find_by_name('puma')
=> # ...

Next, we can make a small change to the document named puma that we have found...

irb(main):009:0> puma.info += ' Also, pumas are fast.'
=> # ...

... and then save the document.

irb(main):010:0> puma.save
=> true

The call to save should trigger the after_commit callback. Moments later, if we open the documents panel in the App Search Dashboard, we should see a document that corresponds to the puma gem.

Ingestion - It starts with one
Puma Document

Our first indexed document! Huzzah! Although, we have many documents yet to index...

Mass Ingestion

If we were building an App Search application from scratch, we would not need to worry about indexing our existing data; the after_commit hook would handle new documents as they are added. However, our example application already has more than 11,000 RubyGem documents.

To index them all without waiting for individual after_commit hooks on each record, we can write a rake task.

We will place our app_search.rake task within the lib/tasks/ directory that lives under the project root. Do note that this is not our app/lib/ directory:

# lib/tasks/app_search.rake

namespace :app_search do
  desc "index every Ruby Gem in batches of 100"
  task seed: [:environment] do |t|
    client = Search.client

    RubyGem.in_batches(of: 100) do |gems|
      Rails.logger.info "Indexing #{gems.count} gems..."

      documents = gems.map { |gem| gem.as_json(only: [:id, :name, :authors, :info, :downloads]) }

      client.index_documents(Search::ENGINE_NAME, documents)
    end
  end
end

The next step is to run this task from the command line. Consider watching the log file in another terminal to see it in action. Seeing documents race into your Engine is fun!

To do so, type: tail -F log/development.log within another terminal window, then use rails to initiate the task:

$ rails app_search:seed

Ingestion begins! If you take another look at the documents panel in the App Search Dashboard, you should see that all of your documents are now indexed within your Engine. Check out your schema, too, and perhaps try some sample queries from the dashboard:

All the Documents - Perhaps read some?
All the Documents

We now have an Engine bubbling with documents. It is time to alter our RubyGemsController#index. This is when search starts to come to life! We will re-construct the controller so that we transform our current 'return all-the-things' text box into a true search bar.

# app/controllers/ruby_gems_controller.rb

class RubyGemsController < ApplicationController

  PAGE_SIZE = 30

  def index
    if search_params[:q].present?
      @current_page = (search_params[:page] || 1).to_i

      search_client = Search.client
      search_options = {
        page: {
          current: @current_page,
          size: PAGE_SIZE,
        },
      }

      search_response = search_client.search(Search::ENGINE_NAME, search_params[:q], search_options)
      @total_pages = search_response['meta']['page']['total_pages']
      result_ids = search_response['results'].map { |rg| rg['id']['raw'].to_i }

      @search_results = RubyGem.where(id: result_ids).sort_by { |rg| result_ids.index(rg.id) }
    end
  end

  def show
    @rubygem = RubyGem.find(params[:id])
  end

  private

  def search_params
    params.permit(:q, :page)
  end
end

Open up localhost:3000 and try it out! Neat! We can search, a little...! But we want to search well.

Highly Relevant

Results appear, which is good. However, searching with a query string of rake returns the rake gem as the 14th result. This is not ideal! What can we do!?

By default, App Search treats all fields with equal importance. We know from experience that users will search by the name of the gem. We should give that field more importance than the others. If we give the name field a higher weight than info and authors, then it will have the greatest impact on the final document score. The document score, the relevance, is what governs the response order.

# app/controllers/ruby_gems_controller.rb

# ...

def index
  if search_params[:q].present?
    @current_page = (search_params[:page] || 1).to_i

    search_client = Search.client
    search_options = {
      search_fields: {
        name: { weight: 2.0 },
        info: {},
        authors: {},
      },
      page: {
        current: @current_page,
        size: PAGE_SIZE,
      },
    }

    search_response = search_client.search(Search::ENGINE_NAME, search_params[:q], search_options)
    @total_pages = search_response['meta']['page']['total_pages']
    result_ids = search_response['results'].map { |rg| rg['id']['raw'].to_i }

    @search_results = RubyGem.where(id: result_ids).sort_by { |rg| result_ids.index(rg.id) }
  end
end

# ...
If you provide the search_fields option to the Searching API, you must include every field you would like to be included in the search. This is why we added info and authors, even though we are not passing any search options for those fields.


Weights are powerful. You can read more about them within our App Search Relevance Tuning guide. If we try another search, we can see that the rake gem is first result! Relevance is improved. But there is even more that we can do.

Change the Field

We want to add an option to our search interface that filters out RubyGems that do not have many downloads. We want to see what is Popular.

First, we will need to make a small change to our Engine's schema to filter the downloads field within a numeric range. Our Engine schema displays the type of data that is contained within each document field. By default, App Search assumes every field is text. Fields can be: text, number, date, or geolocation.

To address this, we can change the downloads field to type Number from within the Schema tab of the App Search Dashboard.

Before:

Existing Field Type
name Text
author Text
info Text
downloads Text

After:

Existing Field Type
name Text
author Text
info Text
downloads Number

Be sure to click Change Types after making the change.

Changing these fields begins a reindex of your data. This might take some time, depending on the size of your Engine. You are unable to change Fields during a reindex.
Field Fixing - From Text to Number
Changing the downloads field type to Number

App Search will now consider the downloads field to be a Number.

Chart Topping

Our designer - as usual - is ahead of the game. The application already contains a check-box to emphasize Only Popular results. This form_tag is what will allow us to define a :popular parameter within our controller:

# app/views/ruby_gems/index.html.erb

# ...

  <%= form_tag({}, {method: :get}) do %>
    <div class="form-group row">
      <%= text_field_tag(:q, params[:q], class: "form-control", placeholder: "My favorite gem...") %>
    </div>

     <div class="form-check">
       <%= check_box_tag('popular', 1, params[:popular], class: 'form-check-input') %>
       <label class="form-check-label" for="popular">Only include gems with more than a million downloads.</label>
     </div>

    <div class="form-group row">
      <%= submit_tag("Search", class: "btn btn-primary mb-2") % >
    </div>
  <% end %>

#...

Back in our controller, we will do just that. Swiftype App Search allows us to pass filters along with our search options. In this case, our filter will prioritize results that have at least 1,000,000 views. When a field is a Number, numerical filtering like this becomes possible.

# app/controllers/ruby_gems_controller.rb

  # ...

  def index
    if search_params[:q].present?
      @current_page = (search_params[:page] || 1).to_i

      search_client = Search.client
      search_options = {
        search_fields: {
          name: { weight: 2.0 },
          info: {},
          authors: {},
        },
        page: {
          current: @current_page,
          size: PAGE_SIZE,
        },
      }

      if search_params[:popular].present?
        search_options[:filters] = {
          downloads: { from: 1_000_000 },
        }
      end

      # ...

    end

  private

  def search_params
    params.permit(:q, :page, :popular)
  end

When we venture back to our application, we can try some nifty queries. Let us search for heyzap-authlogic-oauth. With the checkbox un-checked, it is the first result. With the box checked we return more popular gems with a wider audience, like authlogic and oauth. That is more like it! Our sample is complete. But we have just scratched the surface when it comes to building search.

Summary

Excellent search is delightful for users. Whether you want search to help people explore products, find relevant content or helpful documents, or are basing your entire application around robust discovery through geolocation or time frames, Swiftype App Search is a wise choice. With all the tools that the API presents to you, the power to craft imaginative and intuitive search experiences is at your finger-tips.

Keep Learning!