search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso

Custom API Sources Guide

Checkout the Custom API Source Reference for a more concise API reference.


Enterprise Search provides a variety of popular content sources like GitHub, Google Drive, Salesforce, and Dropbox.

But what if you want to search over internal applications and other homemade sources of data?

The answer is a Custom API Source.

A couple of key points to start:

  1. You can create any number of Custom API Sources.

  2. You can only create, update and destroy documents in one Custom API Source at a time.

  3. You can ingest any sort of data, so long as it adheres to a schema of your choosing and some basic stylistic parameters.

This guide will demonstrate how to create a Custom API Source, ingest data according to a schema, and design your search results.

Create a Custom API Source

A Custom API Source is a content source. It can be access controlled like any other content source.

Once you have created a Custom API Source, you'll receive an Auth Token and a Content Source Key.

You can then ingest and manage data from any source via API for organization wide searching.

Within the administrative dashboard, select Sources from the sidebar:

Custom API Sources - Click Sources within the sidebar.
A picture of the sidebar within the administrative dashboard. The item

You'll see a list of sources from which to choose.

Click the text that reads Create a Custom API Source or click Add under the Custom API card:

Custom API Sources - Select Custom API.
A list of all the available content sources. Custom API has a blue sphere.

Your custom source will need a name.

You might have many custom sources. People will can decide whether to add or remove the source from their own search experiences.

It's best to make it something clear, concise, and descriptive.

Before you can create the source, you need to acknowledge that the source will be available to everyone.

If you understand the implications - that the source's content will be searchable by others -- click I understand:

Custom API Sources - Name the key, then click I Understand to open the Custom API Source to all users.
When creating custom sources, they will be accessible by all users. This agreement makes it clear that people than just you will be able to access and search the content source.

Once you click Continue, the source has been created.

You will receive the Auth Token and a Content Source Key:

Custom API Sources - Retrieve your Auth Token and Content Source Key.
Once you have agreed and created the Custom API Source, you'll receive one token and a key.

With the token and the key, you're ready to start ingesting data.

... But first it is important to understand the basics of schema design.

Custom API Source Schema Overview

You can have up to 64 schema fields.

Data that matches a defined schema helps Enterprise Search interpret and style the information.

Objects can take any shape, but there are some parameters within which you need to work.

You can build a schema before you ingest your data.

Click Sources in the sidebar of the administrative dashboard.

Next to the Custom API Source, select Manage.

Custom API Sources - Clicking Manage
The organization sources view with a pink box over the Manage text.

Enter the Schema view from the menu:

Custom API Sources - Design your schema.
A view showing schema design. You can click Add Field or Update Field. There are dropdown menus next to each type.a

Click Add Field to add a new schema field.

You'll be able to give it a name and provide a type.

Anytime you'd like to change types, select a new one from the dropdown and then click Update Types.

You can ingest a document to automatically create default fields of type text, then alter the type.

When it comes to schema design, remember The Five Keys:

  1. Always index new fields as the same type as existing documents.
    • eg. An existing date field should not receive geolocation data.
  2. Arrays are supported, but nested objects are not supported. You must flatten them.
  3. Fields cannot be deleted once they have been created.
  4. The reserved fields are: external_id, source, content_source_id, updated_at, last_updated, highlight, any, all, none, or, and, not, engine_id.
  5. A field name can only contain lowercase letters, numbers, and underscores.

Enterprise Search fields can be one of four different types:

  1. Text
  2. Number
  3. Date
  4. Geolocation

text fields

Text fields are at the heart of search.

They enable deeply analyzed full text search.

This is the default type for all new fields.

Any group of characters or text that you want to search over should be text.

eg. A description of an object, the name of a product, the content of a review.

number fields

A finite double-precision floating point value: 3.14 or 42.

Number fields enable fine grained sorting, filtering, faceting, and boosting.

eg. A price, a review score, the number of visitors, or a size.

date fields

Dates must be in ISO 8601 format, eg: "2013-02-27T18:09:19".

eg. A product release or publish date, birth date, an air date.

geolocation fields

Geographic coordinates can leverage the location field.

The location field allows filtering by distance from a specified point.

A location is specified using a JSON object containing the longitude and latitude, such as:"37.7894758, -122.3940638".

The separating space after the comma may be omitted: "37.7894758,-122.3940638".

eg. A store where a product is located, location of a venue.

Ingest Data via Custom API Source

There are three first party API clients:

Send objects to your Enterprise Search Custom API Source using the client of your choice.

When your objects have been received by Enterprise Search, they become searchable documents.

An indexing request for data ingestion looks like so:

Example - A POST request to index documents.
curl -X POST http://localhost:3002/api/v1/ent/sources/[CONTENT_SOURCE_KEY]/documents/bulk_create \
-H "Authorization: Bearer [ACCESS_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  {
    "id" : 1234,
    "title" : "A List of Accounting Documents",
    "body" : "The difference between a budding entrepreneur who merely shows promise and one who is already enjoying some success often comes down to mentoring",
    "url" : "https://www.shopify.com/content/5-tips-on-finding-a-mentor",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'

Example - A POST request to index documents.
No Java example available, showing cURL
curl -X POST http://localhost:3002/api/v1/ent/sources/[CONTENT_SOURCE_KEY]/documents/bulk_create \
-H "Authorization: Bearer [ACCESS_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  {
    "id" : 1234,
    "title" : "A List of Accounting Documents",
    "body" : "The difference between a budding entrepreneur who merely shows promise and one who is already enjoying some success often comes down to mentoring",
    "url" : "https://www.shopify.com/content/5-tips-on-finding-a-mentor",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'

Example - A POST request to index documents.
const SwiftypeEnterpriseClient = require('swiftype-enterprise-node')

const accessToken = '' # Your access token const contentSourceKey = '' // Your content source key const documents = [ { id: 1234, title: "5 Tips On Finding A Mentor", body: "The difference between a budding entrepreneur who merely shows promise and one who is already enjoying some success often comes down to mentoring.", url: "https://www.shopify.com/content/5-tips-on-finding-a-mentor", "created_at": "2019-06-01T12:00:00+00:00", "updated_at": "2018-07-11T12:00:00+00:00", "type": "list" }, { id: 1235, title: "How to Profit from Your Passions", body: "Want to know the secret to starting a successful business? Find a void and fill it.", url: "https://www.shopify.com/content/how-to-profit-from-your-passions", "created_at": "2019-06-01T12:00:00+00:00", "updated_at": "2018-07-11T12:00:00+00:00", "type": "list" } ]

const swiftype = new SwiftypeEnterpriseClient(accessToken) swiftype.indexDocuments(contentSourceKey, documents) .catch((error) => { // handle error })

Example - A POST request to index documents.
require 'swiftype-enterprise'

access_token = '' # Your access token content_source_key = '' # Your content source key documents = [ { 'id' => 1234, 'url' => 'https://www.shopify.com/content/5-tips-on-finding-a-mentor', 'title' => '5 Tips On Finding A Mentor', 'body' => 'The difference between a budding entrepreneur who merely shows promise and one who is already enjoying some success often comes down to mentoring', 'created_at' => '2019-06-01T12:00:00+00:00', 'updated_at'=> '2018-07-11T12:00:00+00:00', 'type' => 'list' } ]

SwiftypeEnterprise.access_token = access_token client = SwiftypeEnterprise::Client.new begin ingest = client.index_documents(content_source_key, documents) rescue => e # handle error end

Example - A POST request to index documents.
from swiftype_enterprise import SwiftypeEnterpriseClient
from swiftype_enterprise.exceptions import SynchronousDocumentIndexingFailed

content_source_key = '' # Your content source key access_token = '' # Your access token client = SwiftypeEnterpriseClient(access_token) documents = [ { "id" : 1234, "title" : "5 Tips On Finding A Mentor", "body" : "The difference between a budding entrepreneur who merely shows promise and one who is already enjoying some success often comes down to mentoring", "url" : "https://www.shopify.com/content/5-tips-on-finding-a-mentor", "created_at": "2019-06-01T12:00:00+00:00", "updated_at": "2018-07-11T12:00:00+00:00", "type": "list" } ] try: ingest = client.index_documents(content_source_key, documents, timeout=10, delay=2) except SynchronousDocumentIndexingFailed: # Timed out before documents could finish indexing pass

Example - A POST request to index documents.
No Javascript example available, showing cURL
curl -X POST http://localhost:3002/api/v1/ent/sources/[CONTENT_SOURCE_KEY]/documents/bulk_create \
-H "Authorization: Bearer [ACCESS_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  {
    "id" : 1234,
    "title" : "A List of Accounting Documents",
    "body" : "The difference between a budding entrepreneur who merely shows promise and one who is already enjoying some success often comes down to mentoring",
    "url" : "https://www.shopify.com/content/5-tips-on-finding-a-mentor",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'

It's best to think of these objects as symbols, which represent the greater document to which they link.

Fields should be descriptive enough that your users will be able to find what they need given loose querying attempts.

If the information is concise, clear, and descriptive, a user will find the document, click the url, and then discover the correct page.

The id is vital for keeping your custom source up to date.

If you delete objects within your source, send a POST request with an existing id to the bulk_destroy endpoint.

If the objects are updated, send a POST request with an existing id to the bulk_create endpoint.

Once you have ingested data, the next step is to build and style your results.

Custom API Source Display Settings

You can ingest any sort of object. And then the object becomes a document and a person can search for it.

The search experience should feel good -- as such, we've developed custom display settings.

Display Settings bring your documents to life with style and colour:

Custom API Sources - Click Display Settings.
The Custom API Source menu. Display Settings is present and outlined by a pink box.

A result appears when someone searches.

Depending on how relevant the result is, you will receive a prominent card or an organized list.

A prominent card is a Featured Result.

An organized list is a Standard Result.

A Featured Result is a direct match on a search term -- or an almost direct match.

A Standard Result will gather a set of similar matches that belong within the custom source.

Given that there will be multiple sources, this will bring great "scan-ability" to vague result sets.

There are five categories which together make up a robust result, two of which are optional:

  1. Title: The prominent name of the document.
  2. URL: When you click on a document, where does it go?
  3. Colour: Choose a unique colour for the result. Colours will help at-a-glance separation of results.
  4. Subtitle: An optional field. A smaller title under the main, prominent title.
  5. Description: An optional field. A more extensive description of the document's contents.

By default, the fields will be populated in alphabetical order according to schema.

You'll want to customize that. And so, your schema fields are available to each category.

Click on the dropdown menu and then select whichever field makes the most sense.

A realized result will look like so:

Custom API Sources - A fully styled search result.
A selection of fields as outlined above. The image shows Featured Results and Standard Results.

The other tab is Result Detail.

When you click a search result, a Preview pane appears with deeper document contents.

You can add fields along with a custom header.

And then you can structure your result previews however you see fit:

Custom API Sources - Result details.
The result details view shows a preview pane and a few selectable fields that can be expanded. Some example data is present and a preview of a document is present.

For example, consider a field called "type" which describes whether a document is a list, log, or spreadsheet.

You can click Add Field, select the type field, then write Type of Document as the label.

The preview will then update so that the information is present:

Custom API Sources - Adding another result detail.
The view now has another item in the preview pane: Type of Document. The type list is shown.

Click Save.

You have now styled your Custom API Source and your team is ready to find it in their search results.

Great!


Enjoying the beta? Something broken? Lost? Please send us your feedback or visit the Enterprise Search community.