search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso
For an API-driven, developer-centric solution, consider App Search before continuing!


Document Indexing

To index is to ingest: to upload - to create, to retain, to store - and to organize documents within a clean structure. The act of ingestion takes data from your source and makes a copy of it within your Engine. The data is then structured as an index, so that it may searched over in flexible and clever ways. Your data will exist within your database and within your Engine - but the Engine data is indexed and searchable.

This endpoint allows granular, API-based control over how your documents are ingested and controlled within API-based Engines.

If you are unsure whether you have an API-based or Crawler-based Engine, read here.

This document has four core pieces:

Basic Indexing

The following example demonstrates how you can use POST to ingest a new document into your Engine for indexing. This method requires your private API Key and the name of your Engine, which you can acquire within your dashboard.

This section includes:

curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token":"YOUR_API_KEY",
        "document": {
          "external_id": "1",
          "fields": [
            {"name": "title", "value": "The Great Gatsby", "type": "string"},
            {"name": "author", "value": "F. Scott Fitzgerald", "type": "string"},
            {"name": "genre", "value": "fiction", "type": "enum"}
          ]
        }
      }'

curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token":"YOUR_API_KEY",
        "document": {
          "external_id": "2",
          "fields": [
            {"name": "title", "value": "The Brothers Karamazov", "type": "string"},
            {"name": "author", "value": "Fyodor Dostoevsky", "type": "string"},
            {"name": "genre", "value": "fiction", "type": "enum"}
          ]
        }
      }'

DocumentTypes

DocumentTypes specify the structure of a set of documents in the Engine and are the entry point for searches. There are multiple types of fields on a DocumentType — string, text, enum, integer, float, date, and location.

GET /api/v1/engines/{engine_id}/document_types.json
Example - Get every DocumentType in an Engine
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types.json?auth_token=YOUR_API_KEY'
POST /api/v1/engines/{engine_id}/document_types.json
Example - Create a DocumentType called books in the bookstore Engine
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "document_type": {"name": "books"}
      }'
GET /api/v1/engines/{engine_id}/document_types/{document_type_id}.json
Example - Get a specific DocumentType called books in the bookstore Engine
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books.json?auth_token=YOUR_API_KEY'
DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}.json
Example - Delete a DocumentType called books in the bookstore Engine
curl -X DELETE 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books?auth_token=YOUR_API_KEY'

documents

Documents represent all of the pieces of content in an Engine. They conform to a DocumentType. When you perform a search on a DocumentType, you will receive documents as results.

external_id is the only required field for a document. It can be any value, such as a numeric ID.

GET /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents.json
Example - List every Document in the books DocumentType
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents.json?auth_token=YOUR_API_KEY'
POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents.json
Example - Create a Document in the books DocumentType
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents.json' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_API_KEY",
      "document": {
        "external_id": "2",
        "fields": [
            {"name": "title", "value": "my title 1", "type": "string"},
            {"name": "page_type", "value": "user", "type": "enum"}
      ]}
    }'
GET /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{external_id}.json
Example - Get a specific Document with external_id=1 from the books DocumentType
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/1?auth_token=YOUR_API_KEY'

Updating Fields

Updates fields on an existing document. This method will only update existing fields on the document. It will not create new fields on an existing document. In most cases you probably want to use bulk_create_or_update_verbose, which will create new fields on existing documents.

Existing fields will not change if not listed in the request.

PUT /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{document_id}/update_fields.json
Example - Update fields for Document with external_id 1.
curl -X PUT 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/1/update_fields' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_API_KEY",
      "fields": {"title": "my title 2", "page_type": "user2"}
    }'
POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/create_or_update.json
Example - Create or update a Document in the books DocumentType
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/create_or_update.json' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_API_KEY",
      "document": {
        "external_id": "2",
        "fields": [
            {"name": "title", "value": "my new title", "type": "string"},
            {"name": "page_type", "value": "user", "type": "enum"}
        ]}
      }'
DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{document_id}.json
Example - Delete a Document with external_id=1 from the books DocumentType
curl -X DELETE 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/1?auth_token=YOUR_API_KEY'
DELETE /api/v1/engines/{engine_id}/document_types/page/documents/destroy_url?url={url}
Example - Delete a Document with url=http://example.com/section/page.html
curl -X DELETE 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/page/documents/destroy_url?url=http://example.com/section/page.html&auth_token=YOUR_API_KEY'

Bulk Operations

Bulk operations will allow you to perform rapid updates and avoid the latency overhead of making repeated requests.

You can:

bulk_create

To create documents in bulk, use bulk_create. bulk_create will return an error if a document with a given external_id already exists. In most cases, it is easier to use bulk_create_or_update_verbose.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_create
Example - Create Documents in bulk
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_create' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'

bulk_update

To update documents in bulk, use bulk_update. bulk_update will return an error if a document with a given external_id does not exist. In most cases, it is easier to use bulk_create_or_update_verbose.

PUT /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_update
Example - Update Documents in bulk
curl -X PUT 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "1", "fields": {"title": "my title 5", "page_type": "user2"}},
            {"external_id": "2", "fields": {"title": "my title 4", "page_type": "user2"}}
          ]
      }'

bulk_create_or_update_verbose

To create or update documents in bulk, use bulk_create_or_update_verbose. This endpoint returns an array of responses, one for each document. If the create or update was successful, the response will be true, and documents will be created if they do not exist or updated if they already exist.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_create_or_update
POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_create_or_update_verbose
Example - Create or update documents in bulk (all succeeded)
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_create_or_update_verbose' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
Example Response
[true, true]

If a create or update fails, the response for that document will be a string containing an error message.

Example - Error messages from create or update in bulk (all failed)
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_create_or_update_verbose' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"fields": [{"name": "title", "value": "my title 1", "type": "string"},
                        {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "bogus"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
Example Response
["Missing external_id", "Invalid field type: Invalid type for \"title\": \"bogus\""]

There is also a non-verbose version which only returns false when a create or update fails. This version is not recommended but it is still available for backwards compatibility.

bulk_destroy

To remove documents in bulk, use bulk_destroy. The output is an array of true/false values. If an individual destroy succeeds, its entry in the result array will be true. If it fails, its entry in the result array will be false.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_destroy
Example - Delete Documents in bulk
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_destroy' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents": ["1","2"]
      }'
Example Response
[true, true]

Asynchronous Indexing

The Site Search Document Indexing endpoint supports asynchronous ingestion to efficiently index large numbers of documents. Asynchronous indexing consists of two stages: submission and result. Both stages support batch operations.

This section covers:

Submitting a batch of documents

First, submit a list of one or more documents to the asynchronous indexing API. This API follows create-or-update semantics. The documents will be created if they do not exist, or updated if they already exist.

This endpoint does basic parameter validation, but in order to check and see if the documents were successfully created or updated you must check the document receipt resource. For your convenience, a link to show the status of all the documents included in the batch is returned in the result.

Example - Submitting a batch of documents for asynchronous indexing.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "enum"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
Example Response
{
  "batch_link": "https://api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000001,5473d6142ed96065a9000002",
  "document_receipts": [
    {
      "id": "5473d6142ed96065a9000001",
      "external_id": "3",
      "status": "pending",
      "errors": [],
      "links": {
        "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
        "document": null
      }
    },
    {
      "id": "5473d6342ed96065a9000002",
      "external_id": "4",
      "status": "pending",
      "errors": [],
      "links": {
        "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000002.json",
        "document": null
      }
    }
  ]
}

Checking the status of Document receipts

Once you have begun ingesting documents for indexing, you can check the status of the documents using the document receipts API endpoint. Each receipt will have a status value.

Status Meaning
pending The Document has not yet been processed.
complete The Document was successfully created.
failed The Document could not be created. Check the errors array for details.
expired The receipt is over 24 hours old and can no longer be checked.
unknown The document receipt could not be found.

You may need to check the receipts a few times because the documents will be processed asynchronously.

You can check the status of document receipts with either a GET or POST request. There is a limit 1000 document receipts per API request.

To use a GET request, submit a request with the document receipt IDs joined together with , as the ids parameter (for convenience, a link to this resource is included as the batch_link when you submit a batch of documents).

Example - Checking the status multiple document receipts with a GET request
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000001,5473d6142ed96065a9000002'
Example Response
[
  {
    "id": "5473d6142ed96065a9000001",
    "external_id": "3",
    "status": "pending",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  },
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "complete",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": "https://api.swiftype.com/api/v1/engine/xyz/document_type/abc/document/547e513f2ed9609fa7000001.json"
    }
  }
]

Using a GET request is convenient, but to check the status of many document receipt IDs, a JSON encoded POST is easier.

Example - Checking the status of multiple document receipts with a POST request
curl -X POST 'https://api.swiftype.com/api/v1/document_receipts.json' \
     -H 'Content-Type: application/json'
     -d '{"ids": ["5473d6142ed96065a9000001", "5473d6142ed96065a9000002"]}'
Example Response
[
  {
    "id": "5473d6142ed96065a9000001",
    "external_id": "3",
    "status": "pending",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  },
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "complete",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": "https://api.swiftype.com/api/v1/engine/xyz/document_type/abc/document/547e513f2ed9609fa7000001.json"
    }
  }
]

When a document cannot be created due to incorrect input, the receipt will have a status of failed and include a list of error messages.

Example - A receipt for a failed document creation
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000002'
Example Response
[
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "failed",
    "errors": ["Invalid type for title: Got enum, but title is already defined as string in schema."],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  }
]

Receipts are only available for 24 hours. If you request a receipt more than 24 hours old, you will see the expired status.

Example - Checking the status of an expired document receipt
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000002'
Example Response
[
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": null,
    "status": "expired",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000002.json",
      "document": null
    }
  }
]

Checking the status of a single document receipt

Checking multiple receipt IDs is most efficient, but you can also check the status of an individual receipt. This is especially useful for debugging.

You can check a document receipt's status by executing a GET request to the link returned in the async_bulk_create_or_update response.

Example - Checking the status of a single document receipt
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json'
Example Response
{
  "id": "5473d6142ed96065a9000001",
  "external_id": "3",
  "status": "complete",
  "errors": [],
  "links": {
    "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
    "document": "https://api.swiftype.com/api/v1/engines/5025a35185307b737f000008/document_types/5025a35485307b737f00000a/documents/5025a3036052f6b650000006.json"
  }
}

Handling errors at submission time

The Asynchronous Document Indexing endpoint handles errors related to the document at a later time. However, it performs basic parameter validation at submission time.

Requests must contain a documents key that is a non-empty array. Additionally, there is a 100 document limit on the number of documents that may be submitted in a single batch as well as a maximum request size of 50 megabytes.

If a request cannot be validated, an error response will be returned.

Example - Incorrectly submitting documents for asynchronous indexing.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents": []
      }'
Example Response
{
  "error": "Invalid field type: documents"
}

Document Expiration

If you have documents which you do not want to be searchable after some time in the future, you can use the Site Search document expiration feature to automatically delete them from your search engine.

This section contains instruction on...

Specifying document expiration when creating or updating documents

In order to create (or update) a document with an expiration date, you must use the asynchronous bulk indexing API. Add an expires_after key to the document object with the time to expire as a Unix Epoch timestamp as its value. The expires_after value must be in the future.

Example - Submitting a batch of documents with an expiration date.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {
              "external_id": "3",
              "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                         {"name": "page_type", "value": "user", "type": "enum"}],
              "expires_after": 1450574556
            },
            {
              "external_id": "4",
              "fields": [{"name": "title", "value": "my title 2", "type": "enum"},
                         {"name": "page_type", "value": "user", "type": "enum"}],
              "expires_after": 1450574556
            }
          ]
      }'
Note: There is no guarantee that a document will be removed exactly at its expires_after time. However, it is guaranteed that a document will not be automatically deleted before its expires_after time.

Searching with precise expiration

Documents are not guaranteed to be deleted exactly at their expiration time. Therefore, there is some period of time after a document expires that it will still appear in search results. Depending on your use case, this may be fine. However, if you need to ensure that expired documents do not appear in search, you can use the precise_expiration search option. This option is available for both the search and suggest API endpoints.

When the precise_expiration option is used, all documents that are not not expired (e.g., have an expires_after value in the future) and those that don't have an expires_after value set set will be returned. In other words, it filters all expired documents from the results.

The format of the precise_expiration option is as follows:

"precise_expiration": {
  "<document_type_1_slug>": true,
  "<document_type_2_slug>": true
}

The default value is false for all DocumentTypes - that is, expired but not yet deleted documents will be returned in the results.

Example - Specifying precise_expiration for a DocumentType.
curl -XGET 'https://search-api.swiftype.com/api/v1/engines/bookstore/document_types/books/search.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "q": "brothers",
        "precise_expiration": {
          "books": true
        }
      }'

Stuck? Looking for help? Contact Support!