search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso

Document Indexing

Add, delete, update, or retrieve documents from within your API-based Engine.

This document covers four different indexing methods:

Engine Type Supported?
Crawler-based Engine NO
API-based Engine YES

Read more about Crawler-based Engines and API-based Engines within the API Overview.

API-based Engines

An API-based Engine provides manual control over how documents are indexed into your Engine. Using an API-based Engine forgoes the Site Search Crawler.

The endpoint requires your private API Key and your Engine Slug, which you can acquire within your dashboard.

This section includes:

DocumentTypes

DocumentTypes specify the structure of a set of documents and are the entry point for searches.

For more information, read the API Overview.

Add a DocumentType

Add a new DocumentType to your Engine.

POST /api/v1/engines/{engine_id}/document_types.json
Example - Create a DocumentType called books in the bookstore Engine
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "document_type": {"name": "books"}
      }'

GET a DocumentType

List all DocumentTypes or list a specific DocumentType by id.

List all DocumentTypes
GET /api/v1/engines/{engine_id}/document_types.json
Example - Get every DocumentType in an Engine
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types.json?auth_token=YOUR_API_KEY'
List a specific DocumentType by id
GET /api/v1/engines/{engine_id}/document_types/{document_type_id}.json
Example - Get a specific DocumentType called books in the bookstore Engine
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books.json?auth_token=YOUR_API_KEY'

Delete a DocumentType

Delete a DocumentType from your Engine.

DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}.json
Example - Delete a DocumentType called books in the bookstore Engine
curl -X DELETE 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books?auth_token=YOUR_API_KEY'

Documents

Documents represent all of the pieces of content in an Engine. They conform to a DocumentType. When you perform a search on a DocumentType, you will receive documents as results.

For more information, read the API Overview.

Create a document

Add a new document into your Engine or create or update a document.

external_id
required
Can be any value, such as a number or BSON id.
Create a new document
POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents.json
Example - Create a Document in the books DocumentType
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents.json' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_API_KEY",
      "document": {
        "external_id": "2",
        "fields": [
            {"name": "title", "value": "my title 1", "type": "string"},
            {"name": "page_type", "value": "user", "type": "enum"}
      ]}
    }'
Create or update a document

If the external_id exists, update the document. If it does not, create a new document.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/create_or_update.json
Example - Create or update a Document in the books DocumentType
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/create_or_update.json' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_API_KEY",
      "document": {
        "external_id": "2",
        "fields": [
            {"name": "title", "value": "my new title", "type": "string"},
            {"name": "page_type", "value": "user", "type": "enum"}
        ]}
      }'

GET a document

List all documents within a DocumentType or list a specific document within a DocumentType by id.

List all documents within a document
GET /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents.json
Example - List every Document in the books DocumentType
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents.json?auth_token=YOUR_API_KEY'
List a specific document by id
GET /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{external_id}.json
Example - Get a specific Document with external_id=1 from the books DocumentType
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/1?auth_token=YOUR_API_KEY'

DELETE a document

Within a DocumentType, you can delete a document by external_id or the document URL.

DELETE document by external_id
DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{document_id}.json
Example - Delete a Document with external_id=1 from the books DocumentType
curl -X DELETE 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/1?auth_token=YOUR_API_KEY'
DELETE document by URL
DELETE /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/destroy_url?url={url}
Example - Delete a Document with url=http://example.com/section/page.html
curl -X DELETE 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/page/documents/destroy_url?url=http://example.com/section/page.html&auth_token=YOUR_API_KEY'

Update document fields

Updates fields on an existing document.

Existing fields will not change if not listed in the request.

PUT /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/{document_id}/update_fields.json
Example - Update fields for Document with external_id 1.
curl -X PUT 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/1/update_fields' \
-H 'Content-Type: application/json' \
-d '{
      "auth_token": "YOUR_API_KEY",
      "fields": {"title": "my title 2", "page_type": "user2"}
    }'

Bulk Indexing

Bulk operations allow you to perform rapid updates and avoid the latency overhead of repeated requests.

You can:

Bulk Create or Update, Verbose

To create or update documents in bulk, use bulk_create_or_update_verbose. This endpoint returns an array of responses, one for each document. If the create or update was successful, the response will be true, and documents will be created if they do not exist or updated if they already exist.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_create_or_update_verbose
Example - Create or update documents in bulk (all succeeded)
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_create_or_update_verbose' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
Example Response
[true, true]

If a create or update fails, the response for that document will be a string containing an error message.

Example - Error messages from create or update in bulk (all failed)
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_create_or_update_verbose' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"fields": [{"name": "title", "value": "my title 1", "type": "string"},
                        {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "bogus"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
Example Response
["Missing external_id", "Invalid field type: Invalid type for \"title\": \"bogus\""]

There is also a deprecated non-verbose version which only returns false when a create or update fails. This version is not recommended but it is still available:

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_create_or_update

Bulk Create

To create documents in bulk, use bulk_create. bulk_create will return an error if a document with a given external_id already exists.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_create
Example - Create Documents in bulk
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_create' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'

Bulk Update

To update documents in bulk, use bulk_update. bulk_update will return an error if a document with a given external_id does not exist.

PUT /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_update
Example - Update Documents in bulk
curl -X PUT 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "1", "fields": {"title": "my title 5", "page_type": "user2"}},
            {"external_id": "2", "fields": {"title": "my title 4", "page_type": "user2"}}
          ]
      }'

Bulk Destroy

To remove documents in bulk, use bulk_destroy. The output is an array of true/false values. If an individual destroy succeeds, its entry in the result array will be true. If it fails, its entry in the result array will be false.

POST /api/v1/engines/{engine_id}/document_types/{document_type_id}/documents/bulk_destroy
Example - Delete Documents in bulk
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/bulk_destroy' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents": ["1","2"]
      }'
Example Response
[true, true]

Asynchronous Indexing

The Site Search Document Indexing endpoint supports asynchronous indexing to efficiently index large numbers of documents. Asynchronous indexing consists of two stages: submission and result. Both stages support batch operations.

This section covers:

Submitting a batch of documents

First, submit a list of one or more documents to the asynchronous indexing API. This API follows create-or-update semantics. The documents will be created if they do not exist, or updated if they already exist.

This endpoint does basic parameter validation, but in order to check and see if the documents were successfully created or updated you must check the document receipt resource. For your convenience, a link to show the status of all the documents included in the batch is returned in the result.

Example - Submitting a batch of documents for asynchronous indexing.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "enum"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
Example Response
{
  "batch_link": "https://api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000001,5473d6142ed96065a9000002",
  "document_receipts": [
    {
      "id": "5473d6142ed96065a9000001",
      "external_id": "3",
      "status": "pending",
      "errors": [],
      "links": {
        "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
        "document": null
      }
    },
    {
      "id": "5473d6342ed96065a9000002",
      "external_id": "4",
      "status": "pending",
      "errors": [],
      "links": {
        "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000002.json",
        "document": null
      }
    }
  ]
}

Checking the status of Document receipts

Once you have begun indexing documents, you can check the indexing status by using the document receipts API endpoint. Each receipt will have a status value.

Status Meaning
pending The Document has not yet been processed.
complete The Document was successfully created.
failed The Document could not be created. Check the errors array for details.
expired The receipt is over 24 hours old and can no longer be checked.
unknown The document receipt could not be found.

You may need to check the receipts a few times because the documents will be processed asynchronously.

You can check the status of document receipts with either a GET or POST request. There is a limit 1000 document receipts per API request.

To use a GET request, submit a request with the document receipt IDs joined together with , as the ids parameter (for convenience, a link to this resource is included as the batch_link when you submit a batch of documents).

Example - Checking the status multiple document receipts with a GET request
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000001,5473d6142ed96065a9000002'
Example Response
[
  {
    "id": "5473d6142ed96065a9000001",
    "external_id": "3",
    "status": "pending",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  },
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "complete",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": "https://api.swiftype.com/api/v1/engine/xyz/document_type/abc/document/547e513f2ed9609fa7000001.json"
    }
  }
]

Using a GET request is convenient, but to check the status of many document receipt IDs, a JSON encoded POST is easier.

Example - Checking the status of multiple document receipts with a POST request
curl -X POST 'https://api.swiftype.com/api/v1/document_receipts.json' \
     -H 'Content-Type: application/json'
     -d '{"ids": ["5473d6142ed96065a9000001", "5473d6142ed96065a9000002"]}'
Example Response
[
  {
    "id": "5473d6142ed96065a9000001",
    "external_id": "3",
    "status": "pending",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  },
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "complete",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": "https://api.swiftype.com/api/v1/engine/xyz/document_type/abc/document/547e513f2ed9609fa7000001.json"
    }
  }
]

When a document cannot be created due to incorrect input, the receipt will have a status of failed and include a list of error messages.

Example - A receipt for a failed document creation
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000002'
Example Response
[
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "failed",
    "errors": ["Invalid type for title: Got enum, but title is already defined as string in schema."],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  }
]

Receipts are only available for 24 hours. If you request a receipt more than 24 hours old, you will see the expired status.

Example - Checking the status of an expired document receipt
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000002'
Example Response
[
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": null,
    "status": "expired",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000002.json",
      "document": null
    }
  }
]

Checking the status of a single document receipt

Checking multiple receipt IDs is most efficient, but you can also check the status of an individual receipt. This is especially useful for debugging.

You can check a document receipt's status by executing a GET request to the link returned in the async_bulk_create_or_update response.

Example - Checking the status of a single document receipt
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json'
Example Response
{
  "id": "5473d6142ed96065a9000001",
  "external_id": "3",
  "status": "complete",
  "errors": [],
  "links": {
    "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
    "document": "https://api.swiftype.com/api/v1/engines/5025a35185307b737f000008/document_types/5025a35485307b737f00000a/documents/5025a3036052f6b650000006.json"
  }
}

Handling errors at submission time

The Asynchronous Document Indexing endpoint handles errors related to the document at a later time. However, it performs basic parameter validation at submission time.

Requests must contain a documents key that is a non-empty array. Additionally, there is a 100 document limit on the number of documents that may be submitted in a single batch as well as a maximum request size of 10 megabytes.

If a request cannot be validated, an error response will be returned.

Example - Incorrectly submitting documents for asynchronous indexing.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents": []
      }'
Example Response
{
  "error": "Invalid field type: documents"
}

Document Expiration

If you have documents which you do not want to be searchable after some time in the future, you can use the Site Search document expiration feature to automatically delete them from your Engine.

This section contains instruction on...

Specifying document expiration when creating or updating documents

In order to create (or update) a document with an expiration date, you must use the asynchronous bulk indexing API. Add an expires_after key to the document object with the time to expire as a Unix Epoch timestamp as its value. The expires_after value must be in the future.

Example - Submitting a batch of documents with an expiration date.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {
              "external_id": "3",
              "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                         {"name": "page_type", "value": "user", "type": "enum"}],
              "expires_after": 1450574556
            },
            {
              "external_id": "4",
              "fields": [{"name": "title", "value": "my title 2", "type": "enum"},
                         {"name": "page_type", "value": "user", "type": "enum"}],
              "expires_after": 1450574556
            }
          ]
      }'
Note: There is no guarantee that a document will be removed exactly at its expires_after time. However, it is guaranteed that a document will not be automatically deleted before its expires_after time.

Searching with precise expiration

Documents are not guaranteed to be deleted exactly at their expiration time. Therefore, there is some period of time after a document expires that it will still appear in search results. Depending on your use case, this may be fine. However, if you need to ensure that expired documents do not appear in search, you can use the precise_expiration search option. This option is available for both the search and suggest API endpoints.

When the precise_expiration option is used, all documents that are not not expired (e.g., have an expires_after value in the future) and those that don't have an expires_after value set set will be returned. In other words, it filters all expired documents from the results.

The format of the precise_expiration option is as follows:

"precise_expiration": {
  "<document_type_1_slug>": true,
  "<document_type_2_slug>": true
}

The default value is false for all DocumentTypes - that is, expired but not yet deleted documents will be returned in the results.

Example - Specifying precise_expiration for a DocumentType.
curl -XGET 'https://search-api.swiftype.com/api/v1/engines/bookstore/document_types/books/search.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "q": "brothers",
        "precise_expiration": {
          "books": true
        }
      }'

Stuck? Looking for help? Contact Support!