search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection
Swiftype Documentation / api: Asynchronous Indexing

Asynchronous Indexing

Swiftype's API supports asynchronous indexing to efficiently index large numbers of Documents.

Asynchronous indexing consists of two stages: submission and result. Both stages support batch operations.

Submitting a batch of Documents

First, submit a list of one or more Documents to the asynchronous indexing API. This API follows create-or-update semantics. The Documents will be created if they do not exist, or updated if they already exist.

This endpoint does basic parameter validation, but in order to check and see if the Documents were successfully created or updated you must check the document receipt resource. For your convenience, a link to show the status of all the documents included in the batch is returned in the result.

Example - Submitting a batch of Documents for asynchronous indexing.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents":
          [
            {"external_id": "3", "fields": [{"name": "title", "value": "my title 1", "type": "string"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]},
            {"external_id": "4", "fields": [{"name": "title", "value": "my title 1", "type": "enum"},
                                            {"name": "page_type", "value": "user", "type": "enum"}]}
          ]
      }'
  

Example Response

{
  "batch_link": "https://api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000001,5473d6142ed96065a9000002",
  "document_receipts": [
    {
      "id": "5473d6142ed96065a9000001",
      "external_id": "3",
      "status": "pending",
      "errors": [],
      "links": {
        "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
        "document": null
      }
    },
    {
      "id": "5473d6342ed96065a9000002",
      "external_id": "4",
      "status": "pending",
      "errors": [],
      "links": {
        "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000002.json",
        "document": null
      }
    }
  ]
}

Checking the status of document receipts

Once you have submitted documents for indexing, you can check the status of the documents using the document receipts API endpoint. Each receipt will have a status value.

Status Meaning
pending The Document has not yet been processed.
complete The Document was successfully created.
failed The Document could not be created. Check the errors array for details.
expired The receipt is over 24 hours old and can no longer be checked.
unknown The document receipt could not be found.

You may need to check the receipts a few times, because the Documents will be processed asynchronously.

You can check the status of document receipts with either a GET or POST request. There is a limit 1000 document receipts per API request.

To use a GET request, submit a request with the document receipt IDs joined together with , as the ids parameter (for convenience, a link to this resource is included as the batch_link when you submit a batch of documents).

Example - Checking the status multiple document receipts with a GET request
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000001,5473d6142ed96065a9000002'
  

Example Response

[
  {
    "id": "5473d6142ed96065a9000001",
    "external_id": "3",
    "status": "pending",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  },
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "complete",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": "https://api.swiftype.com/api/v1/engine/xyz/document_type/abc/document/547e513f2ed9609fa7000001.json"
    }
  }
]

Using a GET request is convenient, but to check the status of many document receipt IDs, a JSON encoded POST is easier.

Example - Checking the status of multiple document receipts with a POST request
curl -X POST 'https://api.swiftype.com/api/v1/document_receipts.json' \
     -H 'Content-Type: application/json'
     -d '{"ids": ["5473d6142ed96065a9000001", "5473d6142ed96065a9000002"]}'
  

Example Response

[
  {
    "id": "5473d6142ed96065a9000001",
    "external_id": "3",
    "status": "pending",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  },
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "complete",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": "https://api.swiftype.com/api/v1/engine/xyz/document_type/abc/document/547e513f2ed9609fa7000001.json"
    }
  }
]

When a document cannot be created due to incorrect input, the receipt will have a status of failed and include a list of error messages.

Example - A receipt for a failed document creation
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000002'
  

Example Response

[
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": "4",
    "status": "failed",
    "errors": ["Invalid type for title: Got enum, but title is already defined as string in schema."],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
      "document": null
    }
  }
]

Receipts are only available for 24 hours. If you request a receipt more than 24 hours old, you will see the expired status.

Example - Checking the status of an expired document receipt
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts.json?ids=5473d6142ed96065a9000002'
  

Example Response

[
  {
    "id": "5473d6142ed96065a9000002",
    "external_id": null,
    "status": "expired",
    "errors": [],
    "links": {
      "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000002.json",
      "document": null
    }
  }
]

Checking the status of a single document receipt

Checking multiple receipt IDs is most efficient, but you can also check the status of an individual receipt. This is especially useful for debugging.

You can check a document receipt's status by executing a GET request to the link returned in the async_bulk_create_or_update response.

Example - Checking the status of a single document receipt
curl -X GET 'https://YOUR_API_KEY:@api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json'
  

Example Response

{
  "id": "5473d6142ed96065a9000001",
  "external_id": "3",
  "status": "complete",
  "errors": [],
  "links": {
    "document_receipt": "https://api.swiftype.com/api/v1/document_receipts/5473d6142ed96065a9000001.json",
    "document": "https://api.swiftype.com/api/v1/engines/5025a35185307b737f000008/document_types/5025a35485307b737f00000a/documents/5025a3036052f6b650000006.json"
  }
}

Handling errors at submission time

The asynchronous indexing API handles errors related to the document at a later time. However, it performs basic parameter validation at submission time. Requests must contain a documents key that is a non-empty array. Additionally, there is a 100 document limit on the number of documents that may be submitted in a single batch as well as a maximum request size of 50 megabytes.

If a request cannot be validated, an error response will be returned.

Example - Incorrectly submitting Documents for asynchronous indexing.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/document_types/books/documents/async_bulk_create_or_update.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "documents": []
      }'
  

Example Response

{
  "error": "Invalid field type: documents"
}