search mobile facets autocomplete spellcheck crawler rankings weights synonyms analytics engage api customize documentation install setup technology content domains user history info home business cart chart contact email activate analyticsalt analytics autocomplete cart contact content crawling custom documentation domains email engage faceted history info install mobile person querybuilder search setup spellcheck synonyms weights engage_search_term engage_related_content engage_next_results engage_personalized_results engage_recent_results success add arrow-down arrow-left arrow-right arrow-up caret-down caret-left caret-right caret-up check close content conversions-small conversions details edit grid help small-info error live magento minus move photo pin plus preview refresh search settings small-home stat subtract text trash unpin wordpress x alert case_deflection advanced-permissions keyword-detection predictive-ai sso

Crawler Operations

These commands provide you with greater control over the Site Search Crawler.

Engine Type Supported?
Crawler-based Engine YES
API-based Engine NO

Read more about Crawler-based Engines and API-based Engines within the API Overview.

Crawler-based Engines

If you do not wish to create documents by hand, you can create a domain object within an Engine and the Site Search Crawler will automatically crawl the domain and create a document for each page it finds.

This section contains instructions upon how to:

Reminder: The following crawler specific endpoints only apply to Crawler-based Engines. It is not possible to make use of the crawler specific endpoints for API-based Engines.

Adding a domain

Add a new domain that you own to your Engine.

Please consider the limitations of your plan when adding additional domains. Note that the domains will be crawled and your pages will be indexed, which will impact your document count. See pricing for more details.

POST /api/v1/engines/{engine_id}/domains.json
Example - Create a domain for example.com in the bookstore Engine. After the call, a crawl will begin to index the domain.
curl -X POST 'https://api.swiftype.com/api/v1/engines/bookstore/domains.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "domain": {"url": "http://example.com"}
      }'

GET a domain

List all domains or a single domain by id.

All domains

GET /api/v1/engines/{engine_id}/domains.json
Example - List every domain in the bookstore Engine
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/domains.json?auth_token=YOUR_API_KEY'

Single domain

GET /api/v1/engines/{engine_id}/domains/{domain_id}.json
Example - List a single domain within the bookstore Engine by domain id.
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/domains/590cad86.json?auth_token=YOUR_API_KEY'

Delete a domain

Delete a domain from your Engine. The Engine will not be crawled.

DELETE /api/v1/engines/{engine_id}/domains/{domain_id}.json
Example - Get a specific domain with ID 4fcec5182f527673a0000006 in the bookstore Engine.
curl -X GET 'https://api.swiftype.com/api/v1/engines/bookstore/domains/4fcec5182f527673a0000006.json?auth_token=YOUR_API_KEY'

Recrawl a domain

You may trigger a re-crawl of a domain via the API.

The frequency of the requests are dictated by your specific plan limitations. See pricing for more details.

PUT /api/v1/engines/{engine_id}/domains/{domain_id}/recrawl.json
Example - Recrawl a domain with ID 4fcec5182f527673a0000006 in the bookstore Engine.
curl -X PUT -H 'Content-Length: 0' 'https://api.swiftype.com/api/v1/engines/bookstore/domains/4fcec5182f527673a0000006/recrawl.json?auth_token=YOUR_API_KEY'

Crawl a single URL

Trigger a crawl of a single URL by passing the URL along with your domain to the crawl_url endpoint. If the URL belongs to the parent domain and has a corresponding document, the document will be updated. If the URL validates against the parent domain and has not yet been indexed, it will be added. If the url does not validate against the parent domain and has been indexed, then the matching document will be deleted.

PUT /api/v1/engines/{engine_id}/domains/{domain_id}/crawl_url.json
Example - Crawl a URL in the domain with ID 4fcec5182f527673a0000006 in the bookstore Engine.
curl -X PUT 'https://api.swiftype.com/api/v1/engines/bookstore/domains/4fcec5182f527673a0000006/crawl_url.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "YOUR_API_KEY",
        "url": "http://example.com/new-page"
      }'

Error Cases

Crawling the URL happens asynchronously. As a result, there is limited error checking. It is helpful to be aware of scenarios that will prevent the URL from being added as a document:

  • The URL does not respond successfully (for example, if it responds HTTP 404 or HTTP 503).
  • The URL does not belong to the domain. For example, you try to crawl http://anotherdomain.net/contact on the example.com domain.
  • The URL is a duplicate of an existing document.
  • The URL is excluded by robots.txt, a robots meta tag, or a path whitelist or blacklist.
  • The URL has a canonical link that belongs to a different page or domain.

Stuck? Looking for help? Contact support or check out the Site Search forum!