Meta Engines Guide

NOTE: Meta Engines are only available in the Elastic Cloud and Self-Managed version of App Search, with a Platinum License.

What is a Meta Engine

A Meta Engine is an Engine that has no Documents of its own. Instead, it combines multiple other Engines so that they can be searched together as if they were a single Engine.

The Engines that comprise a Meta Engine are referred to as Source Engines.

Example: Two Engines, "western-national-parks" and "eastern-national-parks" can be combined in a Meta Engine named "parks" so that they can be searched as one data-set.

GET /api/as/v1/engines/parks/search

{
  "results": [
    {
      "id": "eastern-national-parks|park_shenandoah",
      "_meta": {
        "id": "park_shenandoah",
        "engine": "eastern-national-parks"
      }
    },
    {
      "id": "western-national-parks|park_yosemite",
      "_meta": {
        "id": "park_yosemite",
        "engine": "western-national-parks"
      }
    }
  ]
}

Why use a Meta Engine?

Meta Engines allow users to create a document-less Engine that searches over a set of existing Engines, with search settings that are separate from those Source Engines. They enable comprehensive search over a collection of documents that have subsets with different search setting requirements.

For example, search experiences in large organizations often require documents from multiple business units, with standalone sites with Relevance Tunings specific to their needs. They still need to be part of a company-wide search experience, and organization-wide search priorities that require Curations or Relevance Tunings that differ from specific business-unit needs. Meta Engines allow both to exist together with a single engine-source-of-truth.

Permissions also come into play - one unit might not want other units to be able to write new documents into their Engine, for example. Remember, anyone with access to a Meta Engine has read access to all the Source Engines attached to it.

Pitfalls

Meta Engines are useful but there are many other cases where a single Engine is a better choice. Here are some examples where Meta Engines are not the right solution:

• Large amounts of fields: Meta Engines aren't a way to expand the number of fields for a single set of documents. The best search results appear after condensing fields to just the essentials within one Engine. Remember that more fields results in slower queries.
• A way to avoid identifying search language: If you have the same content in multiple languages, indexing all of them using the Universal language choice works well. If you do have different Engines for each language, a much more effective pattern is to identify the language -- e.g. via browser language settings or language detection -- and direct the query to the correct Engine.
• Adjustments to Relevance Tuning for a single Engine: Search settings are controllable from the query side; consider passing Search Fields query parameters instead.

Things to Remember

There are a few important things to remember that will help make Meta Engines work for you:

• Unified Schemas: Meta Engine schemas are the union of all Source Engines schemas. Therefore, if two Source Engines share a field, their names must match - if the names are different, they will be treated as a different field! As well, the fields need to have the same type. See more below on how schema conflicts are treated.
• Schema changes bubble up: Changes to a Source Engine’s schema will bubble up to a Meta Engine - so when changing field names or types in Source Engines, remember those are passed to the Meta Engine. Make sure to adapt any queries that rely on specific fields or their types.
• Settings changes do not bubble up: Things like Curations, Relevance Tuning, or Result Settings that are made in a Source Engine will not affect queries on the Meta Engine.
• Permissions: Meta Engine users have read permissions to all the docs visible to the Meta Engine. Conversely, having write permissions to a Meta Engine doesn't grant write permission to Source Engines.
• Documents and Schemas within Meta Engines are Read-Only - Meta Engines contain no Documents of their own. Because of this, no Document management or Schema updates can be performed directly on a Meta Engine, and instead must be performed on individual Source Engines.

Creating Meta Engines in the UI

You must have the "Admin" or "Owner" role to create a Meta Engine.

Navigate to the Meta Engine page using the Meta Engines link in the main navigation. This is the main page where Meta Engines are managed.

Click on Create a Meta Engine. Name your Meta Engine and pick 2 or more existing Engines to use as Source Engines.

After completing this form your Meta Engine will be available for use immediately.

Changing Source Engines in Meta Engines

Source Engines can be freely changed after a Meta Engine has been created.

Within the navigation of a Meta Engine, navigate to the Engines page via the Engines link. This page lists the current Source Engines for your Meta Engine and allows you to add and remove them.

Managing Meta Engines with the API

Meta Engines can also be created and managed through the API.

See the Meta Engines API reference for more information.

Meta Engine Schema and Conflicts

A Meta Engine's Schema is the combination of the Source Engines' schemas. This can be seen on the Meta Engine's Schema page, which shows the schema fields along with the Source Engines those fields originate from.

If two or more Source Engines define a field with the same name and the same field type, then they will be treated as a single field within the schema.

Example: If two Source Engines, western-national-parks and eastern-national-parks, each have a "name" field of type "text", the Meta Engines schema will also have a "name" field of type "text".

If however, two or more Source Engines have a field with the same name and a different field type, then that is considered a Schema Conflict and that field will be completely disabled; meaning it will not be returned in search results, it cannot be filtered on, it cannot be used for boosting, etc.

Example: If two source Engines, western-national-parks and eastern-national-parks, each have a "visitors" field, but western-national-parks defines it as "text" and eastern-national-parks defines it as "number", then the "visitors" field will be disabled.

Resolving a Schema Conflict

A Schema Conflict can be resolved by navigating to the Source Engine directly in the UI and updating the field type of the conflicting field to match that of the other Source Engines.

Example: To resolve a Schema Conflict on the "visitors" field between the western-national-parks and eastern-national-parks Source Engines, you would navigate to the western-national-parks Engine's Schema page and update the "visitors" field to type "number", or vice versa.

Scoped Document IDs

Documents within Meta Engines always use a Scoped ID, rather than a typical Document ID. A Scoped ID is the combination of Engine Name and Document ID with a pipe delimiter.

{Engine Name}|{Document ID}

Example: eastern-national-parks|park_shenandoah

Keep this in mind, because you will need to use a Scoped ID when using API endpoints like the Curations endpoint for Meta Engines, which requires a Document ID as a parameter.

Stuck? Looking for help? Contact support or check out the App Search community forum!