07d — Performance anti-patterns and templates

Most Elasticsearch performance problems fall into a small set of recognizable patterns — each with a clear fix. And most schema drift problems are preventable with composable index templates. This part covers both: the anti-patterns to eliminate first, and the template system that keeps new indices consistent.

Goals for this part

7.6 Common performance anti-patterns

Each entry: symptom → root cause → fix.

No time filter on large time-series indices

Symptom: A query on logs-* or events-* takes 30+ seconds; CPU spikes across all data nodes.

Cause: Every shard of every index in the pattern is searched. On 30 days of data with 10 shards/day, that is 300 shards touched for a query that only needed the last 15 minutes.

Fix: Always scope time-series queries with a @timestamp range filter. Elasticsearch uses shard-level min/max timestamp metadata to skip entire shards (shard pruning), but only if you provide the filter.

{ "range": { "@timestamp": { "gte": "now-15m" } } }

High-cardinality terms aggregations

Symptom: A terms agg with size: 10000 on user_id or request_id is slow and uses a lot of heap.

Cause: Elasticsearch must collect and merge up to size × number_of_shards buckets in the coordinating node.

Fix: For true pagination over all buckets, use composite aggregation with a cursor:

GET logs-read/_search
{
  "size": 0,
  "aggs": {
    "by_user": {
      "composite": {
        "size": 1000,
        "sources": [
          { "user_id": { "terms": { "field": "user_id" } } }
        ],
        "after": { "user_id": "last-seen-user-id" }
      }
    }
  }
}

Deep from/size pagination

Symptom: Paginating to page 500 with from: 4990, size: 10 is slow and grows slower with each page.

Cause: Elasticsearch must collect and rank from + size documents on every shard, then merge. At from: 10000, that is 10000 docs pulled from each shard before discarding all but 10.

Fix: Use search_after with a Point in Time (PIT) for deep pagination:

POST shop-orders/_pit?keep_alive=1m

Then paginate:

GET shop-orders/_search
{
  "size": 10,
  "sort": [{ "created_at": "desc" }, { "_id": "asc" }],
  "pit": { "id": "<pit-id>", "keep_alive": "1m" },
  "search_after": ["2024-03-15T12:00:00Z", "order-abc"]
}

Wildcard and leading-wildcard queries on large fields

Symptom: A wildcard query like "status": "*omplete*" or a regexp is slow even with modest result counts.

Cause: Wildcard and regex queries with a leading wildcard cannot use the inverted index efficiently. They scan the term dictionary for matching terms.

Fixes: - Use prefix queries (no leading wildcard) — they are optimised. - For suffix/infix matching, consider edge_ngram or ngram tokenizers at index time. - Normalise data so you can query exact keyword values instead. - Never use wildcard/regexp on text fields in a hot path.

Aggregating or sorting on text fields

Symptom: Aggregation fails with "Fielddata is disabled on text fields by default", or if fielddata is enabled, heap usage climbs.

Cause: text fields are analyzed into tokens. Aggregation requires the original (uninverted) value, which text fields do not store in doc_values.

Fix: Add a keyword subfield. Aggregate on field.keyword.

PUT shop-orders-demo/_mapping
{
  "properties": {
    "customer_name": {
      "type": "text",
      "fields": { "keyword": { "type": "keyword" } }
    }
  }
}

Scripts in hot paths

Symptom: A script_score or script aggregation is slow; CPU is high on data nodes.

Cause: Scripts run interpreted (Painless compiles to Java bytecode, but still slower than native aggregation). Applied per document, per shard, per query.

Fixes: - Precompute the value at index time and store it as a numeric field. - Use function_score with built-in decay functions instead of custom scripts where possible.

Mapping/field explosion from dynamic mapping

Symptom: Cluster state updates are slow; GET _cluster/state returns a very large response; heap pressure is unexplained.

Cause: Dynamic mapping is enabled by default. If your documents have unpredictable keys, Elasticsearch creates a new field mapping entry for every unique key it sees. Thousands of unique field names → large cluster state → heap pressure on master nodes.

Fixes: - Set "dynamic": "strict" and define fields explicitly. - Set "dynamic": "false" on sub-objects where you want values stored but not indexed. - Use the flattened field type for metadata blobs you might want to search but don’t need per-field analysis on.

Returning huge _source

Symptom: Fetching 100 documents takes much longer than expected; network throughput is high.

Cause: _source stores the original document JSON. If each document is 50KB and you return 1000 of them, that is 50MB of payload per query.

Fixes: - Use _source: ["field1", "field2"] filtering to return only what you need. - Or use "fields": ["field1", "field2"] to read from doc_values (bypasses _source decompression for supported types).

GET shop-orders-demo/_search
{
  "_source": ["order_id", "status", "total_amount"],
  "query": { "term": { "status": "pending" } }
}

“Just add heap” and “just restart” myths

“Add more heap”: Elasticsearch recommends no more than 30–32 GB heap (above that, JVM pointers expand and you lose compressed oops). More importantly, heap you give to Elasticsearch is heap you take away from the OS filesystem cache — which Lucene depends on heavily. Adding heap past 30 GB often makes performance worse.

“Just restart”: A restart clears caches and resets some in-memory state. It feels faster immediately because warm-up queries benefit from cache. The root cause is unchanged and the problem returns within minutes.

7.7 Index templates and component templates

Why templates matter to developers

Every time you roll over to a new index generation (via ILM or manually), a fresh index is created. Without a template, that index gets Elasticsearch defaults: dynamic mapping on, no lifecycle policy, default shard count. Within a few weeks you have mapping drift — status is text in some generations and keyword in others; @timestamp is missing on one; queries break across generations.

Templates are the contract that every new index in a pattern will have the same mappings, settings, and aliases.

Composable templates: component templates + index templates

Component template (_component_template): a reusable building block. Defines a fragment of mappings, settings, or aliases. Multiple component templates can be shared across many index templates.

Index template (_index_template): applies to indices matching index_patterns. Lists which component templates to compose via composed_of. Sets priority to resolve conflicts when multiple templates match the same index name.

Example: shared timestamp + metadata component template

PUT _component_template/common-timestamp
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "env":     { "type": "keyword" },
        "service": { "type": "keyword" },
        "version": { "type": "keyword" }
      }
    }
  }
}

A second component template for log-specific fields:

PUT _component_template/log-fields
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "level":   { "type": "keyword" },
        "message": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        },
        "trace_id": { "type": "keyword", "index": false }
      }
    }
  }
}

Compose them into an index template:

PUT _index_template/logs-composable-template
{
  "index_patterns": ["logs-*"],
  "composed_of": ["common-timestamp", "log-fields"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-hot-delete",
      "index.lifecycle.rollover_alias": "logs-write"
    }
  },
  "priority": 200,
  "data_stream": {}
}

priority determines which template wins when multiple templates match the same index name. Higher priority wins. Built-in templates use priority 0–100; use 200+ for your custom templates.

Verifying template application

Check the template definition:

GET _index_template/logs-composable-template

Simulate what mappings a new index would get without actually creating it:

POST _index_template/_simulate_index/logs-2024-test

The response shows the fully merged mappings, settings, and aliases that would apply — invaluable for validating that composed_of is resolving correctly before you create any real indices.

Check which templates would match a given index name:

POST _index_template/_simulate/logs-composable-template

Pitfall: The most common mistake is forgetting to set priority. If multiple templates match logs-* and you don’t set priority, Elasticsearch picks one arbitrarily. Always set explicit priorities and simulate before creating indices in production.

When to use component templates vs inline template settings

Lab — Anti-patterns and composable templates

Part A: Spot and fix performance anti-patterns

Goal: Identify slow query patterns and apply the correct fix.

Step 1 — Deep pagination with from/size. Run this and note the response time:

GET /shop-orders-demo/_search
{
  "from": 1000,
  "size": 10,
  "sort": [{ "created_at": "desc" }, { "_id": "asc" }],
  "query": { "match_all": {} }
}

Now rewrite it using a PIT + search_after:

POST /shop-orders-demo/_pit?keep_alive=1m

Take the id from the response, then run:

GET /shop-orders-demo/_search
{
  "size": 10,
  "sort": [{ "created_at": "desc" }, { "_id": "asc" }],
  "pit": { "id": "<pit-id-here>", "keep_alive": "1m" }
}

To get the next page, use the sort values from the last hit:

GET /shop-orders-demo/_search
{
  "size": 10,
  "sort": [{ "created_at": "desc" }, { "_id": "asc" }],
  "pit": { "id": "<pit-id-here>", "keep_alive": "1m" },
  "search_after": ["<last-created_at>", "<last-_id>"]
}

Delete the PIT when done:

DELETE /_pit
{ "id": "<pit-id-here>" }

Step 2 — Oversized _source. Profile and compare returning all fields vs a subset:

GET /shop-orders-demo/_search
{
  "profile": true,
  "query": { "term": { "order_status": "completed" } },
  "size": 20
}

Then restrict _source:

GET /shop-orders-demo/_search
{
  "profile": true,
  "_source": ["order_id", "order_status", "total_amount", "customer.name"],
  "query": { "term": { "order_status": "completed" } },
  "size": 20
}

Compare the fetch phase time in the profile. Is there a difference?

Step 3 — Dynamic mapping field explosion. Create an index with dynamic mapping enabled and index documents with random field names:

PUT /dynamic-demo
{
  "settings": { "number_of_replicas": 0 }
}
POST /dynamic-demo/_doc
{
  "user_field_a": "value",
  "user_field_b": "value",
  "user_field_c": 123
}
GET /dynamic-demo/_mapping

Now create a second index with "dynamic": "false" on a metadata sub-object and observe the difference:

PUT /dynamic-strict-demo
{
  "settings": { "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "order_id": { "type": "keyword" },
      "metadata": {
        "type": "object",
        "dynamic": false
      }
    }
  }
}
POST /dynamic-strict-demo/_doc
{
  "order_id": "ord-001",
  "metadata": {
    "arbitrary_key_1": "value",
    "arbitrary_key_2": 42
  }
}
GET /dynamic-strict-demo/_mapping

Is metadata.arbitrary_key_1 added to the mapping? Can you search on it?

DELETE /dynamic-demo
DELETE /dynamic-strict-demo

Part B: Component templates and simulation

Goal: Build a composable template system and verify it with simulation.

Step 4 — Create a common-timestamp component template:

PUT _component_template/lab-common-timestamp
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "env":     { "type": "keyword" },
        "service": { "type": "keyword" },
        "version": { "type": "keyword" }
      }
    }
  }
}

Step 5 — Create a log-fields component template:

PUT _component_template/lab-log-fields
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "level":    { "type": "keyword" },
        "message":  { "type": "text", "fields": { "raw": { "type": "keyword" } } },
        "trace_id": { "type": "keyword", "index": false }
      }
    }
  }
}

Step 6 — Compose them into an index template:

PUT _index_template/lab-logs-composable
{
  "index_patterns": ["lab-logs-*"],
  "composed_of": ["lab-common-timestamp", "lab-log-fields"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "priority": 200
}

Step 7 — Simulate before creating any real index:

POST _index_template/_simulate_index/lab-logs-2024-test

Questions from the simulation output: 1. Are all the expected fields present in the merged mappings? 2. Is dynamic: strict applied (meaning unexpected fields will be rejected)? 3. What settings are applied?

Step 8 — Create the index and verify the mapping matches the simulation:

PUT /lab-logs-2024-test
GET /lab-logs-2024-test/_mapping

Step 9 — Change a field in lab-log-fields component template (e.g., add a duration_ms field):

PUT _component_template/lab-log-fields
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "level":       { "type": "keyword" },
        "message":     { "type": "text", "fields": { "raw": { "type": "keyword" } } },
        "trace_id":    { "type": "keyword", "index": false },
        "duration_ms": { "type": "integer" }
      }
    }
  }
}

Re-simulate. Does the simulation show duration_ms? Check the existing index mapping — is it affected?

POST _index_template/_simulate_index/lab-logs-2024-test
GET /lab-logs-2024-test/_mapping

Step 10 — Clean up:

DELETE /lab-logs-2024-test
DELETE _index_template/lab-logs-composable
DELETE _component_template/lab-common-timestamp
DELETE _component_template/lab-log-fields

Takeaway: Component templates extract shared schema into reusable building blocks. _simulate_index is the dry run — use it every time before creating real indices in production. Existing indices are unaffected by template changes; only new indices pick up the update. Set explicit priority values; if two templates match the same pattern without priorities, the winner is non-deterministic.