07 — Performance and scaling

You already know how to write queries and aggregations. This chapter is about why some of them destroy a cluster and how to stop that from happening. Every section follows the same discipline: smallest high-impact change first, measure, then go deeper. Sysadmin-level knobs appear last — because in most cases you never need them.

Goals for this chapter

By the end, you should be able to:

• Choose shard count and size rationally, not by instinct
• Set up an ILM policy that keeps shard sizes stable as data grows, and troubleshoot it when it stalls
• Build, query, and manage data streams — the modern default for append-only time-series data
• Explain which caches actually help and which are traps
• Use _profile and the slow query log to diagnose hot queries
• Recognise the most common anti-patterns and fix them without touching cluster settings
• Build composable index templates that prevent schema drift across index generations

7.1 Shard strategy

What a shard actually is

A shard is a Lucene index. When Elasticsearch fans out a query, every shard touched runs a full query cycle — parse, score, collect, reduce. Each shard also consumes:

• Heap: segment metadata, filter caches, field mappings (~few KB to tens of KB per shard, but it adds up fast)
• CPU: query execution, merges, refreshes, aggregation reduction
• Disk IO: segment merges, recovery, relocation
• Cluster coordination overhead: allocation decisions, cluster state updates, rebalancing

The implication: shards are not free. The question is never “how many shards can we have?” but “how many shards keep the cluster stable at peak load and during a node failure?”

Sizing heuristics

These are starting heuristics, not rules. The right size is the one that keeps recovery time, merge pressure, and query fan-out latency within your SLOs.

number_of_shards is immutable after creation

This is the most important constraint. You set it once at index creation. Changing it requires creating a new index and reindexing. This is why you need a strategy before you create the index, and why you use aliases and rollover so you can iterate over time.

number_of_replicas, by contrast, can be changed at any time:

PUT shop-orders-demo/_settings
{
  "number_of_replicas": 2
}

More replicas = more read throughput + higher HA. Each replica is an additional copy of every primary shard that can serve search traffic.

Over-sharding vs under-sharding

Over-sharding symptoms (too many shards, most too small):

• Heap pressure from segment and mapping metadata
• Slower cluster state updates (every shard is tracked)
• Query fan-out overhead dominates actual query work (coordinating node collects results from hundreds of shards for a simple filter)
• Many shards under ~1 GB (overhead exceeds value)

Common cause: creating daily indices with a fixed shard count that was sized for peak, when most days are much smaller.

Under-sharding symptoms (too few shards, each too large):

• Indexing bottleneck: writes can only parallelize across primaries, so a single-primary index can only absorb so much throughput
• Hot nodes: one shard living on one node carries all the load
• Slow recovery after a node failure: a 200 GB shard takes a long time to restore
• Heavy aggregation slowness: Lucene must scan a very large segment set

How to reshard

_split — increases primary count (e.g., 1 → 2). The target shard count must be a multiple of the source. Index must be read-only first. Creates a new index. Rarely the right tool because you still need to plan the original shard count to be splittable.

POST shop-orders-demo/_split/shop-orders-demo-v2
{
  "settings": {
    "index.number_of_shards": 4
  }
}

_shrink — decreases primary count (e.g., 4 → 2). Index must be read-only, all shards must be on the same node. Useful in ILM warm/cold phases.

POST shop-orders-demo/_shrink/shop-orders-demo-small
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1
  }
}

_reindex into a new index (the general safe path) — works in all cases, allows mapping changes alongside the reshard, is restartable, and does not require read-only state. This is the path you will use most often in production:

POST _reindex?wait_for_completion=false
{
  "source": { "index": "shop-orders-demo" },
  "dest":   { "index": "shop-orders-demo-v2" }
}

The safe-change workflow (always use aliases)

The pattern for any structural change — reshard, mapping change, analyzer change:

1. Create new index  (shop-orders-demo-v2)  with correct settings
2. Reindex from old index to new index
3. Validate (count, sample queries, key aggregations)
4. Atomically flip alias: remove old, add new

POST _aliases
{
  "actions": [
    { "remove": { "index": "shop-orders-demo-v1", "alias": "shop-orders" } },
    { "add":    { "index": "shop-orders-demo-v2", "alias": "shop-orders" } }
  ]
}

5. Keep old index as rollback buffer (e.g., 24h), then delete

Your application always queries the alias shop-orders. It never knows that the physical index changed.

Pitfall: If you skip aliases and hard-code index names into your application, every structural change becomes a coordinated deploy. Aliases decouple the physical layout from the logical name.

Read scaling vs HA with replicas

• number_of_replicas: 0 — no redundancy; a node failure loses that shard’s data until recovery. Only acceptable in bulk-ingest-then-restore scenarios.
• number_of_replicas: 1 — the production baseline for most indices; survives one node failure per shard.
• number_of_replicas: 2+ — higher read throughput, can survive multiple concurrent failures; costs proportionally more storage.

Replicas are not a substitute for shard count when you have write throughput bottlenecks — replicas do not increase write parallelism.

7.2 Index Lifecycle Management (ILM)

Why ILM matters to you as a developer

If you build a feature that generates append-only data (events, audit logs, job executions, webhook payloads), you have a choice: either manage index lifecycle manually or automate it. Manual management means: someone eventually forgets the cleanup script runs, a disk fills up, Elasticsearch puts indices into read-only mode, and your feature breaks at 3am.

ILM is the answer. It automates rollover (keeps shard sizes stable) and deletion (enforces retention).

For entity data (shop-orders-demo style — mutable objects with updates), you typically do NOT use ILM rollover. ILM is primarily for append-only time-series workloads where documents flow in continuously and are never updated.

Entity data vs time-series data

Calendar indices vs rollover indices

There are two ways to slice append-only data into multiple indices. ILM is built around the second.

Calendar indices — one index per time bucket, named by date (logs-2026.05.30, logs-2026.05.31):

• ✅ Simple mental model; “delete by day” is trivial.
• ❌ Shard sizes drift as ingest volume changes — a quiet day produces a tiny shard, a spike produces a huge one. You are guessing the daily shard count up front.

Rollover indices — sequentially numbered (logs-000001, logs-000002), with a write alias that ILM advances when a size/age/doc threshold is hit:

• ✅ Shard sizes stay near a target regardless of ingest rate — this is the whole point.
• ✅ Predictable recovery and query performance; works natively with ILM and data streams.
• ❌ Names are less intuitive; requires an alias + template to wire up correctly (or a data stream, which hides all of it — see 7.3).

Rule of thumb: if ingest volume varies at all, or you care about stable shard sizes, use rollover (or a data stream). Reach for calendar indices only when “delete exactly day X” is a hard business requirement.

The hot/warm/cold/delete phases

For most teams: start with hot + delete. Add warm/cold only once you have measured the cost/benefit.

Rollover conditions

ILM checks rollover conditions at intervals (default: every 10 minutes). Rollover triggers when any condition is met:

• max_primary_shard_size — triggers when the largest primary shard in the index exceeds the threshold. Best control for keeping shard sizes on target.
• max_age — triggers after the index reaches a certain age. Useful as a safety net.
• max_docs — triggers after a document count threshold. Useful for evenly-sized batches.

A minimal working ILM policy

PUT _ilm/policy/logs-hot-delete
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This policy rolls over when a shard hits 30 GB or the index is 1 day old (whichever comes first), and deletes indices 30 days after rollover.

Wiring ILM via index template

The policy alone does nothing until it is wired to new indices via a template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-hot-delete",
      "index.lifecycle.rollover_alias": "logs-write"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "service":    { "type": "keyword" },
        "level":      { "type": "keyword" },
        "message": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        }
      }
    }
  },
  "priority": 200
}

Bootstrap the first generation index with the write alias:

PUT logs-000001
{
  "aliases": {
    "logs-write": { "is_write_index": true },
    "logs-read":  {}
  }
}

Write to logs-write. Query from logs-read (which spans all generations). ILM rolls over automatically and creates logs-000002, logs-000003, etc.

Data streams are the modern alternative to the manual write-alias + rollover wiring shown above — they hide the bootstrap index, the write alias, and the generation naming behind a single name. Because they matter so much for the time-series features this audience builds, they get a dedicated deep dive in section 7.3 below. The ILM policy you just wrote is reused there unchanged.

Verifying ILM

GET logs-000001/_ilm/explain
GET _ilm/policy/logs-hot-delete

Pitfall: The most common ILM failure is a misconfigured rollover alias. Check _ilm/explain — it shows the current phase, action, and any error message. If rollover never triggers, check that index.lifecycle.rollover_alias matches the alias name, and that exactly one index has is_write_index: true.

Lifecycle actions beyond delete

rollover and delete cover most needs. The other actions exist to trade search speed and cost as data ages — apply them in warm/cold phases, never in hot during peak traffic.

PUT _ilm/policy/logs-tiered
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "30gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "shrink": { "number_of_shards": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

If you are new to ILM, start with rollover + delete only. Add forcemerge/shrink/tiers later, one at a time, and measure the effect. forcemerge in particular can saturate disk IO if triggered on the hot tier.

Common ILM failures and how to spot them

GET <index>/_ilm/explain is your first stop — it reports the current phase, action, step, and a step_info with the error if a step is stuck.

GET logs-000001/_ilm/explain
POST logs-000001/_ilm/retry

Pitfall: min_age in the delete phase is measured from when the index rolled over, not from the timestamps inside the documents. If you backfill old data into a fresh index, it will not be deleted “on schedule” the way you might expect.

7.3 Data streams in depth

Data streams are the modern, opinionated way to manage append-only time-series data (logs, metrics, events, traces, audit records, webhook deliveries, job runs). They give you stable shard sizes, automatic rollover, and hands-off retention — without the manual write-alias plumbing from 7.2.

What a data stream is

A data stream is a single logical name you write to and search, backed by a sequence of hidden, automatically-managed indices called backing indices.

• You write documents to the stream name (e.g. logs-app).
• Elasticsearch stores them in the current write backing index.
• When rollover fires (driven by ILM), Elasticsearch creates a new backing index and advances the write pointer automatically.
• Searches against the stream name fan out across all backing indices.

            write  ──────────────►  ┌─────────────────────────────────┐
                                     │  logs-app  (data stream name)   │
            search ◄──────────────►  └─────────────────────────────────┘
                                              │ backed by
                 ┌────────────────────────────┼────────────────────────────┐
                 ▼                            ▼                             ▼
   .ds-logs-app-2026.05.30-000001   .ds-logs-app-2026.05.31-000002   ...-000003  ◄─ write index
        (older, read-only-ish)          (older)                        (current writes)

Anatomy and the @timestamp requirement

A data stream has a name, a set of backing indices (.ds-<name>-<date>-<gen>), exactly one write index (the newest backing index), and an associated data-stream-enabled index template.

The non-negotiable requirement: every document must contain a @timestamp field mapped as date. The stream uses it to organise data and to support time-based queries.

Without @timestamp, a data stream is the wrong tool. If your incoming data uses a different timestamp field, rename it to @timestamp in an ingest pipeline (see chapter 08 — Operations, integration and best practices).

Data streams vs classic alias rollover

Use a data stream when the workload is append-only time-series and you want safe defaults with minimal plumbing. Stick with classic indices/aliases when you frequently update or delete documents by _id (entity data like shop-orders-demo), need custom index naming, or already run a large alias-based system that isn’t worth migrating.

Creating a data stream end-to-end

Step 1 — a data-stream index template. It must set data_stream: {}, match a pattern, map @timestamp, and (normally) attach an ILM policy. The logs-hot-delete policy from 7.2 is reused as-is:

PUT _index_template/logs-app-template
{
  "index_patterns": ["logs-app*"],
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-hot-delete"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "service":    { "type": "keyword" },
        "level":      { "type": "keyword" },
        "message": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        }
      }
    }
  },
  "priority": 200
}

Notice what’s missing compared to 7.2: no rollover_alias, no bootstrap index, no is_write_index. The data stream handles all of it.

Step 2 — create the stream (optional; the first write auto-creates it if the template matches):

PUT _data_stream/logs-app

Step 3 — write to the stream name. Because data streams are append-only, you use create (or op_type=create), not arbitrary PUT _doc/<id>:

POST logs-app/_doc
{
  "@timestamp": "2026-05-30T10:11:12Z",
  "service": "api",
  "level": "error",
  "message": "timeout talking to upstream"
}

The first write creates .ds-logs-app-2026.05.30-000001.

Inspecting backing indices

GET _data_stream/logs-app          # stream details: generation, write index, ILM status
GET _data_stream                    # list all streams
GET _cat/indices/.ds-logs-app*?v    # see the backing indices
GET .ds-logs-app*/_mapping          # mappings actually applied to backing indices

Rollover and retention

Rollover is normally driven by the ILM policy attached via the template, exactly like 7.2 — but you never touch an alias. To validate the wiring without waiting, trigger a manual rollover:

POST logs-app/_rollover

Then confirm a new backing index (...-000002) exists, that writes now land in it, and that searches still span both generations. Retention works through the ILM delete phase against the backing indices.

Newer stacks also offer Data Stream Lifecycle (DSL) — a built-in retention mechanism configured directly on the stream (lifecycle.data_retention) instead of via ILM. If your cluster has standardised on DSL, use it; otherwise ILM remains the broadly-compatible choice. Don’t mix both on the same stream.

Querying data streams

Query the stream name; everything from chapters 03, 05, and 06 works unchanged — Elasticsearch fans out across backing indices:

GET logs-app/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "service": "api" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ]
    }
  }
}

Always include a @timestamp range filter. Data streams are designed to be queried by time window, and the filter lets Elasticsearch skip whole backing indices (shard pruning).

Updating, deleting, and mapping evolution

• Append-only by design. Direct update/delete by _id against the stream name is blocked. You can update/delete via _update_by_query / _delete_by_query (which target the backing indices), but if you need frequent per-_id updates, you picked the wrong model — use classic indices.
• Mappings evolve forward only. A template change applies to new backing indices; existing ones keep their old mappings. Adding fields is safe. Changing a field’s type across generations causes search/aggregation conflicts — avoid it; if you must, migrate to a new stream (or reindex).

Migrating classic rollover → data streams (high level)

1. Create a data-stream template (data_stream: {}) for a new stream name.
2. Start writing new data to the stream.
3. Optionally reindex historical indices into the stream (or keep them queryable separately under a shared pattern).
4. Point dashboards/consumers at the stream name.
5. Decommission old indices after a validation + retention window.

Common data stream foot-guns

7.4 Caching

Elasticsearch has three distinct caches relevant to query performance. Understanding what each one caches — and what it does not — determines whether optimising for cache hits is even possible.

Shard request cache

What it caches: the aggregation results (and hits.total) for queries with size: 0, per shard. If the exact same query hits the same shard and the shard has not been refreshed since the last time, the cached result is returned without executing the query.

Controlled by: the request_cache parameter on the request (defaults to true for size: 0 queries). You can opt in for queries with size > 0, but the cache only stores the complete shard-level response — use this carefully.

When it helps: dashboards and monitoring queries that run the same aggregation repeatedly on stable (or slowly-changing) data. Common in Kibana visualizations with fixed time ranges.

When it does not help: queries with now in the time range (Elasticsearch normalises now to a minute-level bucket, so short-range now-15m queries can still hit cache, but now down to milliseconds disables it).

GET logs-read/_search?request_cache=true
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-1h/h", "lte": "now/h" } }
  },
  "aggs": {
    "errors_by_service": {
      "terms": { "field": "service", "size": 20 }
    }
  }
}

Inspect cache stats:

GET logs-read/_stats/request_cache

Query cache (node-level filter cache)

What it caches: the bitset of matching document IDs for individual filter clauses — not scores, not full results. Reused across queries that contain the same filter on the same shard.

Why filter context, not must: filters do not compute relevance scores, so Elasticsearch can cache the result as a simple bitset and reuse it. Queries in must context compute scores and are not cached this way. This is a direct reason to move exact-match and range constraints into bool.filter rather than bool.must — performance and cacheability both improve.

GET shop-orders-demo/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "description": "premium" } }
      ],
      "filter": [
        { "term":  { "status": "completed" } },
        { "range": { "created_at": { "gte": "2024-01-01" } } }
      ]
    }
  }
}

The status and created_at filters are cached. The match on description is not. See chapter 03 for filter vs query context in detail.

Inspect:

GET _nodes/stats/indices/query_cache

Fielddata cache

What it caches: when a text field is used in an aggregation or sort, Elasticsearch must load all field values into the JVM heap (uninverted index). This is the fielddata cache.

The danger: fielddata for text fields can be enormous — it stores every token from every analyzed document. On a large index, this can consume gigabytes of heap, trigger circuit breakers, or cause OOM errors.

The fix: never aggregate or sort on text fields. Map fields that need both full-text search and aggregation as text with a keyword subfield:

PUT shop-orders-demo/_mapping
{
  "properties": {
    "customer_name": {
      "type": "text",
      "fields": {
        "keyword": { "type": "keyword" }
      }
    }
  }
}

Then aggregate on customer_name.keyword, not on customer_name. The keyword subfield uses doc_values (on-disk columnar storage), not fielddata — it does not consume heap.

Inspect fielddata:

GET _nodes/stats/indices/fielddata

If you see evictions climbing, fielddata is under pressure.

Summary of what actually helps: Put constraints in filter context (query cache). Use size: 0 aggregations on stable time windows (request cache). Map aggregation fields as keyword with doc_values (avoids fielddata entirely). These three changes cover 80% of practical cache wins.

7.5 Query profiling

When a query is slow and you need to understand why, Elasticsearch provides two tools: the _profile API and the slow query log.

The _profile API

Add "profile": true to any search request:

GET shop-orders-demo/_search
{
  "profile": true,
  "query": {
    "bool": {
      "filter": [
        { "term": { "status": "completed" } },
        { "range": { "created_at": { "gte": "2024-01-01" } } }
      ],
      "must": [
        { "match": { "description": "premium shipping" } }
      ]
    }
  },
  "aggs": {
    "by_status": {
      "terms": { "field": "status", "size": 10 }
    }
  }
}

The response includes a profile section per shard. Each shard profile contains:

• searches — the query tree, broken down by clause. Each clause shows time_in_nanos, description (the Lucene query type), and breakdown (how time was split across build_scorer, next_doc, advance, etc.)
• aggregations — time spent in each aggregation, per shard
• collector — how results were collected (top-hits, global aggs, etc.)

What to look for:

Profile results are per shard — if you hit 20 shards, you get 20 profile trees. Look for the slowest shard, not just the average.

Search Profiler in Kibana Dev Tools

In Kibana, go to Dev Tools → Search Profiler (the tab next to Console). Paste your query and index pattern, run it. The UI renders the shard tree visually, with time bars for each clause — much easier to read than raw JSON for complex queries.

Use the Search Profiler in development and staging. In production, "profile": true adds overhead — use it on targeted diagnostic queries, not as a permanent flag.

Slow query log

The slow log catches offenders in production without modifying query code. Configure thresholds on an index:

PUT shop-orders-demo/_settings
{
  "index.search.slowlog.threshold.query.warn":  "5s",
  "index.search.slowlog.threshold.query.info":  "2s",
  "index.search.slowlog.threshold.query.debug": "500ms",
  "index.search.slowlog.threshold.fetch.warn":  "1s"
}

Any search that exceeds a threshold is logged to the Elasticsearch slow log file (and, if you ship logs to Kibana, visible there). The log entry includes the full query JSON, the shard it ran on, the took time, and the number of hits.

Strategy:

1. Start with high thresholds (e.g., warn at 5s) to avoid log volume.
2. Look for patterns: wildcard queries, missing time filters, large aggregations on high-cardinality fields.
3. Fix the patterns (see 7.5). Lower thresholds once the worst offenders are resolved.

To disable when done:

PUT shop-orders-demo/_settings
{
  "index.search.slowlog.threshold.query.warn": "-1"
}

Pitfall: Do not leave low thresholds (e.g., debug: 10ms) enabled permanently in production. The log volume causes its own IO pressure.

7.6 Common performance anti-patterns

Each entry: symptom → root cause → fix.

No time filter on large time-series indices

Symptom: A query on logs-* or events-* takes 30+ seconds; CPU spikes across all data nodes.

Cause: Every shard of every index in the pattern is searched. On 30 days of data with 10 shards/day, that is 300 shards touched for a query that only needed the last 15 minutes.

Fix: Always scope time-series queries with a @timestamp range filter. Elasticsearch uses shard-level min/max timestamp metadata to skip entire shards (shard pruning), but only if you provide the filter.

{ "range": { "@timestamp": { "gte": "now-15m" } } }

High-cardinality terms aggregations

Symptom: A terms agg with size: 10000 on user_id or request_id is slow and uses a lot of heap.

Cause: Elasticsearch must collect and merge up to size * number_of_shards buckets in the coordinating node. On high-cardinality fields (millions of unique values), this is massive.

Fix: For true pagination over all buckets, use composite aggregation with a cursor:

GET logs-read/_search
{
  "size": 0,
  "aggs": {
    "by_user": {
      "composite": {
        "size": 1000,
        "sources": [
          { "user_id": { "terms": { "field": "user_id" } } }
        ],
        "after": { "user_id": "last-seen-user-id" }
      }
    }
  }
}

Iterate using the after_key from each response. See chapter 06 for composite aggregations.

Deep from/size pagination

Symptom: Paginating to page 500 with from: 4990, size: 10 is slow and grows slower with each page.

Cause: Elasticsearch must collect and rank from + size documents on every shard, then merge. At from: 10000, that is 10000 docs pulled from each shard before discarding all but 10.

Fix: Use search_after with a Point in Time (PIT) for deep pagination. PIT freezes the index view at a moment in time; search_after uses the sort value of the last-seen document as a cursor:

POST shop-orders/_pit?keep_alive=1m

Then paginate:

GET shop-orders/_search
{
  "size": 10,
  "sort": [{ "created_at": "desc" }, { "_id": "asc" }],
  "pit": { "id": "<pit-id>", "keep_alive": "1m" },
  "search_after": ["2024-03-15T12:00:00Z", "order-abc"]
}

Wildcard and leading-wildcard queries on large fields

Symptom: A wildcard query like "status": "*omplete*" or a regexp is slow even with modest result counts.

Cause: Wildcard and regex queries with a leading wildcard cannot use the inverted index efficiently. They scan the term dictionary for matching terms. On high-cardinality fields or long text, this is expensive.

Fixes: - Use prefix queries (no leading wildcard) — they are optimised. - For suffix/infix matching, consider edge_ngram or ngram tokenizers at index time. - Normalise data so you can query exact keyword values instead. - Never use wildcard/regexp on text fields in a hot path.

Aggregating or sorting on text fields

Symptom: Aggregation fails with "Fielddata is disabled on text fields by default", or if fielddata is enabled, heap usage climbs.

Cause: text fields are analyzed into tokens. Aggregation requires the original (uninverted) value, which text fields do not store in doc_values.

Fix: Add a keyword subfield (see section 7.4 fielddata). Aggregate on field.keyword.

Scripts in hot paths

Symptom: A script_score or script aggregation is slow; CPU is high on data nodes.

Cause: Scripts run interpreted (Painless compiles to Java bytecode, but still slower than native aggregation). Applied per document, per shard, per query.

Fixes: - Precompute the value at index time and store it as a numeric field. - Use function_score with built-in decay functions instead of custom scripts where possible. - Cache script results where the input is a small discrete set (e.g., tier enumeration).

Mapping/field explosion from dynamic mapping

Symptom: Cluster state updates are slow; GET _cluster/state returns a very large response; heap pressure is unexplained.

Cause: Dynamic mapping is enabled by default. If your documents have unpredictable keys (JSON metadata blobs, per-tenant dynamic fields, arbitrary event properties), Elasticsearch creates a new field mapping entry for every unique key it sees. Thousands of unique field names → large cluster state → heap pressure on master nodes.

Fixes: - Set "dynamic": "strict" and define fields explicitly. - Set "dynamic": "false" on sub-objects where you want values stored but not indexed. - Use the flattened field type for metadata blobs you might want to search but don’t need per-field analysis on.

Returning huge _source

Symptom: Fetching 100 documents takes much longer than expected; network throughput is high; fetch phase in _profile is large.

Cause: _source stores the original document JSON. If each document is 50KB and you return 1000 of them, that is 50MB of payload per query.

Fixes: - Use _source: ["field1", "field2"] filtering to return only what you need. - Or use "fields": ["field1", "field2"] to read from doc_values (bypasses _source decompression for supported types). - Consider whether you need _source stored at all — for purely aggregation-driven use cases, disabling _source saves disk, though it makes reindexing from Elasticsearch itself impossible.

GET shop-orders-demo/_search
{
  "_source": ["order_id", "status", "total_amount"],
  "query": { "term": { "status": "pending" } }
}

Too many / too few shards (see 7.1 in detail)

Symptom (too many): Heap pressure, slow cluster state, query fan-out overhead even for simple queries.

Symptom (too few): Indexing bottleneck, hot node, slow recovery.

Fix: Use ILM rollover with max_primary_shard_size to keep sizes in range. Don’t create N shards upfront “just in case”.

“Just add heap” and “just restart” myths

“Add more heap”: Elasticsearch recommends no more than 30–32 GB heap (above that, JVM pointers expand and you lose compressed oops, which makes things worse). More importantly, heap you give to Elasticsearch is heap you take away from the OS filesystem cache — which Lucene depends on heavily. Adding heap past 30 GB often makes performance worse.

“Just restart”: A restart clears caches and resets some in-memory state. It feels faster immediately because warm-up queries benefit from cache. The root cause is unchanged and the problem returns within minutes. Fix the query/mapping/shard issue, not the process.

7.7 Index templates and component templates

Why templates matter to developers

Every time you roll over to a new index generation (via ILM or manually), a fresh index is created. Without a template, that index gets Elasticsearch defaults: dynamic mapping on, no lifecycle policy, default shard count. Within a few weeks you have mapping drift — status is text in some generations and keyword in others; @timestamp is missing on one; queries break across generations.

Templates are the contract that every new index in a pattern will have the same mappings, settings, and aliases.

Composable templates: component templates + index templates

Component template (_component_template): a reusable building block. Defines a fragment of mappings, settings, or aliases. Multiple component templates can be shared across many index templates.

Index template (_index_template): applies to indices matching index_patterns. Lists which component templates to compose via composed_of. Sets priority to resolve conflicts when multiple templates match the same index name.

Example: shared timestamp + metadata component template

PUT _component_template/common-timestamp
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "env":     { "type": "keyword" },
        "service": { "type": "keyword" },
        "version": { "type": "keyword" }
      }
    }
  }
}

A second component template for log-specific fields:

PUT _component_template/log-fields
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "level":   { "type": "keyword" },
        "message": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        },
        "trace_id": { "type": "keyword", "index": false }
      }
    }
  }
}

Compose them into an index template:

PUT _index_template/logs-composable-template
{
  "index_patterns": ["logs-*"],
  "composed_of": ["common-timestamp", "log-fields"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-hot-delete",
      "index.lifecycle.rollover_alias": "logs-write"
    }
  },
  "priority": 200,
  "data_stream": {}
}

priority determines which template wins when multiple templates match the same index name. Higher priority wins. Built-in templates use priority 0–100; use 200+ for your custom templates.

Verifying template application

Check the template definition:

GET _index_template/logs-composable-template

Simulate what mappings a new index would get without actually creating it:

POST _index_template/_simulate_index/logs-2024-test

The response shows the fully merged mappings, settings, and aliases that would apply — invaluable for validating that composed_of is resolving correctly before you create any real indices.

Check which templates would match a given index name:

POST _index_template/_simulate/logs-composable-template

Pitfall: The most common mistake is forgetting to set priority. If multiple templates match logs-* and you don’t set priority, Elasticsearch picks one arbitrarily. Always set explicit priorities and simulate before creating indices in production.

When to use component templates vs inline template settings

• Use component templates for anything shared across multiple index templates: common field definitions (timestamp, env, tenant id), standard settings.
• Put index-specific settings (shard count, ILM policy, rollover alias) directly in the index template’s template block.
• Avoid duplicating the same mapping in multiple index templates — extract it into a component template and compose it.

Labs

Lab 1 — Shard strategy and alias-based reshard

1. Create shop-orders-demo-v1 with 2 primary shards and the alias shop-orders.
2. Index 500 sample documents.
3. Create shop-orders-demo-v2 with 3 primary shards.
4. Reindex from v1 to v2 (_reindex?wait_for_completion=false), monitor with GET _tasks?actions=*reindex&detailed).
5. Atomically flip the alias to v2.
6. Confirm that queries against shop-orders see all data.

Takeaway: resharding is new index → reindex → alias flip. The application never sees the change.

Lab 2 — ILM rollover pipeline end-to-end

1. Create the logs-hot-delete ILM policy (hot rollover at 1 MB for lab speed, delete at 30d).
2. Create the logs-template index template wiring in the policy and rollover alias.
3. Bootstrap logs-000001 with logs-write (write) and logs-read (read) aliases.
4. Index ~20 documents into logs-write.
5. Trigger manual rollover: POST logs-write/_rollover { "conditions": { "max_primary_shard_size": "1mb" } }.
6. Verify logs-000002 was created and the write alias moved.
7. Query logs-read to confirm both generations are visible.

Takeaway: ILM is mostly wiring (policy + template + write alias). Manual rollover validates the wiring instantly without waiting for thresholds.

Lab 3 — Data stream end-to-end

1. Reuse the logs-hot-delete ILM policy from Lab 2.
2. Create a data-stream index template logs-app-template (data_stream: {}, pattern logs-app*, @timestamp mapped, ILM policy attached) — no rollover alias, no bootstrap index.
3. Index a few documents straight into logs-app (each with an @timestamp). Confirm the stream and its first backing index were created: GET _data_stream/logs-app.
4. Trigger POST logs-app/_rollover; confirm .ds-logs-app-...-000002 exists and the write index advanced.
5. Search logs-app with a @timestamp range filter — confirm both generations are visible.
6. Try a direct update by _id against logs-app and observe that it’s rejected (append-only model).

Takeaway: a data stream gives you the whole rollover pipeline from Lab 2 behind one name — no alias bootstrap to manage. The trade-off is the append-only constraint.

Lab 4 — Profiling a slow query

1. Run this query against shop-orders-demo with "profile": true:

   GET shop-orders-demo/_search
   {
     "profile": true,
     "query": {
       "bool": {
         "must": [
           { "wildcard": { "status": "*end*" } }
         ]
       }
     },
     "aggs": { "by_status": { "terms": { "field": "status", "size": 50 } } }
   }

2. Find the clause with the highest time_in_nanos in the profile output.

3. Rewrite it: move status filter to bool.filter, use term instead of wildcard.

4. Re-run and compare timings.

5. Open Kibana Dev Tools → Search Profiler, paste the original query, compare the visual output.

Takeaway: Profile output shows exactly which clause costs the most. Moving exact matches to filter context almost always wins.

Lab 5 — Component templates and simulation

1. Create a common-timestamp component template with @timestamp, env, service.
2. Create a log-fields component template with level, message (text + keyword subfield), trace_id (keyword, not indexed).
3. Create an index template logs-composable that composes both, with priority: 200 and index_patterns: ["logs-*"].
4. Simulate: POST _index_template/_simulate_index/logs-2024-test — confirm the merged mappings are correct.
5. Create index logs-2024-test and verify: GET logs-2024-test/_mapping.
6. Change one field in log-fields component template and re-simulate. Observe that new indices will pick up the change; existing indices are unaffected.

Takeaway: _simulate_index is the “dry run” for templates. Use it before creating real indices in production.

Summary

• A shard is a Lucene index. Each one costs heap, CPU, and cluster overhead. Shard count × shard size × replica factor = total pressure. Right-size before creation, use rollover to stay on target.
• number_of_shards is immutable. Always use aliases so you can change the physical layout without touching application code. The safe-change workflow is always: new index → reindex → alias flip → cleanup.
• ILM automates rollover (stable shard sizes) and deletion (retention) for append-only time-series data — not for mutable entity data. Prefer rollover indices over calendar indices. Start with hot + delete; add forcemerge/shrink/tiers later, one at a time. When something stalls, GET <index>/_ilm/explain tells you the stuck step; min_age counts from rollover time, not document timestamps.
• Data streams are the modern default for new append-only pipelines: you write to and search one name, while Elasticsearch manages backing indices, the write pointer, and rollover behind the scenes. They require @timestamp, are append-only (no per-_id updates), and reuse the same ILM policy. Mappings evolve forward only — never change a field’s type across generations.
• Caching: filter context → query cache (cacheable bitsets). size: 0 aggregations → request cache. keyword with doc_values → avoids fielddata entirely. These three changes cover most practical cache wins.
• Profiling: use _profile to find which query clause dominates. Use the Kibana Search Profiler for readability. Use the slow query log in production to catch patterns without modifying query code.
• Anti-patterns in priority order: no time filter on big indices; wildcard/leading-wildcard on large fields; terms aggs on high-cardinality fields (use composite); deep from/size (use search_after + PIT); aggregating on text (add keyword subfield); mapping explosion from dynamic mapping; oversized _source responses. Fix these before touching any cluster settings.
• Component templates + index templates prevent schema drift. Use _simulate_index before creating real indices. Set explicit priorities.