07b — ILM and data streams

If you build a feature that generates append-only data (events, audit logs, job runs, webhook payloads), you have a choice: manage index lifecycle manually or automate it. Manual management means someone eventually forgets, a disk fills, and your feature breaks at 3am. This part covers ILM and data streams — the two tools that prevent that from happening.

Goals for this part

• Understand when ILM is appropriate (append-only time-series) vs when it is not (entity data)
• Write a working ILM policy with rollover and delete phases
• Wire the policy to indices via an index template and a write alias
• Create and operate a data stream as the modern alternative

7.2 Index Lifecycle Management (ILM)

Why ILM matters to you as a developer

If you build a feature that generates append-only data (events, audit logs, job executions, webhook payloads), you have a choice: either manage index lifecycle manually or automate it. Manual management means: someone eventually forgets the cleanup script runs, a disk fills up, Elasticsearch puts indices into read-only mode, and your feature breaks at 3am.

ILM is the answer. It automates rollover (keeps shard sizes stable) and deletion (enforces retention).

For entity data (shop-orders-demo style — mutable objects with updates), you typically do NOT use ILM rollover. ILM is primarily for append-only time-series workloads where documents flow in continuously and are never updated.

Entity data vs time-series data

Calendar indices vs rollover indices

There are two ways to slice append-only data into multiple indices. ILM is built around the second.

Calendar indices — one index per time bucket, named by date (logs-2026.05.30, logs-2026.05.31):

• ✅ Simple mental model; “delete by day” is trivial.
• ❌ Shard sizes drift as ingest volume changes — a quiet day produces a tiny shard, a spike produces a huge one.

Rollover indices — sequentially numbered (logs-000001, logs-000002), with a write alias that ILM advances when a size/age/doc threshold is hit:

• ✅ Shard sizes stay near a target regardless of ingest rate — this is the whole point.
• ✅ Predictable recovery and query performance; works natively with ILM and data streams.
• ❌ Names are less intuitive; requires an alias + template to wire up correctly.

Rule of thumb: if ingest volume varies at all, or you care about stable shard sizes, use rollover (or a data stream). Reach for calendar indices only when “delete exactly day X” is a hard business requirement.

The hot/warm/cold/delete phases

For most teams: start with hot + delete. Add warm/cold only once you have measured the cost/benefit.

Rollover conditions

ILM checks rollover conditions at intervals (default: every 10 minutes). Rollover triggers when any condition is met:

• max_primary_shard_size — triggers when the largest primary shard exceeds the threshold. Best control for keeping shard sizes on target.
• max_age — triggers after the index reaches a certain age.
• max_docs — triggers after a document count threshold.

A minimal working ILM policy

PUT _ilm/policy/logs-hot-delete
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This policy rolls over when a shard hits 30 GB or the index is 1 day old (whichever comes first), and deletes indices 30 days after rollover.

Wiring ILM via index template

The policy alone does nothing until it is wired to new indices via a template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-hot-delete",
      "index.lifecycle.rollover_alias": "logs-write"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "service":    { "type": "keyword" },
        "level":      { "type": "keyword" },
        "message": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        }
      }
    }
  },
  "priority": 200
}

Bootstrap the first generation index with the write alias:

PUT logs-000001
{
  "aliases": {
    "logs-write": { "is_write_index": true },
    "logs-read":  {}
  }
}

Write to logs-write. Query from logs-read (which spans all generations). ILM rolls over automatically and creates logs-000002, logs-000003, etc.

Verifying ILM

GET logs-000001/_ilm/explain
GET _ilm/policy/logs-hot-delete

Pitfall: The most common ILM failure is a misconfigured rollover alias. Check _ilm/explain — it shows the current phase, action, and any error message. If rollover never triggers, check that index.lifecycle.rollover_alias matches the alias name, and that exactly one index has is_write_index: true.

Common ILM failures and how to spot them

GET <index>/_ilm/explain is your first stop.

GET logs-000001/_ilm/explain
POST logs-000001/_ilm/retry

Pitfall: min_age in the delete phase is measured from when the index rolled over, not from the timestamps inside the documents.

7.3 Data streams in depth

Data streams are the modern, opinionated way to manage append-only time-series data. They give you stable shard sizes, automatic rollover, and hands-off retention — without the manual write-alias plumbing from 7.2.

What a data stream is

A data stream is a single logical name you write to and search, backed by a sequence of hidden, automatically-managed indices called backing indices.

• You write documents to the stream name (e.g. logs-app).
• Elasticsearch stores them in the current write backing index.
• When rollover fires (driven by ILM), Elasticsearch creates a new backing index and advances the write pointer automatically.
• Searches against the stream name fan out across all backing indices.

            write  ──────────────►  ┌─────────────────────────────────┐
                                     │  logs-app  (data stream name)   │
            search ◄──────────────►  └─────────────────────────────────┘
                                              │ backed by
                 ┌────────────────────────────┼────────────────────────────┐
                 ▼                            ▼                             ▼
   .ds-logs-app-2026.05.30-000001   .ds-logs-app-2026.05.31-000002   ...-000003  ◄─ write index
        (older, read-only-ish)          (older)                        (current writes)

Anatomy and the @timestamp requirement

The non-negotiable requirement: every document must contain a @timestamp field mapped as date. The stream uses it to organise data and to support time-based queries.

Without @timestamp, a data stream is the wrong tool.

Data streams vs classic alias rollover

Creating a data stream end-to-end

Step 1 — a data-stream index template:

PUT _index_template/logs-app-template
{
  "index_patterns": ["logs-app*"],
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-hot-delete"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "service":    { "type": "keyword" },
        "level":      { "type": "keyword" },
        "message": {
          "type": "text",
          "fields": { "raw": { "type": "keyword" } }
        }
      }
    }
  },
  "priority": 200
}

Notice what’s missing compared to 7.2: no rollover_alias, no bootstrap index, no is_write_index. The data stream handles all of it.

Step 2 — write to the stream name:

POST logs-app/_doc
{
  "@timestamp": "2026-05-30T10:11:12Z",
  "service": "api",
  "level": "error",
  "message": "timeout talking to upstream"
}

The first write creates .ds-logs-app-2026.05.30-000001.

Inspecting backing indices

GET _data_stream/logs-app
GET _cat/indices/.ds-logs-app*?v

Common data stream foot-guns

Lab — ILM pipeline and data streams

Part A: Classic ILM rollover pipeline

Goal: Wire an ILM policy from scratch and trigger rollover manually to see it work.

Step 1 — Create an ILM policy. For lab speed, use a very small shard size threshold (1 MB):

PUT _ilm/policy/lab-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "1mb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Step 2 — Create an index template that wires in the policy and defines the rollover alias:

PUT _index_template/lab-logs-template
{
  "index_patterns": ["lab-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "lab-logs-policy",
      "index.lifecycle.rollover_alias": "lab-logs-write"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "service":    { "type": "keyword" },
        "level":      { "type": "keyword" },
        "message":    { "type": "text" }
      }
    }
  },
  "priority": 200
}

Step 3 — Bootstrap the first generation index with both aliases:

PUT lab-logs-000001
{
  "aliases": {
    "lab-logs-write": { "is_write_index": true },
    "lab-logs-read":  {}
  }
}

Step 4 — Index ~15 documents via the write alias:

POST lab-logs-write/_bulk
{ "index": {} }
{ "@timestamp": "2026-05-30T09:00:00Z", "service": "auth", "level": "info", "message": "User login successful" }
{ "index": {} }
{ "@timestamp": "2026-05-30T09:01:00Z", "service": "api", "level": "error", "message": "Upstream timeout" }
{ "index": {} }
{ "@timestamp": "2026-05-30T09:02:00Z", "service": "auth", "level": "warn", "message": "Failed login attempt" }
{ "index": {} }
{ "@timestamp": "2026-05-30T09:03:00Z", "service": "worker", "level": "info", "message": "Job completed" }
{ "index": {} }
{ "@timestamp": "2026-05-30T09:04:00Z", "service": "api", "level": "info", "message": "Request processed" }

Step 5 — Manually trigger rollover (don’t wait for the 10-minute ILM check):

POST lab-logs-write/_rollover
{
  "conditions": {
    "max_primary_shard_size": "1mb"
  }
}

Verify lab-logs-000002 was created and the write alias moved to it:

GET /_cat/aliases/lab-logs-*?v
GET /_cat/indices/lab-logs-*?v

Step 6 — Index a few more documents (they should land in lab-logs-000002):

POST lab-logs-write/_doc
{ "@timestamp": "2026-05-30T10:00:00Z", "service": "api", "level": "info", "message": "Post-rollover message" }

Step 7 — Query via lab-logs-read — both generations should be visible:

GET lab-logs-read/_search
{
  "query": { "match_all": {} },
  "sort": [{ "@timestamp": "asc" }],
  "size": 20
}

Step 8 — Check ILM status of both indices:

GET lab-logs-000001/_ilm/explain
GET lab-logs-000002/_ilm/explain

Part B: Data stream

Goal: Achieve the same result with fewer moving parts.

Step 9 — Reuse the lab-logs-policy ILM policy. Create a data-stream template for a new pattern:

PUT _index_template/lab-stream-template
{
  "index_patterns": ["lab-stream*"],
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "lab-logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "service":    { "type": "keyword" },
        "level":      { "type": "keyword" },
        "message":    { "type": "text" }
      }
    }
  },
  "priority": 200
}

Step 10 — Index a document; the stream and its first backing index are created automatically:

POST lab-stream/_doc
{ "@timestamp": "2026-05-30T10:00:00Z", "service": "api", "level": "info", "message": "Data stream start" }

GET _data_stream/lab-stream
GET _cat/indices/.ds-lab-stream*?v

Step 11 — Manually roll over:

POST lab-stream/_rollover

Confirm a second backing index was created.

Step 12 — Try a direct update by _id and observe it is rejected:

POST lab-stream/_update/some-fake-id
{
  "doc": { "level": "warn" }
}

Step 13 — Clean up:

DELETE /_index_template/lab-logs-template
DELETE /_index_template/lab-stream-template
DELETE /_ilm/policy/lab-logs-policy
DELETE /lab-logs-000001
DELETE /lab-logs-000002
DELETE /_data_stream/lab-stream

Takeaway: ILM is mostly wiring (policy + template + write alias). Manual rollover validates the wiring instantly without waiting for thresholds. Data streams give you the whole pipeline behind one name — no alias bootstrap to manage. The trade-off is the append-only constraint: direct per-_id updates and deletes are blocked.