08 — Operations, integration and best practices

You already know how to query and structure data in Elasticsearch. This chapter closes the loop: how to keep a cluster healthy, integrate cleanly, and not make the mistakes that cause 2am incidents. Read it as a developer-facing operations literacy guide — enough to diagnose, integrate safely, and know when to escalate.

Goals for this chapter

By the end, you should be able to:

• Read cluster health and diagnose allocation issues from the REST API
• Know which signals to watch and what they predict
• Snapshot, restore, and validate data safely
• Choose the right ingestion path (ingest pipeline vs Logstash vs Beats) and build a working pipeline
• Apply a practical best-practices checklist covering the whole course

8.1 Cluster health — green, yellow, red, and what to do

What each color means

Yellow is a warning, not an emergency. Red requires immediate attention.

Diagnostics — start here

GET /_cluster/health

GET /_cat/health?v

Spot unassigned shards:

GET /_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason

Filter to unassigned only (pipe in terminal):

curl -s "http://localhost:9200/_cat/shards?v" | grep UNASSIGNED

When you see UNASSIGNED, explain one shard:

GET /_cluster/allocation/explain

This returns the single most important diagnostic — the exact reason Elasticsearch cannot place the shard. Read it literally; it tells you which constraint is blocking.

Safe actions per color

Yellow

PUT /shop-orders-demo/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}

Red

Priority order: 1. Identify which indices have unassigned primaries (_cat/shards?v). 2. Run _cluster/allocation/explain — read the exact blocker. 3. Fix the blocker: - Bring back the missing node (primaries lived there, no replica existed). - Free disk / expand if watermark is blocking. - Fix allocation filters or version compatibility if during an upgrade. 4. If data is gone: restore from snapshot or rebuild from source of truth.

Pitfall: do not randomly delete indices or force-allocate shards. Forced allocation can cause data loss if a shard copy is stale. Deleting the wrong index is unrecoverable without a snapshot. Always understand the cause before acting.

The disk watermark chain

Elasticsearch applies three thresholds (defaults, configurable):

• Low (85%) — stops allocating new shards to the node.
• High (90%) — starts moving shards away from the node.
• Flood stage (95%) — sets all indices on that node to read-only.

Check current settings and node allocation:

GET /_cat/allocation?v
GET /_cluster/settings?include_defaults=true&filter_path=**watermark**

To clear a read-only block after creating headroom:

PUT /_all/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Clear the block only after you have actual disk headroom. Clearing it on a still-full disk just re-triggers the block on the next write.

8.2 Monitoring — what to watch and where to look

The _cat family — quick cluster overview

GET /_cat/indices?v&s=store.size:desc

Lists all indices sorted by disk size — useful for spotting unexpectedly large indices.

GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,node.role

GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected

The thread_pool view is high-value: rejections mean Elasticsearch is dropping work.

_nodes/stats and _stats — deeper numbers

GET /_nodes/stats

For a developer, the key sub-sections to check:

Per-index stats:

GET /shop-orders-demo/_stats

Kibana Stack Monitoring

Kibana’s Stack Monitoring UI (sidebar > Management > Stack Monitoring) presents the same data visually:

• Cluster overview: health, node count, shard summary.
• Per-node views: heap %, CPU, disk, indexing/search rates.
• Index-level views: docs, store size, indexing latency.
• Logs and slow logs (if enabled).

Use it for historical trends and to correlate signals across time. Dev Tools _cat APIs are faster for point-in-time triage.

The signals that predict trouble

These five signals cover the vast majority of production incidents:

Every alert should have an associated action — who checks what, in which order. An alert without a runbook is noise. Pair each signal with the relevant _cat or _nodes API call as a first step.

8.3 Snapshots and restore — backup strategies

Snapshots are the only supported backup mechanism

Elasticsearch snapshots are incremental and segment-based: the first snapshot captures all segments; subsequent snapshots only copy new or changed segments. This makes them efficient for daily/hourly schedules.

Snapshots capture: - Index data (shards → segments) - Index metadata (mappings, settings) - Optionally: cluster state (templates, ILM policies, etc.)

Know your source of truth. Elasticsearch is frequently a derived store — data originally lives in a database or event stream and is indexed into ES for search. In that case, a full data rebuild from source may be more appropriate than a restore, especially after mapping changes. Snapshots are still critical for disaster recovery and for data that only exists in ES.

Step 1: Register a repository

File system (useful for local or NFS mounts):

PUT /_snapshot/my-repo
{
  "type": "fs",
  "settings": {
    "location": "/mnt/snapshots"
  }
}

S3-compatible (common in production):

PUT /_snapshot/my-repo
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-snapshots",
    "region": "eu-west-1"
  }
}

Step 2: Take a snapshot

PUT /_snapshot/my-repo/snap-shop-orders-2026-05-30?wait_for_completion=true
{
  "indices": "shop-orders-demo",
  "include_global_state": false
}

wait_for_completion=true blocks until done (fine for ad-hoc; use async for automation).

Step 3: List and verify

GET /_snapshot/my-repo/_all
GET /_snapshot/my-repo/snap-shop-orders-2026-05-30

Check state: SUCCESS and shards.failed: 0.

Step 4: Restore

Restore to a renamed index to avoid clobbering the live index:

POST /_snapshot/my-repo/snap-shop-orders-2026-05-30/_restore
{
  "indices": "shop-orders-demo",
  "include_global_state": false,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1"
}

This restores shop-orders-demo as restored-shop-orders-demo. Validate doc counts and run a couple of queries before treating the restore as good.

Snapshot Lifecycle Management (SLM)

SLM schedules snapshots automatically. Create a policy:

PUT /_slm/policy/daily-shop-orders
{
  "schedule": "0 30 1 * * ?",
  "name": "<shop-orders-{now/d}>",
  "repository": "my-repo",
  "config": {
    "indices": ["shop-orders-*"],
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

Trigger it manually to test:

POST /_slm/policy/daily-shop-orders/_execute

Check SLM status:

GET /_slm/policy/daily-shop-orders
GET /_slm/stats

Snapshot best practices

• Always restore into a separate index (or separate cluster) to validate — never overwrite a live index with an untested restore.
• Take a snapshot before any reindex, bulk update, or upgrade operation.
• Test restores on a schedule, not just when you need them. A snapshot that cannot restore is not a backup.
• Monitor SLM policy execution and alert on failures.

8.4 Ingest pipelines and Logstash/Beats — when to use what

Ingest pipelines — ES-native, runs on ingest nodes

Ingest pipelines process documents inside Elasticsearch before they are written to shards. They run on ingest-role nodes (or any node in small clusters).

Common processors:

Register a pipeline

PUT /_ingest/pipeline/normalize-orders-v1
{
  "description": "Normalize shop-orders-demo documents on ingest",
  "processors": [
    {
      "set": {
        "field": "@timestamp",
        "value": "{{created_at}}",
        "ignore_failure": true
      }
    },
    {
      "date": {
        "field": "@timestamp",
        "formats": ["ISO8601", "yyyy-MM-dd'T'HH:mm:ssZ"],
        "ignore_failure": true
      }
    },
    {
      "lowercase": {
        "field": "customer.name",
        "ignore_missing": true
      }
    },
    {
      "convert": {
        "field": "amount",
        "type": "double",
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "field": ["internal_debug_field"],
        "ignore_missing": true
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "ingest.error",
        "value": "{{_ingest.on_failure_message}}"
      }
    }
  ]
}

Test with _simulate before deploying

POST /_ingest/pipeline/normalize-orders-v1/_simulate
{
  "docs": [
    {
      "_index": "shop-orders-demo",
      "_source": {
        "created_at": "2026-05-30T09:15:00Z",
        "customer": { "name": "Alice SMITH" },
        "amount": "149.99",
        "internal_debug_field": "drop-me"
      }
    }
  ]
}

Inspect the _simulate response’s doc._source to verify the transformation before attaching it to an index.

Attach to an index

Per-request (one-off or testing):

POST /shop-orders-demo/_doc?pipeline=normalize-orders-v1
{ "created_at": "2026-05-30T09:15:00Z", "amount": "149.99" }

As the index default (applies to all documents indexed to that index):

PUT /shop-orders-demo/_settings
{
  "index.default_pipeline": "normalize-orders-v1"
}

Or set it in the index template so every new index in the pattern picks it up automatically.

Decision guide: pipeline vs Beats vs Logstash

Comparison table:

Use ingest pipelines by default for anything you can express in processors. Add Logstash only when you genuinely need its power — it adds operational complexity. Beats are for collection, not transformation.

8.5 Best practices wrap-up

This section is a concise checklist tying together the entire course. Work through it when designing a new index, reviewing an integration, or doing a health check on an existing cluster.

Mappings

• Use explicit mappings for every critical field — do not rely on dynamic detection for production indices.
• Apply multi-fields (text + .raw keyword) for any field you need to both search full-text and aggregate/sort.
• Set dynamic: strict (reject unexpected fields) or dynamic: false (ignore them) on indices where schema drift is a risk.
• Use flattened for arbitrary key/value blobs (labels, metadata, user-supplied JSON) to avoid field explosion.
• Use nested only when per-item correctness in arrays of objects is required — it has query cost.
• Define mappings in index templates so every new rollover index inherits the correct schema.

Queries

• Put non-scoring conditions in filter context (bool.filter) — they are cached and faster.
• Always scope to a time range on time-series indices; “query all history” will be expensive.
• Avoid high-cardinality terms aggregations on unbounded fields (IDs, UUIDs) — use composite for paging buckets if you need enumeration.
• Use search_after for deep pagination; avoid from/size beyond a few thousand results.
• Profile expensive queries in Dev Tools with "profile": true before running them in production.

Shards, ILM, and templates

• Target 10–50 GB per shard; stay under a few hundred shards per node.
• Use ILM with rollover for time-series indices — it keeps shards predictably sized over time.
• Use write aliases as the indexing endpoint so rollover is transparent to the application.
• Do not create one index per tenant or per day without a retention plan — shard count creep is a real stability risk.

Aliases for zero-downtime changes

When you need to fix a mapping or reindex: 1. Create the new index with the correct mapping. 2. Reindex from old to new. 3. Flip the alias atomically to point to the new index. 4. Applications pointing at the alias see no interruption.

POST /_aliases
{
  "actions": [
    { "remove": { "index": "shop-orders-demo-v1", "alias": "shop-orders" } },
    { "add":    { "index": "shop-orders-demo-v2", "alias": "shop-orders" } }
  ]
}

Elasticsearch as a derived store

• Know what is the source of truth for your data. If it is a database or event stream, ES is a derived index — you can rebuild it.
• If ES is the only store for some data, enforce snapshot discipline and test restores.
• Do not use ES as a primary transactional store — it is optimized for search, not ACID writes.

Integration hygiene

• Use ingest pipelines for light normalization; test every pipeline with _simulate before deploying.
• Set on_failure handlers in pipelines so bad documents are marked with an error field rather than silently dropped.
• Use index.default_pipeline in templates so the pipeline is applied automatically.
• Validate your bulk indexing error responses — the bulk API returns 200 even when individual documents fail.

# Check for per-document errors in a bulk response
# errors: true in the response body means at least one item failed
# Always check "items[].index.error" when errors is true

Monitor heap, disk, and rejections

• Heap > 75–85% sustained: investigate. Reduce cardinality, shard count, or query concurrency.
• Disk usage rising toward 85%: fix retention or add capacity before the watermark hits.
• Any thread pool rejections: act immediately — requests are being dropped.
• Check _cluster/pending_tasks when anything feels slow — a growing queue signals master pressure.

Safety habits

• Take a snapshot before any reindex, bulk delete, or major mapping change.
• Test your restore process — a snapshot you have never restored is unproven.
• Use _cluster/allocation/explain before force-allocating or deleting indices during a health incident.
• Never delete indices under pressure without confirming they are not the only copy of that data.

Labs

Lab 1 — Cluster health triage

1. Run GET /_cluster/health, GET /_cat/shards?v, GET /_cat/nodes?v.
2. Identify any UNASSIGNED shards and run GET /_cluster/allocation/explain.
3. Reproduce a yellow state: create an index with number_of_replicas: 1 on a single-node cluster, then fix it by setting replicas to 0.
4. Check GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected and interpret any non-zero rejected counts.

Takeaway: five API calls give you a near-complete picture of cluster health. The allocation explain is the single most useful diagnostic.

Lab 2 — Snapshot and restore drill

1. Register a filesystem repository: PUT /_snapshot/lab-repo with a writable location.
2. Take a snapshot of shop-orders-demo: PUT /_snapshot/lab-repo/snap-lab-1?wait_for_completion=true.
3. Verify: GET /_snapshot/lab-repo/snap-lab-1 — confirm state: SUCCESS.
4. Restore with rename: restore shop-orders-demo as restored-shop-orders-demo using rename_pattern/rename_replacement.
5. Run GET /restored-shop-orders-demo/_count and compare to the original. Run a sample query to confirm data integrity.
6. Set up an SLM policy with PUT /_slm/policy/lab-daily and execute it manually.

Takeaway: “backup” only exists if you have practiced the restore. Rename-on-restore prevents clobbering live data.

Lab 3 — Build and test an ingest pipeline

1. Create a pipeline normalize-orders-v1 with these processors on shop-orders-demo documents:
   • set to copy created_at into @timestamp
   • date to parse @timestamp into a proper date
   • lowercase on customer.name
   • convert to ensure amount is double
2. Test it with POST /_ingest/pipeline/normalize-orders-v1/_simulate using a sample document.
3. Inspect the simulated output: confirm @timestamp is parsed, customer.name is lowercase, amount is numeric.
4. Attach the pipeline as index.default_pipeline on shop-orders-demo.
5. Index a new test document without specifying the pipeline explicitly — confirm the pipeline ran.

Takeaway: always _simulate before deploying. on_failure handlers ensure bad documents are visible rather than silently lost.

Lab 4 — Monitoring signals baseline

1. Run GET /_nodes/stats and find: heap used %, search query total and time, write rejections.
2. Run GET /_cat/indices?v&s=store.size:desc — identify the three largest indices.
3. Run GET /shop-orders-demo/_stats and locate: doc count, store size, indexing latency, search latency, request cache hit/miss ratio.
4. Open Kibana Stack Monitoring and correlate the same node metrics in the UI.
5. Identify which of the five “signals that predict trouble” (heap, disk, rejections, pending tasks, unassigned shards) are non-zero in your cluster and note what action each would warrant.

Takeaway: you can read the state of a cluster in under five minutes. Monitoring is only useful if you know what action follows each signal.

Summary

• Green = all shards allocated; Yellow = replicas missing (reduced resilience); Red = primaries missing (data unavailable). Diagnose with _cluster/allocation/explain before acting.
• Do not randomly delete indices or force-allocate shards during incidents. Find the cause first.
• The five signals that predict most production incidents: heap pressure, disk approaching watermark, thread pool rejections, growing pending tasks, and unassigned shards.
• Snapshots are the only supported backup. Register a repository, automate with SLM, and validate restores on a schedule — not on the day you need them.
• Use ingest pipelines for light transformation at index time. Test with _simulate. Use Beats to collect and ship. Use Logstash when you need complex parsing, buffering, or multiple destinations.
• Explicit mappings + index templates + aliases are the foundation of a stable, evolvable schema. Treat dynamic mapping as a risk to constrain.
• Know whether Elasticsearch is your source of truth or a derived index — it changes your recovery options and backup obligations.