08a — Cluster health and monitoring

A broken cluster at 3am is not the time to learn what _cluster/allocation/explain does. This part builds the diagnostic vocabulary you need before something breaks: how to read health colors, trace unassigned shards to their root cause, and identify the five signals that predict most production incidents.

Goals for this part

Read cluster health and understand what each color means operationally
Use _cluster/allocation/explain to diagnose shard allocation failures
Know safe actions for yellow and red states
Identify the five monitoring signals that predict production incidents
Use _nodes/stats, _cat APIs, and Kibana Stack Monitoring

8.1 Cluster health — green, yellow, red, and what to do

What each color means

Color	Meaning	Traffic impact
Green	All primary and replica shards are allocated	None — fully healthy
Yellow	All primaries allocated; some replicas unassigned	Reads and writes still work, but resilience is reduced (one node loss may cause red)
Red	Some primary shards unassigned	Data for those shards is unavailable; queries/writes to affected indices fail

Yellow is a warning, not an emergency. Red requires immediate attention.

Diagnostics — start here

GET /_cluster/health

GET /_cat/health?v

Spot unassigned shards:

GET /_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason

Filter to unassigned only (pipe in terminal):

curl -s "http://localhost:9200/_cat/shards?v" | grep UNASSIGNED

When you see UNASSIGNED, explain one shard:

GET /_cluster/allocation/explain

This returns the single most important diagnostic — the exact reason Elasticsearch cannot place the shard. Read it literally; it tells you which constraint is blocking.

Safe actions per color

Yellow

Root cause	Safe action
Single-node cluster, `number_of_replicas: 1`	Set replicas to 0 for those indices — replicas cannot be placed on the only node
Disk watermark reached on some nodes	Free disk, expand capacity, or rebalance data; do not fight the watermarks
Not enough nodes for awareness/zone rules	Add a node or relax rules intentionally
Node recently left; replicas recovering	Wait and monitor; watch `initializing_shards` count down

PUT /shop-orders-demo/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}

Red

Priority order: 1. Identify which indices have unassigned primaries (_cat/shards?v). 2. Run _cluster/allocation/explain — read the exact blocker. 3. Fix the blocker: - Bring back the missing node (primaries lived there, no replica existed). - Free disk / expand if watermark is blocking. - Fix allocation filters or version compatibility if during an upgrade. 4. If data is gone: restore from snapshot or rebuild from source of truth.

Pitfall: do not randomly delete indices or force-allocate shards. Forced allocation can cause data loss if a shard copy is stale. Deleting the wrong index is unrecoverable without a snapshot. Always understand the cause before acting.

The disk watermark chain

Elasticsearch applies three thresholds (defaults, configurable):

Low (85%) — stops allocating new shards to the node.
High (90%) — starts moving shards away from the node.
Flood stage (95%) — sets all indices on that node to read-only.

Check current settings and node allocation:

GET /_cat/allocation?v
GET /_cluster/settings?include_defaults=true&filter_path=**watermark**

To clear a read-only block after creating headroom:

PUT /_all/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Clear the block only after you have actual disk headroom. Clearing it on a still-full disk just re-triggers the block on the next write.

8.2 Monitoring — what to watch and where to look

The `_cat` family — quick cluster overview

GET /_cat/indices?v&s=store.size:desc

Lists all indices sorted by disk size — useful for spotting unexpectedly large indices.

GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,node.role

GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected

The thread_pool view is high-value: rejections mean Elasticsearch is dropping work.

`_nodes/stats` and `_stats` — deeper numbers

GET /_nodes/stats

For a developer, the key sub-sections to check:

Section	What to look at
`indices.docs`	Total doc count and deleted docs (deleted docs cause merge overhead)
`indices.store.size_in_bytes`	Disk consumed by shards
`indices.indexing`	`index_total`, `index_time_in_millis` → indexing rate and average latency
`indices.search`	`query_total`, `query_time_in_millis`, `fetch_total`, `fetch_time_in_millis`
`indices.merges`	`current`, `total_time_in_millis` → merge activity and pressure
`indices.refresh`	`total_time_in_millis` → how expensive refreshes are
`indices.query_cache` / `request_cache`	Hit rate (low hit rate with high latency = no caching benefit)
`jvm.mem.heap_used_percent`	Over 75% sustained = concerning; over 85% = act
`thread_pool.search.rejected` / `thread_pool.write.rejected`	Any rejections → capacity problem

Per-index stats:

GET /shop-orders-demo/_stats

Kibana Stack Monitoring

Kibana’s Stack Monitoring UI (sidebar > Management > Stack Monitoring) presents the same data visually:

Cluster overview: health, node count, shard summary.
Per-node views: heap %, CPU, disk, indexing/search rates.
Index-level views: docs, store size, indexing latency.
Logs and slow logs (if enabled).

Use it for historical trends and to correlate signals across time. Dev Tools _cat APIs are faster for point-in-time triage.

The signals that predict trouble

These five signals cover the vast majority of production incidents:

Signal	Threshold to watch	What it predicts
JVM heap %	> 75–85% sustained	GC storms, circuit breaker trips, OOM
Disk usage	Approaching 85% watermark	Shard allocation failures, eventual read-only
Search/write thread pool rejections	Any rejections	Requests being dropped, latency spikes
Growing pending tasks	Queue that does not drain	Master bottleneck, allocation stall
Unassigned shards	Count > 0	Resilience loss (yellow) or data loss (red)

Every alert should have an associated action — who checks what, in which order. An alert without a runbook is noise. Pair each signal with the relevant _cat or _nodes API call as a first step.

Lab — Cluster health triage and monitoring signals

Part A: Health diagnosis

Goal: Build the muscle memory for the five-command cluster health check.

Step 1 — Run the standard health triage sequence and interpret each response:

GET /_cluster/health

GET /_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason

GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,node.role

GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected

GET /_cat/allocation?v

Questions: 1. What is the cluster health status? What does each color imply for traffic? 2. Are any shards UNASSIGNED? Which index and which reason? 3. What is the heap usage? Is it concerning? 4. Are there any thread pool rejections?

Step 2 — Reproduce a yellow state intentionally. Create an index with number_of_replicas: 1 on a single-node cluster:

PUT /health-demo
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1
  }
}

GET /_cluster/health
GET /_cat/shards/health-demo?v

Questions: 1. Why is the cluster now yellow? 2. Which shards are UNASSIGNED? What is their unassigned.reason?

Step 3 — Run _cluster/allocation/explain to get the precise blocking reason:

GET /_cluster/allocation/explain

Read the explanation. What does Elasticsearch say is preventing the replica from being placed?

Step 4 — Fix the yellow state by reducing replicas to 0:

PUT /health-demo/_settings
{
  "index": { "number_of_replicas": 0 }
}

GET /_cluster/health

The cluster should return to green.

Step 5 — Clean up:

DELETE /health-demo

Part B: Monitoring signals baseline

Goal: Read the key monitoring numbers from your cluster and interpret them.

Step 6 — Deep dive into node stats:

GET /_nodes/stats/indices,jvm

Find and record: 1. jvm.mem.heap_used_percent — is it under 75%? 2. indices.search.query_total and indices.search.query_time_in_millis — what is the average query latency? (time / total) 3. indices.indexing.index_total — how many documents have been indexed? 4. thread_pool.search.rejected — any non-zero rejections?

Step 7 — Per-index stats for shop-orders-demo:

GET /shop-orders-demo/_stats

Find: 1. primaries.docs.count — total document count 2. primaries.store.size_in_bytes — disk usage 3. primaries.search.query_time_in_millis and primaries.search.query_total — average search latency 4. primaries.request_cache.hit_count and primaries.request_cache.miss_count — cache hit rate

Step 8 — Find the largest indices on the cluster:

GET /_cat/indices?v&s=store.size:desc&h=index,docs.count,store.size,health

Are there any indices that seem unexpectedly large?

Step 9 — Check pending cluster tasks (a growing queue signals master pressure):

GET /_cluster/pending_tasks

Step 10 — Open Kibana Stack Monitoring (if available in your environment) and find the same metrics: heap %, search rate, indexing rate, and disk usage. Correlate what you see in the UI with what you read from the APIs.

Takeaway: Five API calls give you a near-complete picture of cluster health: _cluster/health, _cat/shards, _cat/nodes, _cat/thread_pool, _cat/allocation. _cluster/allocation/explain is the single most useful diagnostic when shards are unassigned. Never act on a red or yellow cluster without reading the explain first.