07a — Shard strategy

Every performance and stability issue in Elasticsearch eventually traces back to shards. This part explains what a shard actually costs, how to size them correctly, and — critically — how to safely change the physical layout without touching application code.

Goals for this part

7.1 Shard strategy

What a shard actually is

A shard is a Lucene index. When Elasticsearch fans out a query, every shard touched runs a full query cycle — parse, score, collect, reduce. Each shard also consumes:

The implication: shards are not free. The question is never “how many shards can we have?” but “how many shards keep the cluster stable at peak load and during a node failure?”

Sizing heuristics

Workload Primary shard target
General-purpose search (entity data) 10–50 GB
Low-latency search / high concurrency 5–20 GB (smaller = fewer docs per shard, faster fan-out)
High-throughput append-only (logs) 20–50 GB (larger = fewer total shards)
Very fast SSDs + heavy aggregations Can tolerate 50 GB, but measure

These are starting heuristics, not rules. The right size is the one that keeps recovery time, merge pressure, and query fan-out latency within your SLOs.

number_of_shards is immutable after creation

This is the most important constraint. You set it once at index creation. Changing it requires creating a new index and reindexing. This is why you need a strategy before you create the index, and why you use aliases and rollover so you can iterate over time.

number_of_replicas, by contrast, can be changed at any time:

PUT shop-orders-demo/_settings
{
  "number_of_replicas": 2
}

More replicas = more read throughput + higher HA. Each replica is an additional copy of every primary shard that can serve search traffic.

Over-sharding vs under-sharding

Over-sharding symptoms (too many shards, most too small):

Common cause: creating daily indices with a fixed shard count that was sized for peak, when most days are much smaller.

Under-sharding symptoms (too few shards, each too large):

How to reshard

_split — increases primary count (e.g., 1 → 2). The target shard count must be a multiple of the source. Index must be read-only first. Creates a new index.

POST shop-orders-demo/_split/shop-orders-demo-v2
{
  "settings": {
    "index.number_of_shards": 4
  }
}

_shrink — decreases primary count (e.g., 4 → 2). Index must be read-only, all shards must be on the same node. Useful in ILM warm/cold phases.

POST shop-orders-demo/_shrink/shop-orders-demo-small
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1
  }
}

_reindex into a new index (the general safe path) — works in all cases, allows mapping changes alongside the reshard, is restartable, and does not require read-only state. This is the path you will use most often in production:

POST _reindex?wait_for_completion=false
{
  "source": { "index": "shop-orders-demo" },
  "dest":   { "index": "shop-orders-demo-v2" }
}

The safe-change workflow (always use aliases)

The pattern for any structural change — reshard, mapping change, analyzer change:

1. Create new index  (shop-orders-demo-v2)  with correct settings
2. Reindex from old index to new index
3. Validate (count, sample queries, key aggregations)
4. Atomically flip alias: remove old, add new

POST _aliases
{
  "actions": [
    { "remove": { "index": "shop-orders-demo-v1", "alias": "shop-orders" } },
    { "add":    { "index": "shop-orders-demo-v2", "alias": "shop-orders" } }
  ]
}

5. Keep old index as rollback buffer (e.g., 24h), then delete

Your application always queries the alias shop-orders. It never knows that the physical index changed.

Pitfall: If you skip aliases and hard-code index names into your application, every structural change becomes a coordinated deploy. Aliases decouple the physical layout from the logical name.

Read scaling vs HA with replicas

Replicas are not a substitute for shard count when you have write throughput bottlenecks — replicas do not increase write parallelism.

Lab — Shard strategy and alias-based reshard

Goal: Practice the safe reshard workflow end-to-end using aliases.

Step 1 — Create shop-orders-v1 with 2 primary shards and the alias shop-orders:

PUT /shop-orders-v1
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 0
  },
  "aliases": {
    "shop-orders": {}
  }
}

Step 2 — Index some sample documents via the alias (the alias target is transparent):

POST /shop-orders/_bulk
{ "index": {} }
{ "order_id": "ord-001", "channel": "web",    "total_amount": 150.00, "order_status": "completed" }
{ "index": {} }
{ "order_id": "ord-002", "channel": "mobile", "total_amount": 89.90,  "order_status": "pending" }
{ "index": {} }
{ "order_id": "ord-003", "channel": "web",    "total_amount": 320.00, "order_status": "completed" }
{ "index": {} }
{ "order_id": "ord-004", "channel": "api",    "total_amount": 45.00,  "order_status": "cancelled" }
{ "index": {} }
{ "order_id": "ord-005", "channel": "mobile", "total_amount": 210.00, "order_status": "completed" }

Confirm the document count:

GET /shop-orders/_count

Confirm shard distribution:

GET /_cat/shards/shop-orders-v1?v

Step 3 — Decide to reshard to 3 primaries. Create the target index:

PUT /shop-orders-v2
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0
  }
}

Step 4 — Reindex from v1 to v2 asynchronously and monitor:

POST /_reindex?wait_for_completion=false
{
  "source": { "index": "shop-orders-v1" },
  "dest":   { "index": "shop-orders-v2" }
}

Monitor the task (use the task ID from the response above):

GET /_tasks?actions=*reindex&detailed

Wait for the task to complete, then verify the document count:

GET /shop-orders-v2/_count

Does it match v1? Run a sample query to validate data integrity:

GET /shop-orders-v2/_search
{
  "query": { "term": { "order_status": "completed" } }
}

Step 5 — Atomically flip the alias from v1 to v2. The application (using the alias shop-orders) sees no downtime:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "shop-orders-v1", "alias": "shop-orders" } },
    { "add":    { "index": "shop-orders-v2", "alias": "shop-orders" } }
  ]
}

Verify the alias now points to v2:

GET /_cat/aliases/shop-orders?v

Query via the alias and confirm it reads from v2:

GET /shop-orders/_search
{
  "query": { "match_all": {} }
}

Step 6 — Check shard count of the new index:

GET /_cat/shards/shop-orders-v2?v

Confirm there are 3 primary shards.

Step 7 — Clean up (in production you would keep v1 as a rollback buffer for at least 24h):

DELETE /shop-orders-v1
DELETE /shop-orders-v2

Takeaway: Resharding is always new index → reindex → alias flip → cleanup. The application never sees the physical index change. _reindex with wait_for_completion=false is safe for large datasets — monitor with _tasks. Always use aliases; hard-coded index names turn every structural change into a coordinated deploy.