06a — Bucket and metric aggregations

Aggregations are Elasticsearch’s analytics engine — running in parallel across shards, returning summarised results in milliseconds on billions of documents. This part covers the foundations: how the aggs block works, the three core bucket aggregations, and the full set of metric aggregations including cardinality and percentiles.

Goals for this part

Understand the aggs / query relationship and the size: 0 pattern
Write terms, date_histogram, and range bucket aggregations
Compute avg, sum, min, max, cardinality, and percentiles metrics
Understand accuracy trade-offs for approximate aggregations

6.0 — Aggregation basics

The `aggs` block

Aggregations live in an aggs (or aggregations) top-level key alongside query. The query narrows the set of documents the aggregation sees — aggregations always run on the result set of the enclosing query, not on the full index.

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "range": {
      "created_at": {
        "gte": "2024-01-01",
        "lte": "2024-12-31"
      }
    }
  },
  "aggs": {
    "orders_by_channel": {
      "terms": {
        "field": "channel"
      }
    }
  }
}

Key patterns to internalise:

Pattern	Why
`"size": 0`	Skips returning the top-N search hits entirely. Use this whenever you only need aggregation results — saves serialisation and network cost.
`query` scopes aggs	Any query (bool, range, term…) applied here reduces the doc set before aggregations run. This is how you do “revenue for Q1 only”.
Named aggs	Every aggregation gets a user-chosen name (`orders_by_channel` above). This name appears in the response and is required for `buckets_path` references in pipeline aggs.

Reading the response

{
  "hits": { "total": { "value": 48321 }, "hits": [] },
  "aggregations": {
    "orders_by_channel": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 1204,
      "buckets": [
        { "key": "web",    "doc_count": 22100 },
        { "key": "mobile", "doc_count": 18017 },
        { "key": "api",    "doc_count": 7000  }
      ]
    }
  }
}

The hits.hits array is empty because of size: 0. The aggregations key holds the named results.

On doc_values: Aggregations read from doc_values, the columnar on-disk structure that supports efficient sorting and aggregation. This means you can aggregate on keyword, numeric, and date fields — but not on full-text analyzed fields (type text). If you try to terms on a text field you will get an error, or at best on its unanalysed .keyword sub-field if one exists.

6.1 — Bucket aggregations

Bucket aggregations partition documents into groups (buckets). Each bucket can contain sub-aggregations.

6.1.1 `terms` — top values

GET shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "orders_per_country": {
      "terms": {
        "field": "delivery.country_code",
        "size": 10,
        "order": { "_count": "desc" },
        "min_doc_count": 1,
        "missing": "UNKNOWN"
      }
    }
  }
}

Important parameters:

Parameter	Default	Notes
`size`	10	How many buckets to return. More = more memory.
`order`	`_count desc`	Can order by `_count`, `_key`, or a sub-agg metric.
`min_doc_count`	1	Buckets with fewer docs are dropped from results. Set to `0` to surface zero-count buckets.
`missing`	(omitted)	Assigns docs with a null value to a named bucket.
`shard_size`	`size * 1.5 + 10`	Per-shard candidate count.

The accuracy caveat: `doc_count_error_upper_bound` and `shard_size`

terms is approximate in a distributed setting. Each shard independently computes its top-N buckets and the coordinator merges them. A term that is the 11th most popular on each shard may never reach the coordinator, even if it is globally in the top 10.

doc_count_error_upper_bound in the response tells you the maximum potential undercount for any bucket.
sum_other_doc_count counts docs that fell into buckets not returned.
Increase shard_size to improve accuracy at the cost of more data transferred per shard. A shard_size of size * 5 is a reasonable starting point when precision matters.

Pitfall: Never use terms on a high-cardinality field like customer.id or order_id to “get all values” — that is pagination, not aggregation, and it will OOM your nodes. Use the composite aggregation for that.

6.1.2 `date_histogram` — orders over time

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "range": {
      "created_at": { "gte": "2024-01-01", "lt": "2025-01-01" }
    }
  },
  "aggs": {
    "orders_per_month": {
      "date_histogram": {
        "field": "created_at",
        "calendar_interval": "month",
        "time_zone": "Europe/Prague",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "2024-01-01",
          "max": "2024-12-31"
        }
      }
    }
  }
}

calendar_interval vs fixed_interval:

Type	Examples	Use when
`calendar_interval`	`minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`	Human-readable buckets; DST-aware with `time_zone`.
`fixed_interval`	`60m`, `7d`, `4h`	Strict equal-width intervals; not DST-aware.

min_doc_count: 0 fills in empty buckets (e.g., a month with zero orders). Without it, months with no data are silently omitted, which breaks charts and pipeline aggs that expect a complete series.
extended_bounds forces the histogram to span the full range even if the query already constrains it. Useful when you want 12 monthly buckets regardless of whether January had any orders.

6.1.3 `range` — custom revenue buckets

GET shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "revenue_buckets": {
      "range": {
        "field": "total_amount",
        "ranges": [
          { "to": 50,            "key": "micro (< 50)"     },
          { "from": 50,  "to": 200,  "key": "small (50-200)"  },
          { "from": 200, "to": 500,  "key": "medium (200-500)"},
          { "from": 500,             "key": "large (500+)"    }
        ]
      }
    }
  }
}

date_range works identically but accepts date math expressions (now-30d/d, 2024-Q1, etc.) and a format parameter for the response keys. Ranges are always [from, to).

6.2 — Metric aggregations

Metric aggregations compute a single value (or a set of values) from a bucket’s documents. They are typically leaf nodes in the aggregation tree.

6.2.1 Core single-value metrics

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "term": { "order_status": "completed" }
  },
  "aggs": {
    "avg_order_value":     { "avg":         { "field": "total_amount" } },
    "total_revenue":       { "sum":         { "field": "total_amount" } },
    "min_order":           { "min":         { "field": "total_amount" } },
    "max_order":           { "max":         { "field": "total_amount" } },
    "order_count":         { "value_count": { "field": "total_amount" } }
  }
}

6.2.2 `stats` and `extended_stats`

One call, multiple metrics:

"aggs": {
  "order_stats": {
    "stats": { "field": "total_amount" }
  }
}

Response includes count, min, max, avg, sum. extended_stats adds sum_of_squares, variance, std_deviation, and a std_deviation_bounds envelope — useful for anomaly detection baselines.

6.2.3 `cardinality` — unique customers

"aggs": {
  "unique_customers": {
    "cardinality": {
      "field": "customer.id",
      "precision_threshold": 3000
    }
  }
}

cardinality uses the HyperLogLog++ algorithm — it is approximate. The relative error is ~1.04 / sqrt(precision_threshold). The default precision_threshold is 3000 (roughly ±1% error), at a memory cost of about precision_threshold * 8 bytes per agg per shard. Setting it above 40000 has no further accuracy benefit.

When exact counts matter: pre-aggregate into a rollup, or use composite agg + external counting. For most analytics use cases the ±1-3% error from cardinality is perfectly acceptable.

6.2.4 `percentiles` and `percentile_ranks`

"aggs": {
  "delivery_percentiles": {
    "percentiles": {
      "field": "delivery.days",
      "percents": [50, 75, 90, 95, 99],
      "tdigest": { "compression": 100 }
    }
  },
  "sla_compliance": {
    "percentile_ranks": {
      "field": "delivery.days",
      "values": [3, 5, 7]
    }
  }
}

percentile_ranks answers “what fraction of orders were delivered within N days?” — the inverse of percentiles.

Both use TDigest. Increase compression for higher accuracy (default 100); this uses more memory. Values at extreme percentiles (p99.9+) are less accurate than p50/p95.

Lab — Bucket and metric aggregations on shop-orders-demo

Goal: Write real analytics queries against the course dataset.

Step 1 — Count orders by channel (web, mobile, api). Use size: 0:

GET /shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "by_channel": {
      "terms": {
        "field": "channel",
        "size": 10
      }
    }
  }
}

Questions: 1. What channels exist? Which is the most popular? 2. What is sum_other_doc_count? What does it mean?

Step 2 — Add avg and sum of total_amount inside each channel bucket:

GET /shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "by_channel": {
      "terms": { "field": "channel", "size": 10 },
      "aggs": {
        "avg_order_value": { "avg": { "field": "total_amount" } },
        "total_revenue":   { "sum": { "field": "total_amount" } }
      }
    }
  }
}

Step 3 — Monthly order histogram for the current data range with min_doc_count: 0:

GET /shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "orders_per_month": {
      "date_histogram": {
        "field": "created_at",
        "calendar_interval": "month",
        "min_doc_count": 0
      }
    }
  }
}

Does every month have a bucket, or are some months missing? What does that tell you about the data?

Step 4 — Revenue range buckets: how many orders fall into micro/small/medium/large categories?

GET /shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "order_size_distribution": {
      "range": {
        "field": "total_amount",
        "ranges": [
          { "to": 50,   "key": "micro" },
          { "from": 50,  "to": 200, "key": "small" },
          { "from": 200, "to": 500, "key": "medium" },
          { "from": 500,            "key": "large" }
        ]
      }
    }
  }
}

Step 5 — Unique customer count (cardinality) and order stats for completed orders:

GET /shop-orders-demo/_search
{
  "size": 0,
  "query": { "term": { "order_status": "completed" } },
  "aggs": {
    "unique_customers": {
      "cardinality": {
        "field": "customer.id",
        "precision_threshold": 3000
      }
    },
    "order_value_stats": {
      "extended_stats": { "field": "total_amount" }
    }
  }
}

Questions: 1. How many unique customers placed completed orders? 2. What is the standard deviation of order values? Is the distribution wide or narrow?

Step 6 — Delivery time percentiles: what fraction of orders were delivered within 3, 5, and 7 days?

GET /shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "delivery_days_percentiles": {
      "percentiles": {
        "field": "delivery.days",
        "percents": [50, 75, 90, 95, 99]
      }
    },
    "sla_compliance": {
      "percentile_ranks": {
        "field": "delivery.days",
        "values": [3, 5, 7]
      }
    }
  }
}

Questions: 1. What is the median delivery time (p50)? 2. What percentage of orders were delivered within 5 days?

Takeaway: size: 0 is not optional for analytics queries. Bucket aggs (terms, date_histogram, range) partition documents; metric aggs (avg, sum, cardinality, percentiles) compute values per bucket. terms is approximate — watch doc_count_error_upper_bound. cardinality and percentiles are also approximate; tune their accuracy with precision_threshold and compression.

06a — Bucket and metric aggregations

Goals for this part

6.0 — Aggregation basics

The aggs block

Reading the response

6.1 — Bucket aggregations

6.1.1 terms — top values

The accuracy caveat: doc_count_error_upper_bound and shard_size

6.1.2 date_histogram — orders over time

6.1.3 range — custom revenue buckets