06 — Aggregations

Aggregations are Elasticsearch’s analytics engine — running in parallel across shards, returning summarised results in milliseconds on billions of documents. This chapter assumes you already know how to query; now you’ll shape, slice, and pipeline the data those queries scope.

Goals for this chapter

6.0 — Aggregation basics

The aggs block

Aggregations live in an aggs (or aggregations) top-level key alongside query. The query narrows the set of documents the aggregation sees — aggregations always run on the result set of the enclosing query, not on the full index.

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "range": {
      "created_at": {
        "gte": "2024-01-01",
        "lte": "2024-12-31"
      }
    }
  },
  "aggs": {
    "orders_by_channel": {
      "terms": {
        "field": "channel"
      }
    }
  }
}

Key patterns to internalise:

Pattern Why
"size": 0 Skips returning the top-N search hits entirely. Use this whenever you only need aggregation results — saves serialisation and network cost.
query scopes aggs Any query (bool, range, term…) applied here reduces the doc set before aggregations run. This is how you do “revenue for Q1 only”.
Named aggs Every aggregation gets a user-chosen name (orders_by_channel above). This name appears in the response and is required for buckets_path references in pipeline aggs.

Reading the response

{
  "hits": { "total": { "value": 48321 }, "hits": [] },
  "aggregations": {
    "orders_by_channel": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 1204,
      "buckets": [
        { "key": "web",    "doc_count": 22100 },
        { "key": "mobile", "doc_count": 18017 },
        { "key": "api",    "doc_count": 7000  }
      ]
    }
  }
}

The hits.hits array is empty because of size: 0. The aggregations key holds the named results.

On doc_values: Aggregations read from doc_values, the columnar on-disk structure that supports efficient sorting and aggregation. This means you can aggregate on keyword, numeric, and date fields — but not on full-text analyzed fields (type text). If you try to terms on a text field you will get an error, or at best on its unanalysed .keyword sub-field if one exists. See ch04 for mapping decisions around keyword vs text and when to add a .keyword sub-field.

6.1 — Bucket aggregations

Bucket aggregations partition documents into groups (buckets). Each bucket can contain sub-aggregations.

6.1.1 terms — top values

GET shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "orders_per_country": {
      "terms": {
        "field": "delivery.country_code",
        "size": 10,
        "order": { "_count": "desc" },
        "min_doc_count": 1,
        "missing": "UNKNOWN"
      }
    }
  }
}

Important parameters:

Parameter Default Notes
size 10 How many buckets to return. More = more memory.
order _count desc Can order by _count, _key, or a sub-agg metric.
min_doc_count 1 Buckets with fewer docs are dropped from results. Set to 0 to surface zero-count buckets (rarely useful here; more relevant in date_histogram).
missing (omitted) Assigns docs with a null value to a named bucket.
shard_size size * 1.5 + 10 Per-shard candidate count.

The accuracy caveat: doc_count_error_upper_bound and shard_size

terms is approximate in a distributed setting. Each shard independently computes its top-N buckets and the coordinator merges them. A term that is the 11th most popular on each shard may never reach the coordinator, even if it is globally in the top 10.

Pitfall: Never use terms on a high-cardinality field like customer.id or order_id to “get all values” — that is pagination, not aggregation, and it will OOM your nodes. Use the composite aggregation for that (see §6.5).

6.1.2 date_histogram — orders over time

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "range": {
      "created_at": { "gte": "2024-01-01", "lt": "2025-01-01" }
    }
  },
  "aggs": {
    "orders_per_month": {
      "date_histogram": {
        "field": "created_at",
        "calendar_interval": "month",
        "time_zone": "Europe/Prague",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "2024-01-01",
          "max": "2024-12-31"
        }
      }
    }
  }
}

calendar_interval vs fixed_interval:

Type Examples Use when
calendar_interval minute, hour, day, week, month, quarter, year Human-readable buckets; DST-aware with time_zone.
fixed_interval 60m, 7d, 4h Strict equal-width intervals; not DST-aware.

6.1.3 range — custom revenue buckets

GET shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "revenue_buckets": {
      "range": {
        "field": "total_amount",
        "ranges": [
          { "to": 50,            "key": "micro (< 50)"     },
          { "from": 50,  "to": 200,  "key": "small (50-200)"  },
          { "from": 200, "to": 500,  "key": "medium (200-500)"},
          { "from": 500,             "key": "large (500+)"    }
        ]
      }
    }
  }
}

date_range works identically but accepts date math expressions (now-30d/d, 2024-Q1, etc.) and a format parameter for the response keys. Ranges are always [from, to).

6.2 — Metric aggregations

Metric aggregations compute a single value (or a set of values) from a bucket’s documents. They are typically leaf nodes in the aggregation tree.

6.2.1 Core single-value metrics

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "term": { "order_status": "completed" }
  },
  "aggs": {
    "avg_order_value":     { "avg":         { "field": "total_amount" } },
    "total_revenue":       { "sum":         { "field": "total_amount" } },
    "min_order":           { "min":         { "field": "total_amount" } },
    "max_order":           { "max":         { "field": "total_amount" } },
    "order_count":         { "value_count": { "field": "total_amount" } }
  }
}

6.2.2 stats and extended_stats

One call, multiple metrics:

"aggs": {
  "order_stats": {
    "stats": { "field": "total_amount" }
  }
}

Response includes count, min, max, avg, sum. extended_stats adds sum_of_squares, variance, std_deviation, and a std_deviation_bounds envelope — useful for anomaly detection baselines.

6.2.3 cardinality — unique customers

"aggs": {
  "unique_customers": {
    "cardinality": {
      "field": "customer.id",
      "precision_threshold": 3000
    }
  }
}

cardinality uses the HyperLogLog++ algorithm — it is approximate. The relative error is ~1.04 / sqrt(precision_threshold). The default precision_threshold is 3000 (roughly ±1% error), at a memory cost of about precision_threshold * 8 bytes per agg per shard. Setting it above 40000 has no further accuracy benefit.

When exact counts matter: pre-aggregate into a rollup, or use composite agg + external counting. For most analytics use cases the ±1-3% error from cardinality is perfectly acceptable.

6.2.4 percentiles and percentile_ranks

"aggs": {
  "delivery_percentiles": {
    "percentiles": {
      "field": "delivery.days",
      "percents": [50, 75, 90, 95, 99],
      "tdigest": { "compression": 100 }
    }
  },
  "order_value_percentiles": {
    "percentiles": {
      "field": "total_amount",
      "percents": [50, 95, 99]
    }
  },
  "sla_compliance": {
    "percentile_ranks": {
      "field": "delivery.days",
      "values": [3, 5, 7]
    }
  }
}

percentile_ranks answers “what fraction of orders were delivered within N days?” — the inverse of percentiles.

Both use TDigest. Increase compression for higher accuracy (default 100); this uses more memory. Values at extreme percentiles (p99.9+) are less accurate than p50/p95.

6.3 — Nested aggregations (multi-level)

6.3.1 Metric inside bucket — average order value per country

GET shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "by_country": {
      "terms": {
        "field": "delivery.country_code",
        "size": 10
      },
      "aggs": {
        "avg_order_value": {
          "avg": { "field": "total_amount" }
        },
        "total_revenue": {
          "sum": { "field": "total_amount" }
        }
      }
    }
  }
}

Each country bucket now carries its own avg_order_value and total_revenue.

6.3.2 Bucket inside bucket — channel → monthly orders

"aggs": {
  "by_channel": {
    "terms": { "field": "channel", "size": 5 },
    "aggs": {
      "orders_over_time": {
        "date_histogram": {
          "field": "created_at",
          "calendar_interval": "month",
          "min_doc_count": 0
        }
      }
    }
  }
}

Nesting depth is not limited by Elasticsearch, but every extra level multiplies the number of buckets and memory consumption. Two or three levels is typical in production.

6.3.3 The filter aggregation — scoped sub-bucket

Use filter to compute a metric on a subset of a bucket’s documents without introducing a full new bucket axis:

"aggs": {
  "by_country": {
    "terms": { "field": "delivery.country_code" },
    "aggs": {
      "high_value_orders": {
        "filter": { "range": { "total_amount": { "gte": 500 } } },
        "aggs": {
          "avg_high_value": { "avg": { "field": "total_amount" } }
        }
      }
    }
  }
}

6.3.4 The nested aggregation — diving into items

The items field is mapped as nested (each item is an isolated document internally — see ch04). You cannot aggregate across nested objects with a flat terms — you must wrap the inner agg in a nested context.

GET shop-orders-demo/_search
{
  "size": 0,
  "aggs": {
    "into_items": {
      "nested": { "path": "items" },
      "aggs": {
        "by_category": {
          "terms": {
            "field": "items.category",
            "size": 20
          },
          "aggs": {
            "total_units_sold": {
              "sum": { "field": "items.quantity" }
            },
            "total_item_revenue": {
              "sum": { "field": "items.unit_price" }
            }
          }
        }
      }
    }
  }
}

Note on line-total arithmetic: Elasticsearch does not have a native multiply-then-sum agg. Options are: (a) pre-compute items.line_total = unit_price * quantity at index time (recommended for performance), then sum on it; or (b) use a script in the metric agg ("script": "doc['items.unit_price'].value * doc['items.quantity'].value"). Scripts work but disable doc_values optimisations — see ch07 for performance impact.

reverse_nested — climbing back to the parent document

After descending into items, use reverse_nested to access parent-level fields:

"aggs": {
  "into_items": {
    "nested": { "path": "items" },
    "aggs": {
      "by_category": {
        "terms": { "field": "items.category", "size": 10 },
        "aggs": {
          "back_to_order": {
            "reverse_nested": {},
            "aggs": {
              "avg_order_total": {
                "avg": { "field": "total_amount" }
              },
              "unique_customers": {
                "cardinality": { "field": "customer.id" }
              }
            }
          }
        }
      }
    }
  }
}

This produces: for each item category, the average order total and unique customer count — linking item-level bucketing back to order-level metrics.

6.4 — Pipeline aggregations

Pipeline aggregations operate on the output of other aggregations — their input is bucket values or metrics from a sibling/parent agg, not raw documents. They fall into two families:

Family Examples Input
Parent derivative, cumulative_sum, moving_fn, bucket_script, bucket_selector Each bucket in the parent series
Sibling avg_bucket, max_bucket, min_bucket, sum_bucket, stats_bucket All buckets of a sibling agg

6.4.1 buckets_path syntax

Pipeline aggs reference their inputs with buckets_path. The path walks the agg tree using > as a separator:

"monthly_revenue>_count"          — the doc_count of the monthly_revenue agg
"monthly_revenue>avg_order_value" — the avg_order_value sub-agg inside each bucket

6.4.2 gap_policy

When a bucket has no data (e.g., a month with zero orders), pipeline aggs face a gap. gap_policy controls this:

6.4.3 Full example: monthly revenue analytics

GET shop-orders-demo/_search
{
  "size": 0,
  "query": {
    "range": { "created_at": { "gte": "2024-01-01", "lt": "2025-01-01" } }
  },
  "aggs": {
    "monthly": {
      "date_histogram": {
        "field": "created_at",
        "calendar_interval": "month",
        "min_doc_count": 0,
        "extended_bounds": { "min": "2024-01-01", "max": "2024-12-31" }
      },
      "aggs": {
        "revenue":          { "sum": { "field": "total_amount"    } },
        "discount_total":   { "sum": { "field": "discount_amount" } },
        "subtotal_total":   { "sum": { "field": "subtotal_amount" } },

        "mom_derivative": {
          "derivative": {
            "buckets_path": "revenue",
            "gap_policy": "skip"
          }
        },

        "cumulative_revenue": {
          "cumulative_sum": {
            "buckets_path": "revenue",
            "gap_policy": "insert_zeros"
          }
        },

        "moving_avg_3m": {
          "moving_fn": {
            "buckets_path": "revenue",
            "window": 3,
            "script": "MovingFunctions.unweightedAvg(values)",
            "gap_policy": "skip"
          }
        },

        "discount_ratio": {
          "bucket_script": {
            "buckets_path": {
              "disc": "discount_total",
              "sub":  "subtotal_total"
            },
            "script": "params.sub > 0 ? params.disc / params.sub : 0"
          }
        }
      }
    },

    "best_month": {
      "max_bucket": {
        "buckets_path": "monthly>revenue"
      }
    },

    "avg_monthly_revenue": {
      "avg_bucket": {
        "buckets_path": "monthly>revenue"
      }
    }
  }
}

What each pipeline does:

Agg Type What it produces
mom_derivative Parent Month-over-month revenue change (positive = growth)
cumulative_revenue Parent Running total of revenue from Jan → Dec
moving_avg_3m Parent 3-month unweighted moving average
discount_ratio Parent discount / subtotal per month — a computed ratio
best_month Sibling The single month with peak revenue, across all buckets
avg_monthly_revenue Sibling Average monthly revenue across all 12 months

moving_avg is deprecated. The dedicated moving_avg aggregation was deprecated in Elasticsearch 6.4. Use moving_fn with MovingFunctions.unweightedAvg(values), MovingFunctions.ewma(values, alpha), or MovingFunctions.holt(values, alpha, beta) instead. The moving_fn script has access to the full MovingFunctions helper library.

6.4.4 bucket_selector — filtering buckets by metric value

Remove buckets where the average order value is below a threshold:

"aggs": {
  "by_country": {
    "terms": { "field": "delivery.country_code", "size": 20 },
    "aggs": {
      "avg_value": { "avg": { "field": "total_amount" } },
      "only_high_value_markets": {
        "bucket_selector": {
          "buckets_path": { "avgVal": "avg_value" },
          "script": "params.avgVal >= 150"
        }
      }
    }
  }
}

Buckets where avg_value < 150 are silently dropped from the response. This is a parent pipeline agg — it runs after all bucket metrics are computed.

6.5 — Practical notes and pitfalls

High-cardinality terms — don’t do it

terms on order_id, customer.id, or any UUID-like field will: - Force Elasticsearch to build a massive in-memory bucket structure per shard - Produce inaccurate results because shard_size can never be large enough - Potentially cause OOM errors in extreme cases

Use composite aggregation instead when you need to enumerate all values (pagination of buckets):

"aggs": {
  "paginated_customers": {
    "composite": {
      "size": 1000,
      "sources": [
        { "customer": { "terms": { "field": "customer.id" } } }
      ],
      "after": { "customer": "last-seen-customer-id" }
    }
  }
}

Pass the after_key from the previous response as after in the next request to page through all buckets.

Always scope with a time/range filter

Unbounded aggregations across the full index are expensive and rarely what you want. Always apply a query with a range on created_at (or another date field) before running analytics aggregations. Index lifecycle and rollover strategies (ch08) help by keeping active indices small.

size: 0 is not optional for analytics

Fetching hits alongside aggregations doubles serialisation cost. Unless you genuinely need the documents, set "size": 0.

Memory and accuracy trade-offs — summary

Agg Accuracy Memory lever
terms Approximate (distributed) shard_size
cardinality Approximate (HLL++) precision_threshold
percentiles Approximate (TDigest) compression
date_histogram, range, filter Exact N/A
Pipeline aggs Exact (operate on already-computed values) N/A

Aggregations power Kibana

Every Kibana visualisation (bar chart, line chart, heatmap, data table) issues exactly the aggregation queries covered in this chapter under the hood. Understanding the raw DSL makes it trivial to reproduce or customise any Kibana visualisation manually — and essential when diagnosing why a dashboard shows unexpected results. Forward reference: ch08 covers Kibana configuration and index patterns in the context of operations.

Performance — defer to ch07

Deep performance topics (shard sizing for aggregations, eager_global_ordinals for frequently-bucketed terms fields, doc_values codec choices, search.max_buckets circuit breaker) are covered in ch07. The rule of thumb: if an aggregation is slow or rejected, the root cause is almost always either missing doc_values, too many buckets, or an unscoped query on a large index — all addressed in ch07.

Labs

Lab 1 — Revenue by loyalty tier and country

Build a two-level terms aggregation: outer bucket on customer.loyalty_tier, inner bucket on delivery.country_code. For each country bucket, compute sum of total_amount, avg of total_amount, and cardinality of customer.id. Use size: 0 and scope to the last 12 months.

Takeaway: Multi-level terms bucketing and combining multiple metric aggs in one request; observe how sum_other_doc_count grows as you add inner buckets.

Lab 2 — Monthly sales trend with pipeline analytics

Using date_histogram on created_at with calendar_interval: month and min_doc_count: 0 + extended_bounds for the full year, compute: monthly sum of total_amount, month-over-month derivative, cumulative_sum, and a 3-month moving_fn (unweighted average). Add a sibling max_bucket to identify peak month.

Takeaway: Parent vs. sibling pipeline families; the importance of min_doc_count: 0 for a continuous series; moving_fn replacing deprecated moving_avg.

Lab 3 — Item-level category analysis with nested and reverse_nested

Write an aggregation that descends into the items nested path, buckets by items.category (top 15), sums items.quantity and items.unit_price per category, then uses reverse_nested to compute the average parent total_amount and count of unique customer.id per category.

Takeaway: Why nested is required for nested objects; reverse_nested to cross the boundary back to the parent document; limitations of in-query arithmetic vs. pre-computed fields.

Lab 4 — High-value market filter with bucket_selector

Bucket orders by delivery.country_code (size 20), compute avg total_amount and sum total_amount per country. Use bucket_script to compute a revenue_share ratio (country revenue / a hardcoded total, or simply revenue as a fraction — use a constant for simplicity). Then add a bucket_selector to keep only countries where avg total_amount >= 120. Observe the difference in bucket count with and without the selector.

Takeaway: bucket_script for per-bucket computed fields; bucket_selector as a post-processing filter on metrics — both are parent pipeline aggs and run after metric aggs complete.

Summary