04 — Mapping and analyzers

Mapping is the schema of your Elasticsearch index. It controls what you can search, filter, sort, and aggregate — and getting it wrong early means a painful reindex later. This chapter gives you a precise mental model of how Elasticsearch understands your data, right down to the individual token.

Goals for this chapter

By the end, you should be able to:

• Read and interpret the mapping of any index
• Choose the right field type for each use case
• Explain the text analysis pipeline (character filters → tokenizer → token filters) and run it live with the _analyze API
• Know which built-in analyzers are available and when to prefer each one
• Decide between dynamic and explicit mapping — and explain why explicit wins in production
• Know what to expect from Chapter 05 (custom analyzers) and Chapter 07 (reindexing)

4.1 What a mapping is and why it matters

The basic idea

Every time you index a document, Elasticsearch maps each field to a data type. The type determines:

• How the field is stored (term dictionary, doc values, BKD tree, …)
• Which queries work on it (match, term, range, geo_distance, …)
• Whether you can sort or aggregate on it
• How much disk and memory it uses

Think of it as a contract between your data producers and your query consumers.

Viewing the mapping

You can inspect the current mapping of any index at any time:

GET /shop-orders-demo/_mapping

The response shows every field, its type, and any extra options (analyzers, multi-fields, doc_values, etc.):

{
  "shop-orders-demo": {
    "mappings": {
      "properties": {
        "order_id":       { "type": "keyword" },
        "created_at":     { "type": "date" },
        "channel":        { "type": "keyword" },
        "total_amount":   { "type": "double" },
        "returned":       { "type": "boolean" },
        "rating":         { "type": "float" },
        "notes": {
          "type": "text",
          "analyzer": "czech",
          "fields": {
            "std": { "type": "text", "analyzer": "standard" },
            "keyword": { "type": "keyword" }
          }
        },
        "customer": {
          "properties": {
            "id":            { "type": "keyword" },
            "name": {
              "type": "text",
              "analyzer": "czech",
              "fields": {
                "keyword": { "type": "keyword" }
              }
            },
            "loyalty_tier":  { "type": "keyword" }
          }
        },
        "delivery": {
          "properties": {
            "city":         { "type": "keyword" },
            "country_code": { "type": "keyword" },
            "location":     { "type": "geo_point" },
            "days":         { "type": "integer" }
          }
        },
        "items": {
          "type": "nested",
          "properties": {
            "product_id":  { "type": "keyword" },
            "name": {
              "type": "text",
              "analyzer": "czech",
              "fields": {
                "keyword": { "type": "keyword" }
              }
            },
            "category":    { "type": "keyword" },
            "unit_price":  { "type": "double" },
            "quantity":    { "type": "integer" }
          }
        },
        "categories": { "type": "keyword" },
        "tags":        { "type": "keyword" }
      }
    }
  }
}

Key rule: many mapping changes — especially changing a field’s type — require creating a new index and reindexing your data. We cover that process in Chapter 07. Design your mapping thoughtfully the first time.

4.2 Core data types — a practical tour

text — for full-text search

"notes": { "type": "text", "analyzer": "czech" }

• The value is analyzed (tokenized, lowercased, stemmed, …) before indexing.
• Powers match and match_phrase queries with relevance scoring.
• Cannot be used for sorting or aggregations without significant extra cost (fielddata must be enabled, which loads the whole field into heap memory — avoid this).
• Use it when humans write the content and humans search it.

keyword — for exact values

"order_id":     { "type": "keyword" },
"channel":      { "type": "keyword" },
"country_code": { "type": "keyword" },
"loyalty_tier": { "type": "keyword" },
"categories":   { "type": "keyword" },
"tags":         { "type": "keyword" }

• Stored as-is, never analyzed.
• Efficient for term, terms, range filters, sort, and aggregations.
• Elasticsearch uses a compact column store (doc values) for keyword fields — fast and memory-efficient.
• Use it for IDs, status codes, enum values, tags, country codes — any value you match exactly or group by.

The classic mistake: mapping a status field like "order_status" as text. You lose the ability to aggregate by status cleanly, and full-text search on a 5-character code is meaningless.

text vs keyword — the deep comparison

The multi-field pattern

Many fields need both behaviors. The solution is multi-fields — a single logical field that is indexed in two or more ways simultaneously:

PUT /shop-orders-demo/_mapping
{
  "properties": {
    "customer": {
      "properties": {
        "name": {
          "type": "text",
          "analyzer": "czech",
          "fields": {
            "keyword": { "type": "keyword" }
          }
        }
      }
    }
  }
}

Now you can: - Search customer.name with a match query (full-text, analyzed, stemmed Czech) - Aggregate or sort on customer.name.keyword (exact, up to 256 bytes by default)

The notes field in our dataset goes further with a three-way multi-field:

"notes": {
  "type": "text",
  "analyzer": "czech",
  "fields": {
    "std":     { "type": "text",    "analyzer": "standard" },
    "keyword": { "type": "keyword" }
  }
}

• notes — full-text search with Czech stemming and stopword removal
• notes.std — full-text search with the standard (language-neutral) analyzer
• notes.keyword — exact match and aggregation

Storage cost: each sub-field is indexed separately. Multi-fields increase index size proportionally. Add them intentionally, not by default.

Numeric types

From our dataset: delivery.days and items.quantity are integer; total_amount, subtotal_amount, discount_amount, shipping_fee, and unit_price are double; rating is float.

Money tip: consider scaled_float with scaling_factor: 100 if you want to store prices in cents internally. It uses less disk than double and avoids floating-point representation surprises. Example: "unit_price": { "type": "scaled_float", "scaling_factor": 100 }. A value of 19.99 is stored internally as the integer 1999.

date

"created_at": { "type": "date" }

• Accepts ISO 8601 ("2024-11-23T14:05:00Z"), epoch milliseconds, or custom formats.
• Stored internally as a long (epoch millis) regardless of input format.
• Supports range queries, date_histogram aggregations, and date math (now-7d/d).

Always store in UTC. Elasticsearch stores dates as UTC internally. Mixing timezones in ingestion is a common source of “my date histogram is off by 1 hour” bugs. Be explicit: send Z or +00:00 on every timestamp.

boolean

"returned": { "type": "boolean" }

Simple, small, and efficient. Used for binary flags. Accepts true/false JSON booleans (not strings). Filter queries on booleans are extremely fast.

object — nested JSON without the cost

By default, any JSON sub-object becomes an object field. Elasticsearch flattens its properties into the parent document’s field namespace:

{
  "delivery": {
    "city": "Prague",
    "country_code": "CZ",
    "days": 2
  }
}

This is stored internally as delivery.city, delivery.country_code, delivery.days. You access them with dotted paths in queries:

GET /shop-orders-demo/_search
{
  "query": {
    "term": { "delivery.country_code": "CZ" }
  }
}

Cost: low. Object is just a namespace — no special data structure.

Limitation: when object fields appear in arrays, the flattening destroys the per-object relationship. This is where nested becomes necessary.

nested — arrays of objects with preserved relationships

Consider an order with multiple items:

{
  "items": [
    { "product_id": "P001", "name": "Wireless Headphones", "quantity": 1, "unit_price": 79.99 },
    { "product_id": "P002", "name": "USB-C Cable",         "quantity": 3, "unit_price":  9.99 }
  ]
}

With a plain object mapping, Elasticsearch flattens this to:

items.product_id: ["P001", "P002"]
items.name:       ["Wireless Headphones", "USB-C Cable"]
items.quantity:   [1, 3]
items.unit_price: [79.99, 9.99]

A query for “product P001 with quantity > 2” would incorrectly match this document, because P001 and quantity=3 happen to coexist in the same arrays — even though they belong to different items.

With nested, each item is stored as a hidden separate Lucene document, preserving the object’s internal relationships. A nested query will only match items where ALL conditions hold within the same object.

"items": {
  "type": "nested",
  "properties": {
    "product_id": { "type": "keyword" },
    "name":       { "type": "text", "analyzer": "czech",
                    "fields": { "keyword": { "type": "keyword" } } },
    "category":   { "type": "keyword" },
    "unit_price": { "type": "double" },
    "quantity":   { "type": "integer" }
  }
}

Cost: nested fields require special nested queries and aggregations — they cannot be used in regular top-level queries. They also increase index size because each nested object is a separate document internally. Chapter 05 covers nested queries in detail.

geo_point — geographic coordinates

"delivery": {
  "properties": {
    "location": { "type": "geo_point" }
  }
}

Accepts coordinates in several formats:

{ "lat": 50.0755, "lon": 14.4378 }   // object
"50.0755,14.4378"                      // string
[14.4378, 50.0755]                     // array — note: [lon, lat] order!

Array format warning: geo arrays use [longitude, latitude] order (GeoJSON convention) — the opposite of human intuition. Stick with the object {"lat": ..., "lon": ...} format to avoid subtle bugs.

Enables geo_distance queries, geo_bounding_box filters, and geo_centroid aggregations.

4.3 What happens when a document is indexed — text analysis

When Elasticsearch receives a text field, it does not store the raw string in the inverted index. Instead, it runs the value through an analyzer — a pipeline with three stages.

The analysis pipeline

Raw text
    │
    ▼
┌─────────────────────┐
│  Character filters  │  (optional) modify raw characters before tokenization
│                     │  e.g. strip HTML tags, replace & → and
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│     Tokenizer       │  splits the text into individual tokens
│                     │  e.g. split on whitespace, punctuation
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│   Token filters     │  (optional, chainable) transform tokens
│                     │  e.g. lowercase, remove stopwords, stem
└─────────────────────┘
    │
    ▼
Tokens → stored in inverted index

Walking through an example

Let us analyze the string "The Wireless Headphones, 2-pack!" with the standard analyzer using the _analyze API:

POST /_analyze
{
  "analyzer": "standard",
  "text": "The Wireless Headphones, 2-pack!"
}

Response (simplified):

{
  "tokens": [
    { "token": "the",         "position": 0 },
    { "token": "wireless",    "position": 1 },
    { "token": "headphones",  "position": 2 },
    { "token": "2",           "position": 3 },
    { "token": "pack",        "position": 4 }
  ]
}

What happened: - The standard tokenizer split on whitespace and punctuation (Headphones, → headphones; 2-pack! → 2, pack) - The lowercase token filter turned everything to lowercase - The comma, hyphen, and exclamation mark were discarded

These 5 tokens are what goes into the inverted index.

Analysis at query time too

Here is the crucial insight: analysis happens both at index time and at query time (for match and match_phrase queries). When you send:

GET /shop-orders-demo/_search
{
  "query": { "match": { "notes": "Wireless Headphones" } }
}

Elasticsearch applies the same analyzer to "Wireless Headphones", producing tokens ["wireless", "headphones"], and then looks those up in the inverted index.

The analyzer used at query time must be compatible with the one used at index time. If you index with an English stemmer (turning headphones into headphon) but query without it, the tokens won’t match.

Rule of thumb: unless you explicitly need different index/search analyzers, use the same one for both — this is the default behavior.

Why keyword fields are never analyzed

keyword fields skip the entire analysis pipeline. The raw string is stored exactly as-is. This is intentional: when you filter on order_status: "pending", you want an exact byte-for-byte match, not tokenized fragments.

Try it on your own index

You can test the analyzer configured on any specific field in your index:

POST /shop-orders-demo/_analyze
{
  "field": "notes",
  "text": "Bezdrátová sluchátka nebyla součástí balení."
}

POST /shop-orders-demo/_analyze
{
  "field": "notes.std",
  "text": "Bezdrátová sluchátka nebyla součástí balení."
}

Compare the tokens from the Czech analyzer versus the standard analyzer on the same text — you will see stemming and stopword removal in the Czech version.

4.4 Built-in analyzers

Elasticsearch ships with several ready-to-use analyzers. Here is a practical guide to the ones you will encounter most often.

standard (the default)

Used by default for all text fields that do not specify an analyzer.

Pipeline: - No character filters - Standard tokenizer (splits on Unicode word boundaries — handles most European languages reasonably) - Lowercase token filter

POST /_analyze
{
  "analyzer": "standard",
  "text": "The Wireless Headphones, 2-pack!"
}

Result: ["the", "wireless", "headphones", "2", "pack"]

Good general-purpose choice for multilingual content where language-specific stemming is not critical.

simple

Splits on any non-letter character and lowercases. Discards numbers entirely.

POST /_analyze
{
  "analyzer": "simple",
  "text": "The Wireless Headphones, 2-pack!"
}

Result: ["the", "wireless", "headphones", "pack"] — the 2 is gone.

whitespace

Splits only on whitespace. Does not lowercase. Keeps punctuation attached to tokens.

POST /_analyze
{
  "analyzer": "whitespace",
  "text": "The Wireless Headphones, 2-pack!"
}

Result: ["The", "Wireless", "Headphones,", "2-pack!"]

Useful when case and punctuation are meaningful (e.g., log analysis, code tokens). Note that "Headphones," includes the comma and "The" is capitalized — matching must account for this.

keyword analyzer

Treats the entire input as a single token. Equivalent to no analysis at all.

POST /_analyze
{
  "analyzer": "keyword",
  "text": "The Wireless Headphones, 2-pack!"
}

Result: ["The Wireless Headphones, 2-pack!"] — one token, exact string.

This is different from the keyword field type (though the effect is similar). You would use this analyzer in edge cases like a text field where you still want fielddata behavior but no tokenization.

stop

Like standard, but also removes common stopwords (the, a, is, and, …).

POST /_analyze
{
  "analyzer": "stop",
  "text": "The Wireless Headphones, 2-pack!"
}

Result: ["wireless", "headphones", "2", "pack"] — "the" is gone.

Language analyzers (english, czech, and others)

Language analyzers combine tokenization, lowercasing, stopword removal, and stemming for a specific language.

Our shop-orders-demo index uses the czech analyzer for notes, customer.name, and items.name. Let us compare standard vs english vs czech on a product name:

Standard:

POST /_analyze
{
  "analyzer": "standard",
  "text": "wireless headphones premium"
}

Result: ["wireless", "headphones", "premium"]

English:

POST /_analyze
{
  "analyzer": "english",
  "text": "wireless headphones premium"
}

Result: ["wireless", "headphon", "premium"]

Stemming reduced headphones → headphon. Now a search for headphone (singular) also matches documents containing headphones (plural). This is the power of language analyzers — they normalize morphological variants.

Stopword effect:

POST /_analyze
{
  "analyzer": "english",
  "text": "the best headphones for running"
}

Result: ["best", "headphon", "run"] — "the" and "for" removed as stopwords; "running" stemmed to "run".

Pitfall: if you use the english or czech analyzer at index time, you must also use the same analyzer at query time. The match query does this automatically (it uses the field’s configured search_analyzer, which defaults to the index analyzer). But term queries bypass analysis — a term query for "headphones" against an English-analyzed field will not match, because the indexed token is "headphon". Use match for analyzed fields.

Available language analyzers include: arabic, bulgarian, catalan, czech, danish, dutch, english, estonian, finnish, french, galician, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, thai, turkish.

Quick comparison table

4.5 Dynamic vs explicit mapping — the decision

Dynamic mapping: convenient but dangerous

When you index a document without an explicit mapping, Elasticsearch inspects the first value of each field and guesses the type:

This is enabled by default ("dynamic": true) and is great for quick prototyping and exploration. For production, it has serious pitfalls:

Pitfall 1 — Wrong type guessed from the first document. The first document sends "order_status": "200" as a string, so the field is mapped as text + keyword. Six months later someone sends "order_status": 200 as a number. Elasticsearch rejects those documents with a mapping conflict error.

Pitfall 2 — Mapping explosion. If your documents contain arbitrary keys (user-defined labels, parsed JSON blobs, HTTP headers), the number of fields in the mapping can grow into the thousands. Each field consumes memory in the cluster state and heap. This is a known cause of cluster instability.

Pitfall 3 — Text guessed when keyword is needed. Dynamic mapping maps plain strings as text + .keyword. If you never need full-text search on a field (e.g., a tag or status code), you are wasting an inverted index.

Dynamic mapping settings

You can control dynamic mapping at the index or object level:

PUT /shop-orders-demo-v2
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "order_id":     { "type": "keyword" },
      "channel":      { "type": "keyword" },
      "created_at":   { "type": "date" },
      "total_amount": { "type": "double" }
    }
  }
}

Use "dynamic": "strict" for any index where you control the producer — it acts as a schema enforcer and surfaces ingestion bugs immediately instead of silently creating wrong types.

Use "dynamic": false for objects where you want to keep raw data in _source but do not need to query the arbitrary keys — e.g., a metadata blob from a third-party system.

Dynamic templates — a brief mention

Dynamic templates let you customize how dynamically-detected fields are mapped, based on patterns or detected types. For example, you can say “any field whose name ends in _at should be mapped as date” or “any field that Elasticsearch would map as text should instead be keyword”. Dynamic templates are a powerful middle ground between fully dynamic and fully explicit mapping. They are a Day 2 topic worth exploring once you are comfortable with explicit mappings.

Explicit mapping — the production standard

The recommended approach for any index that matters in production is to define the mapping explicitly before you start ingesting data:

PUT /shop-orders-demo-v2
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "order_id":        { "type": "keyword" },
      "created_at":      { "type": "date" },
      "channel":         { "type": "keyword" },
      "currency":        { "type": "keyword" },
      "payment_method":  { "type": "keyword" },
      "order_status":    { "type": "keyword" },
      "shipping_method": { "type": "keyword" },
      "subtotal_amount": { "type": "double" },
      "discount_amount": { "type": "double" },
      "shipping_fee":    { "type": "double" },
      "total_amount":    { "type": "double" },
      "returned":        { "type": "boolean" },
      "rating":          { "type": "float" },
      "promotion_code":  { "type": "keyword" },
      "categories":      { "type": "keyword" },
      "tags":            { "type": "keyword" },
      "notes": {
        "type": "text",
        "analyzer": "czech",
        "fields": {
          "std":     { "type": "text",    "analyzer": "standard" },
          "keyword": { "type": "keyword" }
        }
      },
      "customer": {
        "properties": {
          "id":          { "type": "keyword" },
          "name": {
            "type": "text",
            "analyzer": "czech",
            "fields": { "keyword": { "type": "keyword" } }
          },
          "loyalty_tier": { "type": "keyword" }
        }
      },
      "delivery": {
        "properties": {
          "city":         { "type": "keyword" },
          "country_code": { "type": "keyword" },
          "location":     { "type": "geo_point" },
          "days":         { "type": "integer" }
        }
      },
      "items": {
        "type": "nested",
        "properties": {
          "product_id": { "type": "keyword" },
          "name": {
            "type": "text",
            "analyzer": "czech",
            "fields": { "keyword": { "type": "keyword" } }
          },
          "category":   { "type": "keyword" },
          "unit_price": { "type": "double" },
          "quantity":   { "type": "integer" }
        }
      }
    }
  }
}

You cannot change an existing field’s type

Once a field is mapped, you cannot change its type. Attempting to do so returns an error:

illegal_argument_exception: mapper [channel] of different type,
current_type [text], merged_type [keyword]

The only safe fix is: 1. Create a new index with the correct mapping 2. Reindex the data (Chapter 07 covers this with the _reindex API) 3. Flip an alias to point to the new index

The one exception: you can add new fields to an existing mapping at any time (as long as dynamic is not strict). You can also add new multi-fields to an existing field. You just cannot change the type of an already-mapped field.

Labs

Lab 1 — Inspect and compare mapping behavior

1. Run GET /shop-orders-demo/_mapping and identify: which fields are keyword, which are text, which have multi-fields, and which use a language analyzer.
2. Try aggregating on a text field (e.g., customer.name) and observe the error.
3. Repeat the aggregation on customer.name.keyword and see it succeed.
4. Explain in your own words why the .keyword sub-field exists.

Takeaway: the field type — not the data — determines what queries and aggregations are possible.

Lab 2 — Explore analysis with _analyze

1. Run the same phrase (e.g., "Wireless Headphones, 2-pack!") through standard, whitespace, english analyzers using POST /_analyze. Compare the token lists.
2. Run POST /shop-orders-demo/_analyze { "field": "notes", "text": "..." } and POST /shop-orders-demo/_analyze { "field": "notes.std", "text": "..." } on the same text. Note differences in stemming and stopword removal.
3. Run POST /_analyze { "analyzer": "keyword", "text": "Order-2024-XYZ" } — confirm it produces exactly one token.

Takeaway: the _analyze API is your primary debugging tool for understanding why a search query does or does not match a document.

Lab 3 — Dynamic vs strict mapping

1. Create a test index with "dynamic": true (or no mapping at all). Index a document with a field "delivery_note": "leave at door". Check GET /test-dynamic/_mapping — observe the auto-generated text + keyword multi-field.
2. Now index a second document that sends "delivery_note": 42 (a number). Observe the mapping conflict error.
3. Create a second test index with "dynamic": "strict" and only order_id defined. Try indexing a document with an extra field. Observe the rejection error.
4. Fix it: add the field to the explicit mapping, then index again successfully.

Takeaway: dynamic mapping is convenient for exploration but causes type conflicts and field explosion in production. Explicit mapping with strict mode enforces your schema.

Lab 4 — Multi-field design

1. Create a new index with customer_name mapped as text only (no multi-field).
2. Index 5 documents with different customer names.
3. Try a terms aggregation on customer_name — observe the fielddata error.
4. Create a new version of the index with customer_name as text + .keyword multi-field.
5. Reindex the 5 documents. Now run both a match query (full-text) and a terms aggregation (exact) on the same logical field.

Takeaway: multi-fields give you both full-text search and exact operations on the same data — a pattern you will use constantly in production.

Summary

• A mapping defines field types and options. It controls what queries, sorts, and aggregations are possible, and it largely determines index performance.
• Use keyword for IDs, codes, tags, and any value you match exactly or group by. Use text for human-written content you search with relevance.
• The multi-field pattern (text + .keyword) is the standard solution when you need both full-text search and exact operations on the same field.
• Text analysis is a pipeline: character filters → tokenizer → token filters. Analysis runs at both index time and query time; the analyzers must be compatible.
• Use the _analyze API liberally — it shows you exactly what tokens are stored for any analyzer and field combination.
• Built-in analyzers cover most needs: standard for general use, language analyzers (english, czech, …) for stemming and stopword removal in a specific language.
• Dynamic mapping is dangerous in production: it guesses wrong types and can cause field explosion. Prefer explicit mappings with "dynamic": "strict" for important indices.
• You cannot change an existing field’s type — design carefully, and use a reindex (Chapter 07) when you must change the schema.

Coming up in Day 2: In Chapter 05 you will build your own custom analyzers — combining character filters, tokenizers, and token filters to handle specific requirements like synonym expansion, edge n-gram prefix search, or language-specific normalization that the built-in analyzers do not cover.