05a — Custom analyzers

You already know how built-in analyzers work from chapter 04. This part goes deeper: building your own analysis pipeline, combining char filters, tokenizers, and token filters, and injecting synonyms — everything you need to handle real-world text like HTML-rich notes, diacritic variants, and domain-specific vocabulary.

Goals for this part

Understand the three-stage analysis pipeline and the role of each stage
Build a custom analyzer that combines html_strip, asciifolding, stop words, stemming, and synonyms
Verify the pipeline with _analyze before writing any documents
Choose between index-time and search-time synonym expansion

5.1 Custom analyzers

5.1.1 The pipeline: char filter → tokenizer → token filter

Every analyzer is a three-stage pipeline:

Stage	What it does	Common examples
char filter	Rewrites the character stream before tokenization	`html_strip`, `mapping`, `pattern_replace`
tokenizer	Splits the (rewritten) stream into tokens	`standard`, `whitespace`, `pattern`, `edge_ngram`
token filter	Transforms the token list	`lowercase`, `stop`, `stemmer`, `synonym`, `asciifolding`

The tokenizer is mandatory and exactly one is used. Char filters and token filters are optional and ordered — order matters.

5.1.2 Building a custom analyzer

The following creates an index with a custom English analyzer that: - Strips HTML markup from incoming text (useful if notes or descriptions come from a rich-text editor). - Applies standard tokenization and lowercasing. - Expands synonyms so that “failure” and “error” are treated as equivalent at search time.

PUT /shop-orders-demo-v2
{
  "settings": {
    "analysis": {
      "char_filter": {
        "html_strip_filter": {
          "type": "html_strip"
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "order_synonyms": {
          "type": "synonym",
          "synonyms": [
            "failure, fault => error",
            "cancelled, canceled => cancelled"
          ]
        },
        "ascii_fold": {
          "type": "asciifolding",
          "preserve_original": true
        }
      },
      "analyzer": {
        "notes_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip_filter"],
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "ascii_fold",
            "english_stop",
            "english_stemmer",
            "order_synonyms"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "notes": {
        "type": "text",
        "analyzer": "notes_analyzer",
        "search_analyzer": "notes_analyzer"
      }
    }
  }
}

5.1.3 Verify with `_analyze`

Always confirm the pipeline produces the tokens you expect before indexing data:

POST /shop-orders-demo-v2/_analyze
{
  "analyzer": "notes_analyzer",
  "text": "<p>There was a <b>failure</b> in the payment gateway.</p>"
}

Expected output tokens (roughly): failure is first normalized to error by the synonym filter. HTML tags are stripped. Stopwords like “there”, “was”, “a”, “in”, “the” are removed. “payment” and “gateway” are stemmed.

{
  "tokens": [
    { "token": "error",    "position": 3 },
    { "token": "payment",  "position": 5 },
    { "token": "gatewai",  "position": 6 }
  ]
}

Tip: Use _analyze with "explain": true to see which filter produces each token — invaluable when debugging synonym or stemmer surprises.

5.1.4 Prove synonyms work end-to-end

Index a document, then search with the synonym term:

POST /shop-orders-demo-v2/_doc/1
{
  "notes": "There was a failure in the payment gateway."
}

GET /shop-orders-demo-v2/_search
{
  "query": {
    "match": {
      "notes": "error"
    }
  }
}

The document is returned because “failure” was stored as “error” at index time (expand style). If you used a search-time-only synonym filter, the query term “error” would expand to “failure” at search time instead — both approaches work, but they have different implications for index size and analyzer updates (see below).

5.1.5 Index-time vs search-time synonyms and `search_analyzer`

A field can have two different analyzers:

	Index-time (`analyzer`)	Search-time (`search_analyzer`)
When applied	When `_doc` is indexed	When a query text is analyzed
Synonym strategy	Expand at index time → larger index, can’t change synonyms without reindex	Expand at search time → smaller index, update synonym filter + `POST _reload_search_analyzers`
Typical use	Edge-ngrams (see 5.5), aggressive stemming	Synonyms, user-facing spell variants

For synonyms, search-time expansion is usually preferable because you can update the synonym list without a full reindex:

POST /shop-orders-demo-v2/_reload_search_analyzers

Czech diacritics tip: Add asciifolding (with "preserve_original": true) to both index and search analyzers. This maps “č” → “c”, “ž” → “z”, etc., so users who omit diacritics still find documents that contain them — and vice-versa. Preserving the original keeps exact diacritic matches scoring higher.

5.1.6 Prefix / autocomplete tokenizers

For prefix-search use cases, the edge_ngram tokenizer (or token filter) is the right tool — covered in part 05c. The ngram tokenizer generates all substrings (expensive). whitespace tokenizer skips punctuation-based splitting, useful for things like order IDs or product codes.

Lab — Custom analyzer with synonyms and diacritics

Goal: Build, verify, and test a full custom analysis pipeline.

Step 1 — Create a new index notes-analyzer-lab with a custom analyzer that combines: - html_strip char filter - standard tokenizer - lowercase token filter - asciifolding (with "preserve_original": true) - A synonym filter mapping at least 3 domain-specific synonyms from the order domain, e.g.: "refund, return => return", "cancelled, canceled => cancelled", "broken, damaged => defective"

Map the notes field to use this analyzer at index time, and a version without html_strip as search_analyzer.

PUT /notes-analyzer-lab
{
  "settings": {
    "analysis": {
      "char_filter": {
        "html_strip_filter": { "type": "html_strip" }
      },
      "filter": {
        "ascii_fold": {
          "type": "asciifolding",
          "preserve_original": true
        },
        "order_synonyms": {
          "type": "synonym",
          "synonyms": [
            "refund, return => return",
            "cancelled, canceled => cancelled",
            "broken, damaged => defective"
          ]
        }
      },
      "analyzer": {
        "notes_index_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip_filter"],
          "tokenizer": "standard",
          "filter": ["lowercase", "ascii_fold", "order_synonyms"]
        },
        "notes_search_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "ascii_fold", "order_synonyms"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "notes": {
        "type": "text",
        "analyzer": "notes_index_analyzer",
        "search_analyzer": "notes_search_analyzer"
      }
    }
  }
}

Step 2 — Use _analyze to verify the index analyzer strips HTML and produces correct tokens:

POST /notes-analyzer-lab/_analyze
{
  "analyzer": "notes_index_analyzer",
  "text": "<b>The item was broken</b> and customer requested a refund."
}

Questions to answer: 1. Was the HTML tag stripped? 2. Was “broken” normalized to “defective”? 3. Was “refund” normalized to “return”?

Step 3 — Use "explain": true to see which filter fired for each token:

POST /notes-analyzer-lab/_analyze
{
  "analyzer": "notes_index_analyzer",
  "explain": true,
  "text": "damaged packaging"
}

Step 4 — Index two documents: one using a synonym source word, one using the target word:

POST /notes-analyzer-lab/_doc/1
{
  "notes": "Customer received broken item and wants a refund."
}

POST /notes-analyzer-lab/_doc/2
{
  "notes": "Item was defective. Issued a return."
}

Step 5 — Confirm both documents are found by querying either synonym term:

GET /notes-analyzer-lab/_search
{
  "query": {
    "match": { "notes": "defective" }
  }
}

Both documents should match. Explain why doc 1 also matches even though it uses “broken” and the query uses “defective”.

Step 6 — Test diacritic folding: index a document with Czech text, search without diacritics:

POST /notes-analyzer-lab/_doc/3
{
  "notes": "Zákazník vrátil poškozené zboží."
}

GET /notes-analyzer-lab/_search
{
  "query": {
    "match": { "notes": "poskozene zbozi" }
  }
}

Step 7 — Clean up:

DELETE /notes-analyzer-lab

Takeaway: Index-time and search-time analyzers serve different purposes. Synonym expansion at search time is cheaper to maintain but requires _reload_search_analyzers after updates. asciifolding with preserve_original: true is the right default for multilingual search.