05 — Advanced querying

You already know bool, match, term, range, and how mappings and analyzers work from Day 1. This chapter goes deeper: custom text pipelines, relationship modeling, relevance engineering, and the features (highlighting, autocomplete, suggestions) that turn a search index into a product feature. The emphasis is on trade-offs and when not to use something.

Goals for this chapter

• Build and verify a custom analysis pipeline (char filters, tokenizer, token filters, synonym injection).
• Understand why flat arrays break queries and when to reach for nested, join, or denormalization.
• Add highlighting and completions to queries without surprising performance cliffs.
• Control relevance numerically with function_score and script_score.

5.1 Custom analyzers

5.1.1 The pipeline: char filter → tokenizer → token filter

Every analyzer is a three-stage pipeline:

The tokenizer is mandatory and exactly one is used. Char filters and token filters are optional and ordered — order matters.

5.1.2 Building a custom analyzer

The following creates an index with a custom English analyzer that: - Strips HTML markup from incoming text (useful if notes or descriptions come from a rich-text editor). - Applies standard tokenization and lowercasing. - Expands synonyms so that “failure” and “error” are treated as equivalent at search time.

PUT /shop-orders-demo-v2

{
  "settings": {
    "analysis": {
      "char_filter": {
        "html_strip_filter": {
          "type": "html_strip"
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "order_synonyms": {
          "type": "synonym",
          "synonyms": [
            "failure, fault => error",
            "cancelled, canceled => cancelled"
          ]
        },
        "ascii_fold": {
          "type": "asciifolding",
          "preserve_original": true
        }
      },
      "analyzer": {
        "notes_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip_filter"],
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "ascii_fold",
            "english_stop",
            "english_stemmer",
            "order_synonyms"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "notes": {
        "type": "text",
        "analyzer": "notes_analyzer",
        "search_analyzer": "notes_analyzer"
      }
    }
  }
}

5.1.3 Verify with _analyze

Always confirm the pipeline produces the tokens you expect before indexing data:

POST /shop-orders-demo-v2/_analyze

{
  "analyzer": "notes_analyzer",
  "text": "<p>There was a <b>failure</b> in the payment gateway.</p>"
}

Expected output tokens (roughly): failure is first normalized to error by the synonym filter. HTML tags are stripped. Stopwords like “there”, “was”, “a”, “in”, “the” are removed. “payment” and “gateway” are stemmed.

{
  "tokens": [
    { "token": "error",    "position": 3 },
    { "token": "payment",  "position": 5 },
    { "token": "gatewai",  "position": 6 }
  ]
}

Tip: Use _analyze with "explain": true to see which filter produces each token — invaluable when debugging synonym or stemmer surprises.

5.1.4 Prove synonyms work end-to-end

Index a document, then search with the synonym term:

POST /shop-orders-demo-v2/_doc/1

{
  "notes": "There was a failure in the payment gateway."
}

GET /shop-orders-demo-v2/_search

{
  "query": {
    "match": {
      "notes": "error"
    }
  }
}

The document is returned because “failure” was stored as “error” at index time (expand style). If you used a search-time-only synonym filter, the query term “error” would expand to “failure” at search time instead — both approaches work, but they have different implications for index size and analyzer updates (see below).

5.1.5 Index-time vs search-time synonyms and search_analyzer

A field can have two different analyzers:

For synonyms, search-time expansion is usually preferable because you can update the synonym list without a full reindex:

POST /shop-orders-demo-v2/_reload_search_analyzers

Czech diacritics tip: Add asciifolding (with "preserve_original": true) to both index and search analyzers. This maps “č” → “c”, “ž” → “z”, etc., so users who omit diacritics still find documents that contain them — and vice-versa. Preserving the original keeps exact diacritic matches scoring higher.

5.1.6 Prefix / autocomplete tokenizers

For prefix-search use cases, the edge_ngram tokenizer (or token filter) is the right tool — covered in section 5.5. The ngram tokenizer generates all substrings (expensive). whitespace tokenizer skips punctuation-based splitting, useful for things like order IDs or product codes.

5.2 Modeling relationships: nested queries

5.2.1 Why flat object arrays cause cross-object false matches

When items is a plain object array, Elasticsearch internally flattens all values into parallel arrays:

items.category  → ["Elektronika", "Clothing"]
items.unit_price → [1200.00, 49.99]

A query asking for “category=Elektronika AND unit_price>1000 in the same item” will match a document where one item has category=Elektronika (unit_price=49.99) and another item has unit_price=1200 (category=Clothing) — a false positive.

nested type stores each array element as a hidden, separately indexed document that retains its own field grouping, eliminating cross-object correlation.

5.2.2 The nested query with inner_hits

GET /shop-orders-demo/_search

{
  "query": {
    "nested": {
      "path": "items",
      "query": {
        "bool": {
          "must": [
            { "term":  { "items.category": "Elektronika" } },
            { "range": { "items.unit_price": { "gt": 1000 } } }
          ]
        }
      },
      "inner_hits": {
        "name": "matching_items",
        "size": 5,
        "_source": ["items.name", "items.unit_price", "items.quantity"]
      }
    }
  },
  "_source": ["order_id", "customer.name", "order_status"]
}

inner_hits returns the specific nested items that matched — without it, you only know the parent document matched. Each inner hit also has its own _score relative to the nested query.

5.2.3 Sorting and filtering by nested fields

Sort orders by the maximum unit price of any matching item:

{
  "sort": [
    {
      "items.unit_price": {
        "order": "desc",
        "mode": "max",
        "nested": {
          "path": "items",
          "filter": {
            "term": { "items.category": "Elektronika" }
          }
        }
      }
    }
  ]
}

The nested sort context is required — Elasticsearch needs to know which nested path to navigate and optionally which subset of nested docs to consider.

Use nested in aggs the same way (covered in chapter 06).

5.2.4 Trade-offs

Pitfall: Declaring a field as nested after data is already indexed does nothing — nested documents are written at index time. You must reindex existing data after changing the mapping.

5.3 Modeling relationships: join / parent-child

5.3.1 The join field type

join implements a parent-child relationship within the same index. The child documents are independent Lucene documents (unlike nested, which are hidden docs). This means children can be updated without rewriting the parent.

Define the join field in mappings:

PUT /orders-with-events

{
  "mappings": {
    "properties": {
      "join_field": {
        "type": "join",
        "relations": {
          "order": "event"
        }
      },
      "order_id":    { "type": "keyword" },
      "event_type":  { "type": "keyword" },
      "event_at":    { "type": "date" }
    }
  }
}

Index a parent (order):

PUT /orders-with-events/_doc/order-1?routing=order-1

{
  "order_id": "order-1",
  "join_field": "order"
}

Index a child (event) — must use the same routing as its parent:

PUT /orders-with-events/_doc/event-1?routing=order-1

{
  "order_id": "order-1",
  "event_type": "shipped",
  "event_at": "2024-03-15T10:00:00Z",
  "join_field": {
    "name": "event",
    "parent": "order-1"
  }
}

Critical: Parent and child must reside on the same shard. This is enforced by routing. Forgetting ?routing= is the most common operational mistake with join fields.

5.3.2 has_child and has_parent queries

Find orders that have a “shipped” event:

GET /orders-with-events/_search

{
  "query": {
    "has_child": {
      "type": "event",
      "query": {
        "term": { "event_type": "shipped" }
      },
      "inner_hits": {}
    }
  }
}

Find all events for orders with order_status = completed:

{
  "query": {
    "has_parent": {
      "parent_type": "order",
      "query": {
        "term": { "order_status": "completed" }
      }
    }
  }
}

Tip: has_child with score_mode: "max" lets child scores influence the parent’s ranking — useful when you want orders ranked by how relevant their events are.

5.3.3 Nested vs join vs denormalization vs application-side join

Rule of thumb for make.com’s order domain: Denormalize aggressively. Embed items in the order document (as nested if you query across item fields, as plain objects if you don’t). Use join only when children are truly independent entities updated at very different rates (e.g., audit events appended to orders, where you never want to reindex the order to add an event).

5.4 Highlighting matches

Highlighting returns the fragment of text that caused the document to match, with match terms wrapped in markup — essential for any user-facing search UI.

5.4.1 Basic highlight

GET /shop-orders-demo/_search

{
  "query": {
    "match": { "notes": "payment failure" }
  },
  "highlight": {
    "fields": {
      "notes": {},
      "customer.name": {}
    }
  }
}

By default, matched terms are wrapped in <em> tags. The response includes a highlight object alongside each hit’s _source.

5.4.2 Custom tags and fragment control

{
  "highlight": {
    "pre_tags":  ["<b>"],
    "post_tags": ["</b>"],
    "fields": {
      "notes": {
        "fragment_size": 150,
        "number_of_fragments": 3,
        "no_match_size": 100
      }
    }
  }
}

5.4.3 Highlighting nested fields via inner_hits

Highlighting works inside inner_hits — you declare the highlight inside the nested query block:

{
  "query": {
    "nested": {
      "path": "items",
      "query": {
        "match": { "items.name": "kabel" }
      },
      "inner_hits": {
        "highlight": {
          "fields": {
            "items.name": {
              "fragment_size": 80,
              "number_of_fragments": 1
            }
          }
        }
      }
    }
  }
}

5.4.4 Highlighter types

Switch per field:

"notes": {
  "type": "fvh"
}

Pitfall: Highlighting requires the field’s text to be available — either from _source (default) or from stored fields. If you set "_source": false globally, highlighting fails unless fields are explicitly stored. Also: highlighting keyword fields is a no-op — only analyzed text fields produce sensible fragments.

5.5 Suggest / completion — autocomplete and typo tolerance

5.5.1 Suggesters overview

Elasticsearch has three suggester types with very different use cases:

5.5.2 Term and phrase suggesters (spell-check style)

Use when a search returns zero results and you want to suggest “Did you mean…?”:

GET /shop-orders-demo/_search

{
  "query": { "match_none": {} },
  "suggest": {
    "notes_suggestion": {
      "text": "payemnt failurr",
      "phrase": {
        "field": "notes",
        "size": 3,
        "gram_size": 2,
        "highlight": {
          "pre_tag": "<em>",
          "post_tag": "</em>"
        },
        "collate": {
          "query": {
            "source": {
              "match": { "notes": "{{suggestion}}" }
            }
          },
          "prune": true
        }
      }
    }
  }
}

collate verifies each suggestion actually returns results before offering it — avoids “Did you mean X?” when X also returns nothing.

5.5.3 Completion suggester — prefix autocomplete

The completion suggester uses an in-memory Finite State Transducer (FST) for O(1) prefix lookup. It requires a dedicated field type in the mapping.

Mapping:

PUT /product-suggestions

{
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion",
        "analyzer": "simple"
      },
      "product_id": { "type": "keyword" },
      "name":       { "type": "text" }
    }
  }
}

Indexing with weights (for popularity-based ranking):

POST /product-suggestions/_doc

{
  "product_id": "EL-001",
  "name": "USB-C Charging Cable",
  "suggest": {
    "input": ["USB-C Charging Cable", "USB Cable", "Charging Cable"],
    "weight": 85
  }
}

Querying:

GET /product-suggestions/_search

{
  "suggest": {
    "product_autocomplete": {
      "prefix": "usb",
      "completion": {
        "field": "suggest",
        "size": 5,
        "skip_duplicates": true,
        "fuzzy": {
          "fuzziness": "AUTO",
          "min_length": 4,
          "prefix_length": 2
        }
      }
    }
  }
}

fuzzy on completion allows one edit for typos while still using the FST — at the cost of needing to walk more branches.

5.5.4 search_as_you_type and edge-ngram alternatives

search_as_you_type example:

{
  "mappings": {
    "properties": {
      "name": {
        "type": "search_as_you_type"
      }
    }
  }
}

Query it with multi_match across the auto-generated sub-fields:

{
  "query": {
    "multi_match": {
      "query": "usb cab",
      "type": "bool_prefix",
      "fields": ["name", "name._2gram", "name._3gram"]
    }
  }
}

Decision guide: - High-volume autocomplete with popularity ranking → completion - Autocomplete that must incorporate relevance signals from other fields → search_as_you_type or edge_ngram - Spell correction for failed queries → phrase suggester with collate

5.6 function_score and script_score — custom relevance

5.6.1 Why custom relevance?

BM25 measures textual similarity. Business relevance often adds non-textual signals: product rating, recency, geographic proximity, promotion priority, popularity. function_score and script_score let you blend these signals mathematically.

5.6.2 function_score

function_score wraps a base query and applies one or more scoring functions. The final score is controlled by boost_mode (how function scores combine with the base _score) and score_mode (how multiple functions combine with each other).

GET /shop-orders-demo/_search

{
  "query": {
    "function_score": {
      "query": {
        "match": { "notes": "damaged packaging" }
      },
      "functions": [
        {
          "field_value_factor": {
            "field": "rating",
            "factor": 1.5,
            "modifier": "log1p",
            "missing": 1
          }
        },
        {
          "gauss": {
            "created_at": {
              "origin": "now",
              "scale":  "30d",
              "offset": "7d",
              "decay":  0.5
            }
          }
        },
        {
          "gauss": {
            "delivery.location": {
              "origin": "50.0755,14.4378",
              "scale":  "50km",
              "offset": "5km",
              "decay":  0.5
            }
          }
        },
        {
          "filter": { "term": { "tags": "priority" } },
          "weight": 3.0
        }
      ],
      "score_mode": "multiply",
      "boost_mode": "multiply",
      "max_boost": 10
    }
  }
}

field_value_factor parameters:

Decay functions (gauss, linear, exp):

score_mode (how the list of function scores combine): multiply, sum, avg, first, max, min.

boost_mode (how combined function score interacts with base _score): multiply, replace, sum, avg, max, min.

Tip: Use "boost_mode": "sum" when you want the function score to add to relevance rather than override it — useful when text match should still matter even if the item is not perfectly boosted.

5.6.3 script_score

When the built-in functions are not expressive enough, use script_score with a Painless script:

GET /shop-orders-demo/_search

{
  "query": {
    "script_score": {
      "query": {
        "match": { "notes": "missing item" }
      },
      "script": {
        "source": """
          double rating = doc['rating'].size() > 0 ? doc['rating'].value : 1.0;
          double recencyBoost = 1.0;
          if (doc['created_at'].size() > 0) {
            long ageMs = System.currentTimeMillis() - doc['created_at'].value.toInstant().toEpochMilli();
            long ageDays = ageMs / 86400000L;
            recencyBoost = Math.max(0.1, 1.0 / (1.0 + ageDays / 30.0));
          }
          return _score * Math.log(2 + rating) * recencyBoost;
        """
      }
    }
  }
}

Rules for performant Painless scripts:

Parameterized version (recommended):

{
  "query": {
    "script_score": {
      "query": { "match": { "notes": "missing item" } },
      "script": {
        "source": "return _score * Math.log(2 + (doc['rating'].size() > 0 ? doc['rating'].value : params.default_rating));",
        "params": {
          "default_rating": 1.0
        }
      }
    }
  }
}

5.6.4 function_score vs script_score — when to use which

Pitfall: Scores from script_score must be non-negative. If your formula can produce negative values, Elasticsearch will throw an error. Guard with Math.max(0, ...).

Pitfall: function_score with random_score uses the shard ID and document ID by default. For consistent random ordering across paginated results, provide a fixed seed and field parameter — otherwise the order changes on each query.

{
  "random_score": {
    "seed": 42,
    "field": "_seq_no"
  }
}

Labs

Lab 1 — Custom analyzer with synonyms and diacritics

1. Create a new index with a custom analyzer that combines html_strip, lowercase, asciifolding (preserve original), and a synonym filter mapping at least 3 domain-specific synonyms from the order domain (e.g. "refund, return => return", "cancelled, canceled => cancelled").
2. Map the notes field to use this analyzer at index time and a version without html_strip as search_analyzer.
3. Use _analyze to verify that HTML-tagged input produces the correct tokens.
4. Index two documents — one using a synonym source word, one using the target word — and confirm both are found by querying either term.

Takeaway: Index-time and search-time analyzers serve different purposes. Synonym expansion at search time is cheaper to maintain but requires _reload_search_analyzers after updates.

Lab 2 — nested query with inner_hits and sorting

1. Query shop-orders-demo for orders that contain at least one item where category = "Elektronika" AND unit_price > 500. Use a nested query with inner_hits to identify which specific items matched.
2. Sort the results by the maximum unit_price of matching items (descending), scoping the sort to the Elektronika category only.
3. Add a highlight on items.name inside inner_hits.
4. Compare the result count with a naïve bool query on the flattened fields (without nested). Observe the false positives.

Takeaway: nested prevents cross-object false matches but adds query cost. Always benchmark before choosing between nested and denormalization.

Lab 3 — Completion suggester

1. Create a product-suggestions index with a completion-type field named suggest.
2. Index 10 products from the shop-orders-demo item names as suggestions with varying weight values (simulate popularity).
3. Issue a prefix query for “usb” and observe ordering by weight.
4. Enable fuzzy on the completion query and test with a one-character typo (“uab” for “usb”). Observe the fuzziness threshold behavior.
5. Repeat the same prefix search using search_as_you_type on a separate index and compare index size and query response time.

Takeaway: completion is fastest for pure autocomplete with popularity ranking. search_as_you_type is more flexible but carries a larger index footprint.

Lab 4 — Custom relevance with function_score and script_score

1. Write a function_score query matching notes for “delayed” that boosts by:
   • field_value_factor on rating (modifier log1p, factor 2.0).
   • A gauss decay on created_at (origin now, scale 14d).
   • A weight of 5.0 for documents tagged “priority”.
2. Use "boost_mode": "sum" and "score_mode": "sum". Observe how different boost_mode values change the ranking.
3. Rewrite the same logic as a script_score query in Painless. Guard all field accesses against missing values.
4. Use the explain API (GET /shop-orders-demo/_explain/<id>) to inspect the score breakdown for both queries on the same document.

Takeaway: function_score covers most production relevance needs declaratively. Reach for script_score only when you need logic that the built-in functions cannot express — and always profile the impact on query latency.

Summary