01 — Introduction and architecture

Welcome to Day 1. This chapter assumes you have never used Elasticsearch before. We will build a mental model of what Elasticsearch is, how it stores data, and how to get a local environment running — everything you need before writing your first query.

Goals for this chapter

By the end of this chapter you should be able to:

1.1 What Elasticsearch is and when to use it

The short answer

Elasticsearch is an open-source, distributed search and analytics engine. You send it JSON documents, it indexes them, and you can search and aggregate them at scale through a simple HTTP/JSON API.

Think of it as a highly specialized system that sits next to your primary database, holds a searchable copy of your data, and answers questions your primary database struggles with — such as “find all orders containing a Czech product name that sounds roughly like this spelling” or “show me a live breakdown of revenue by city for the last 7 days.”

What Elasticsearch is great at

Capability Example
Full-text search with relevance ranking “find me all orders whose notes mention ‘poškozený obal’”
Fast filtering on structured fields orders from channel web, paid with card, shipped to CZ
Aggregations and analytics average order value by loyalty tier, histogram of orders per hour
Scale — hundreds of millions of documents log pipelines, e-commerce catalogues, clickstream
Near real-time — new data visible in ~1 second monitoring dashboards, live order tracking
Geo queries find all delivery locations within 50 km of Prague

What Elasticsearch is NOT

Comparison: Elasticsearch vs. relational database

Dimension Relational DB (PostgreSQL, MySQL…) Elasticsearch
Schema Fixed schema, DDL required to change Flexible by default; mappings can be defined or auto-detected
Data model Tables, rows, columns, foreign keys Indices, documents (JSON), fields
Joins Native, powerful (JOINs, FK constraints) Very limited (nested, has_child); denormalize instead
Transactions Full ACID, multi-table Single-document atomicity only
Full-text search Possible (tsvector) but not the primary use-case Core competency — relevance, analyzers, stemming
Aggregations SQL GROUP BY, window functions Built-in aggregation framework (fast, approximate where needed)
Scaling writes Vertical (+ read replicas) Horizontal sharding by default
Scaling reads Read replicas More shards + more replicas
Primary data store Yes — designed for it No — usually secondary/derived

“Use Elasticsearch when…” vs “Don’t use it as your only store when…”

Use Elasticsearch when: - Users need full-text search with typo tolerance, synonyms, or multi-language support - You need fast aggregations over millions of events (logs, orders, clicks) - You have geo data and need proximity/bounding-box queries - Your relational DB is choking on complex LIKE '%term%' queries - You need a faceted search UI (filter by category + price range + rating simultaneously)

Don’t use Elasticsearch as your only data store when: - You need strict ACID guarantees (financial transactions, inventory mutations) - Your data model requires complex multi-table joins - You need guaranteed durability without a separate backup/rebuild pipeline - Your team is small and the operational overhead of running a cluster is too high for the value

1.2 Core concepts: cluster, node, index, shard, replica

Every term below maps to something you already know from relational databases. We’ll use those analogies throughout the course.

Cluster

A cluster is a named group of Elasticsearch nodes (processes) that work together. They share the same cluster state — which indices exist, how many shards they have, and which node holds which shard.

Every cluster has a name (default: elasticsearch). All nodes in the same cluster must share that name.

You can think of a cluster the way you think of a database server installation — but one that can span many machines.

Node

A node is a single running Elasticsearch process (a JVM process). One node = one machine or one container. Nodes have roles:

Role What it does
Master Manages cluster state (index creation, shard assignment)
Data Stores shards, executes queries and indexing
Coordinating Routes requests, merges shard results — every node can do this

For local development we run a single-node cluster where one node plays all roles. That is fine for learning.

Index

An index is a named collection of documents plus its mapping (field types) and settings (number of shards, analyzers, etc.).

Relational analogy: index ≈ table

Example indices we will use in this course: - shop-orders-demo — our main e-commerce orders dataset

Document

A document is a single JSON record stored in an index. Every document has a unique _id (auto-generated or provided by you).

Relational analogy: document ≈ row

{
  "_index": "shop-orders-demo",
  "_id": "ord-1001",
  "_source": {
    "order_id": "ord-1001",
    "created_at": "2024-03-15T09:23:00Z",
    "channel": "web",
    "total_amount": 1290.00,
    "customer": {
      "name": "Jana Nováková",
      "loyalty_tier": "gold"
    }
  }
}

A field in a document is analogous to a column in a row.

Shard

Here is where Elasticsearch diverges from a relational database. An index is not stored as a single unit — it is split into shards.

A shard is the actual unit of storage and parallelism. Each shard is internally a complete Lucene index (more on Lucene in section 1.3).

The number of primary shards cannot be changed after the index is created (without reindexing). Choose carefully — chapter 07 covers shard sizing.

Replica

A replica shard is a copy of a primary shard. Replicas provide:

By default, Elasticsearch creates 1 replica per primary shard. You can change this at any time.

Putting it all together — a text diagram

Here is a 3-node cluster with one index my-index, 3 primary shards (P0, P1, P2), and 1 replica each (R0, R1, R2):

Cluster: my-cluster
├── Node 1
│   ├── P0  (primary shard 0 of my-index)
│   └── R1  (replica of shard 1)
├── Node 2
│   ├── P1  (primary shard 1 of my-index)
│   └── R2  (replica of shard 2)
└── Node 3
    ├── P2  (primary shard 2 of my-index)
    └── R0  (replica of shard 0)

Key observations: - Each primary and its replica live on different nodes (Elasticsearch enforces this automatically) - If Node 1 goes down, R1 on Node 2 is promoted to primary — no data loss - A search on my-index fans out to all 3 shards (P0, P1, P2), results are merged and returned

One-node cluster caveat: if you only have one node (like in our Docker Compose lab below), replicas will remain unassigned (yellow cluster health). That is expected — there is nowhere to put the replica copy. It does not affect functionality for local development.

1.3 How Elasticsearch stores data: Lucene, the inverted index, and segments

This section explains the “magic” behind why Elasticsearch can find a word inside millions of documents in milliseconds.

The inverted index — a concrete example

A traditional database stores data row by row. To find a word you scan rows. That is slow for text.

Elasticsearch (via Lucene) builds an inverted index: instead of “for each document, what words does it contain?”, it stores “for each word, which documents contain it?”

Let’s say we have three tiny documents:

doc id text
1 “Bluetooth sluchátka Sony”
2 “Sony televizor 4K”
3 “Sluchátka s mikrofonem”

After indexing and analysis (lowercasing, tokenizing), the inverted index looks like this:

Term Posting list (document IDs)
bluetooth [1]
sluchatka [1, 3]
sony [1, 2]
televizor [2]
4k [2]
mikrofonem [3]

When a user searches for sony sluchatka, Elasticsearch: 1. Looks up both terms in the inverted index — O(1) lookup, like a hash map 2. Finds docs that match: sony → [1, 2], sluchatka → [1, 3] 3. Intersects or scores them — doc 1 appears in both, so it ranks highest 4. Returns results in milliseconds even across millions of documents

The inverted index is why Elasticsearch is fast at text search. A relational DB doing WHERE notes LIKE '%sony%' scans every row. Elasticsearch does a dictionary lookup.

Analyzers — from raw text to terms

Before text is stored in the inverted index it goes through an analyzer, which typically does: 1. Tokenize — split “Bluetooth sluchátka Sony” into ["Bluetooth", "sluchátka", "Sony"] 2. Normalize — lowercase → ["bluetooth", "sluchátka", "sony"] 3. Filter — remove stop words, apply stemming, etc.

We will explore analyzers in depth in chapter 04 (Mapping and analyzers).

Segments — immutable building blocks

Each Elasticsearch shard is a Lucene index made up of segments. A segment is an immutable mini-inverted-index written to disk.

Shard (Lucene index)
├── segment_1  (100 docs, written 09:00)
├── segment_2  (50 docs, written 09:05)
├── segment_3  (200 docs, written 09:10)
│         ↓  (merge happens)
└── segment_4  (350 docs, merged result)

Refresh and near real-time (NRT)

After you index a document it is NOT immediately searchable. It sits in an in-memory buffer. Once per refresh interval (default: 1 second), Elasticsearch:

  1. Flushes the in-memory buffer into a new segment on disk
  2. Opens the segment so it becomes visible to searches

This is why Elasticsearch is called near real-time: new data is visible within about 1 second, not immediately.

Pitfall: if you index a document and immediately search for it in the same test, you may get 0 results. Always wait for refresh, or manually call POST /my-index/_refresh in your tests. We will practice this in Labs.

You can tune the refresh interval:

PUT /my-index/_settings
{
  "index": {
    "refresh_interval": "5s"
  }
}

Set it to "-1" to disable automatic refresh during heavy bulk loads (then re-enable it afterwards) — this is one of the most effective bulk-indexing optimizations.

1.4 Installing and running a local environment with Docker Compose

We will run a single-node Elasticsearch + Kibana stack locally using Docker Compose. Security is disabled to keep the local setup simple — never disable security in production.

Prerequisites

docker-compose.yml

Create a file named docker-compose.yml in a new folder (e.g. ~/es-training/):

version: "3.8"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.4.2
    container_name: es01
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
      - cluster.routing.allocation.disk.threshold_enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:9.4.2
    container_name: kibana01
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

volumes:
  es-data:
    driver: local

Note on versions: We use 9.4.2 above. Swap in the latest 8.x or 9.x tag — just make sure ES and Kibana versions match exactly. Check https://www.docker.elastic.co for available tags.

RAM: The ES_JAVA_OPTS=-Xms1g -Xmx1g setting gives the JVM 1 GB heap. For small datasets that is fine. If you see out-of-memory errors, increase to -Xms2g -Xmx2g and make sure Docker has enough RAM allocated.

Start the stack

docker compose up -d

Elasticsearch takes 20–40 seconds to start. Check the logs if needed:

docker compose logs -f elasticsearch

Look for the line: started or recovered [0] indices.

Verify Elasticsearch is running

curl -s http://localhost:9200

You should see a JSON response like:

{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "version" : {
    "number" : "9.4.2",
    ...
  },
  "tagline" : "You Know, for Search"
}

Open Kibana

Open your browser and navigate to: http://localhost:5601.

Kibana may take a minute to start. Once loaded, go to:

Hamburger menu → Management → Dev Tools

This is where we will run all our API calls throughout the course.

Alternative: Elastic Cloud If you prefer not to run Docker locally, Elastic Cloud (https://cloud.elastic.co) offers a free trial with a hosted cluster. Everything in this course works identically against a cloud cluster — just replace localhost:9200 with your cloud endpoint and add authentication headers.

Stop the stack (when done)

docker compose down

Add -v to also delete the stored data volume:

docker compose down -v

1.5 First contact with the REST API via Kibana Dev Tools

The Dev Tools console

In Kibana, go to Management → Dev Tools (or click the wrench icon in the left sidebar).

You will see a split panel: - Left side: where you type requests - Right side: where responses appear

Click the green triangle (▶) next to a request to execute it, or press Ctrl+Enter (Cmd+Enter on Mac).

The request format

Every Elasticsearch REST API call has this shape:

METHOD /path
{
  optional JSON body
}

The same JSON you type in Dev Tools works identically from curl, the JavaScript client, the Python client, or any HTTP library — ES is just HTTP+JSON.

Your first API calls

1. Check if Elasticsearch is responding

GET /

Returns cluster name, version, and the famous tagline.

2. Check cluster health

GET /_cluster/health

Look at the status field: - green — all primary and replica shards are assigned - yellow — all primary shards assigned, but some replicas are not (expected on a single-node cluster) - red — some primary shards are not assigned (data may be unavailable)

On your fresh single-node cluster you will see yellow — that is normal and expected (no second node to hold replicas).

3. List nodes

GET /_cat/nodes?v

The ?v adds column headers. You will see one row — your single es01 node. Notice the heap.percent, cpu, and role columns.

4. List indices

GET /_cat/indices?v

On a fresh cluster this returns system indices (.kibana_*, etc.). Later, once we load shop-orders-demo, you will see it listed here with its document count, store size, and health.

5. A quick preview: index and search a tiny document

Let’s index one document to confirm everything works end-to-end:

POST /test-hello/_doc/1
{
  "message": "Hello Elasticsearch!"
}

Now search for it:

GET /test-hello/_search
{
  "query": {
    "match": {
      "message": "hello"
    }
  }
}

You should see your document in the hits.hits array. If you get 0 hits, wait a second and retry — recall the near-real-time refresh from section 1.3.

Clean up:

DELETE /test-hello

Tip: The _cat APIs (like _cat/nodes, _cat/indices, _cat/shards) are designed for human-readable terminal output. For programmatic use, always use the structured JSON endpoints instead (e.g. GET /_cluster/health). We use _cat APIs here because they are easy to read at a glance.

The same call from curl (for reference)

Everything you type in Dev Tools can be run from the terminal:

curl -s -X GET "http://localhost:9200/_cluster/health" | python3 -m json.tool

Or with the official Python client (preview):

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")
print(es.cluster.health())

We use Dev Tools throughout the course because it has autocomplete and makes it easy to focus on the query syntax rather than HTTP boilerplate.

Labs

Lab 1 — Explore the cluster

Goal: get comfortable with the Dev Tools console and the cat APIs.

Open Kibana Dev Tools and run the following calls one by one. For each, read the output and note what you find.

GET /
GET /_cluster/health
GET /_cat/nodes?v
GET /_cat/indices?v
GET /_cat/shards?v

Questions to answer from the output:

  1. What is the cluster name?

  2. What is the cluster health status? Why is it yellow?

  3. How many nodes does your cluster have? What roles does the node have?

  4. Do you see any unassigned shards? On which index?

Takeaway: The cat APIs give you a quick operational overview of your cluster. You will use _cluster/health and _cat/shards every time something looks wrong.

Lab 2 — Index a document and observe near real-time refresh

Goal: see the refresh delay with your own eyes.

Step 1: Set a slow refresh interval on a new index.

PUT /nrt-demo
{
  "settings": {
    "index": {
      "refresh_interval": "30s",
      "number_of_replicas": 0
    }
  }
}

Step 2: Index a document.

POST /nrt-demo/_doc/1
{
  "product": "Wireless Mouse Logitech",
  "price": 21
}

Step 3: Search immediately — you should get 0 hits.

GET /nrt-demo/_search
{
  "query": { "match_all": {} }
}

Step 4: Manually trigger a refresh.

POST /nrt-demo/_refresh

Step 5: Search again — now you should see the document.

GET /nrt-demo/_search
{
  "query": { "match_all": {} }
}

Step 6: Clean up.

DELETE /nrt-demo

Takeaway: Documents are only visible after a refresh. In tests and scripts, always call _refresh after indexing before asserting on search results.

Lab 3 — Understand shard and replica distribution

Goal: see how shard settings affect cluster health.

Step 1: Create an index with 2 replicas on your single-node cluster.

PUT /shard-demo
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 2
  }
}

Step 2: Check cluster health — it should be yellow (replicas cannot be assigned).

GET /_cluster/health

Step 3: Look at the shard list to see which shards are unassigned.

GET /_cat/shards/shard-demo?v

Look at the state column. Primary shards (p) should be STARTED, replicas (r) should be UNASSIGNED.

Step 4: Reduce replicas to 0 — cluster should turn green.

PUT /shard-demo/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}
GET /_cluster/health

Step 5: Clean up.

DELETE /shard-demo

Takeaway: On a single-node cluster, set number_of_replicas: 0 for local development to keep the cluster green. In production with multiple nodes, replicas provide both HA and read scaling.

Lab 4 — Preview the course dataset

Goal: get a feel for the shop-orders-demo dataset you will use throughout the rest of the course.

Your instructor will load the shop-orders-demo index before this lab. If it is not yet available, skip to the next chapter and return here.

Step 1: Confirm the index exists and see its document count.

GET /_cat/indices/shop-orders-demo?v

Step 2: Look at the mapping (field types — we cover this in chapter 04).

GET /shop-orders-demo/_mapping

Note the fields: order_id, created_at, channel, total_amount, customer, items, etc.

Step 3: Retrieve one document to see what the data looks like.

GET /shop-orders-demo/_search
{
  "size": 1,
  "query": { "match_all": {} }
}

Look at the _source field in the response. You will recognize the structure from section 1.3 — nested customer, nested items, geo delivery.location, and so on.

Takeaway: Understanding the shape of your data before writing queries is essential. Always start with _mapping and a sample document.

Summary