02 — Data model and CRUD

Before you can search anything, you need to store it. This chapter walks you through the mental model Elasticsearch uses to organize data and shows you every way to create, read, update, and delete documents — one at a time and in bulk. If you have ever worked with a relational database, most concepts will feel familiar, just wearing different clothes.

Goals for this chapter

1. Understand what an index, a document, a field, and a mapping are, and how they map to RDBMS concepts.
2. Index documents with PUT and POST; understand the response envelope.
3. Read single documents by ID and multiple documents at once.
4. Perform partial updates, scripted updates, and bulk updates.
5. Delete single documents and documents matching a query.
6. Use the Bulk API for efficient batch ingestion.
7. Understand the difference between dynamic and explicit mapping, and know why it matters.

2.1 Document, index, mapping — the RDBMS analogy

Elasticsearch stores data as JSON documents inside indices. If you are coming from SQL, the mapping below is a useful starting point — just keep in mind it is an analogy, not a perfect equivalence.

What is a document?

A document is a JSON object. It can be nested (objects inside objects), contain arrays, and even hold geo-coordinates or date ranges. Here is a minimal example:

{
  "order_id": "ORD-0001",
  "customer_name": "Jana Nováková",
  "total_amount": 1299.90,
  "created_at": "2025-03-15T10:22:00Z"
}

Every document stored by Elasticsearch comes back wrapped in a metadata envelope. The key metadata fields are:

What is a mapping?

A mapping is the schema for an index. It tells Elasticsearch what data type each field holds — text, keyword, date, number, geo_point, nested object, and so on. Elasticsearch can guess types automatically the first time a document is indexed (dynamic mapping), or you can define them explicitly upfront. We will cover all the details in chapter 04 — Mapping and analyzers; in this chapter we just touch on the basics so you know what is happening when you index your first document.

2.2 Indexing documents — PUT vs POST

PUT — you provide the ID

Use PUT when you already have a meaningful identifier (e.g., your database primary key, a UUID from your application).

PUT /shop-orders-demo/_doc/ORD-0001

{
  "order_id": "ORD-0001",
  "created_at": "2025-03-15T10:22:00Z",
  "channel": "web",
  "currency": "CZK",
  "payment_method": "card",
  "order_status": "delivered",
  "shipping_method": "courier",
  "subtotal_amount": 1199.00,
  "discount_amount": 0.00,
  "shipping_fee": 99.00,
  "total_amount": 1299.00,
  "customer": {
    "id": "CUST-42",
    "name": "Jana Nováková",
    "loyalty_tier": "gold"
  },
  "delivery": {
    "city": "Praha",
    "country_code": "CZ",
    "days": 2
  },
  "items": [
    {
      "product_id": "PROD-888",
      "name": "Wireless Mouse Logitech MX Master 3",
      "category": "electronics",
      "unit_price": 1199.00,
      "quantity": 1
    }
  ],
  "categories": ["electronics"],
  "returned": false,
  "rating": 4.8
}

Typical response:

{
  "_index": "shop-orders-demo",
  "_id": "ORD-0001",
  "_version": 1,
  "result": "created",
  "_shards": { "total": 2, "successful": 1, "failed": 0 },
  "_seq_no": 0,
  "_primary_term": 1
}

The result field tells you whether this was a fresh created or an updated (the whole document was replaced). Re-run the same request and the version bumps to 2 and result becomes updated.

Pitfall — PUT replaces the whole document. If you PUT {"rating": 5.0} against an existing ID, you end up with a document that has only one field. Everything else is gone. Use _update (section 2.4) for partial edits.

POST — Elasticsearch generates the ID

When you do not have an ID ready, use POST without an ID in the path:

POST /shop-orders-demo/_doc

{
  "order_id": "ORD-0002",
  "created_at": "2025-03-16T08:05:00Z",
  "channel": "store",
  "total_amount": 599.00,
  "order_status": "processing"
}

Response:

{
  "_index": "shop-orders-demo",
  "_id": "A1b2C3dEfGhIjKlM",
  "_version": 1,
  "result": "created",
  ...
}

The auto-generated ID is a random URL-safe Base64 string. It is unique and safe to use, but you lose control over it — you will need to store it somewhere if you want to reference it later.

PUT /_create/<id> — fail if the document already exists

Sometimes you want to guarantee you are not accidentally overwriting an existing document. Use the _create endpoint:

PUT /shop-orders-demo/_create/ORD-0001

{
  "order_id": "ORD-0001",
  "total_amount": 999.00
}

If ORD-0001 already exists, Elasticsearch returns 409 Conflict with "result": "version_conflict_engine_exception". No data is changed.

You can achieve the same with POST using the op_type query parameter:

POST /shop-orders-demo/_doc?op_type=create

Tip — when to use _create: any time your business logic requires that an ID must be unique and must not be silently overwritten. Think of it as INSERT vs INSERT OR REPLACE in SQL.

2.3 Reading documents

GET by ID

GET /shop-orders-demo/_doc/ORD-0001

Response:

{
  "_index": "shop-orders-demo",
  "_id": "ORD-0001",
  "_version": 2,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "order_id": "ORD-0001",
    "total_amount": 1299.00,
    ...
  }
}

If the document does not exist, "found": false and HTTP status 404.

GET only _source

If you just want the raw JSON without the metadata envelope:

GET /shop-orders-demo/_source/ORD-0001

Returns the _source object directly — handy when your code just needs the data.

HEAD — check existence without fetching

HEAD /shop-orders-demo/_doc/ORD-0001

Returns 200 OK if found, 404 if not. No body. Useful for existence checks without transferring data.

_mget — fetch multiple documents in one round trip

GET /_mget

{
  "docs": [
    { "_index": "shop-orders-demo", "_id": "ORD-0001" },
    { "_index": "shop-orders-demo", "_id": "ORD-0002" },
    { "_index": "shop-orders-demo", "_id": "ORD-DOES-NOT-EXIST" }
  ]
}

Each entry in the docs array of the response has its own found flag. Missing documents do not cause an error — they simply return "found": false. This lets you fetch many IDs in a single HTTP call without a round-trip per document.

When to use GET-by-ID vs search: use GET /_doc/<id> when you already know the exact ID (e.g., you stored it in your database). When you need to find documents matching criteria, use the Search API — covered in chapter 03 — Basic search.

2.4 Updating documents

Full replace (PUT)

As mentioned in section 2.2, PUT /index/_doc/<id> replaces the entire document. Use this when you have the complete document and want a clean overwrite.

Partial update

To change only specific fields without rewriting the whole document, use _update:

POST /shop-orders-demo/_update/ORD-0001

{
  "doc": {
    "order_status": "returned",
    "returned": true
  }
}

Elasticsearch fetches the existing document, merges the fields you provided into _source, re-indexes the result, and marks the old version for deletion. All other fields remain untouched.

Response:

{
  "_index": "shop-orders-demo",
  "_id": "ORD-0001",
  "_version": 3,
  "result": "updated",
  ...
}

If you send a doc that contains exactly the same values already stored, Elasticsearch detects no change and responds with "result": "noop" — no new version is created. This is good for performance.

Scripted update

For more complex logic — incrementing a counter, adding to an array, conditional updates — use a Painless script:

POST /shop-orders-demo/_update/ORD-0001

{
  "script": {
    "source": "ctx._source.rating = params.new_rating; ctx._source.notes = params.note",
    "lang": "painless",
    "params": {
      "new_rating": 5.0,
      "note": "Zákazník přidal hodnocení po doručení"
    }
  }
}

ctx._source gives you access to the current document. Use params to pass values in instead of hardcoding them in the script — it avoids script recompilation on every call and is safer.

A common pattern — incrementing a field:

POST /shop-orders-demo/_update/ORD-0001

{
  "script": {
    "source": "ctx._source.delivery.days += params.extra_days",
    "params": { "extra_days": 1 }
  }
}

Upsert — update or create

What if the document might not exist yet? Use upsert:

POST /shop-orders-demo/_update/ORD-9999

{
  "doc": {
    "order_status": "pending",
    "total_amount": 0.00
  },
  "doc_as_upsert": true
}

If ORD-9999 exists, the fields in doc are merged in. If it does not exist, the doc becomes the new document. Equivalent to SQL INSERT ... ON CONFLICT DO UPDATE.

_update_by_query — update many documents matching a query

POST /shop-orders-demo/_update_by_query

{
  "query": {
    "term": { "order_status": "processing" }
  },
  "script": {
    "source": "ctx._source.order_status = 'on_hold'",
    "lang": "painless"
  }
}

This is the Elasticsearch equivalent of UPDATE ... WHERE .... It runs a search, then applies the script to each matching document. The response tells you how many documents were updated, how many failed, and how long it took.

Under the hood — documents are immutable. Elasticsearch segments (the files on disk that store documents) are never modified in place. Every update is actually: fetch the old document → apply changes → write a new version → mark the old version as deleted. The deleted documents are cleaned up lazily during segment merges (introduced in chapter 01 — Introduction and architecture). This design is why updates have slightly more overhead than raw inserts, and why index size can temporarily grow after heavy updates.

Tip — optimistic concurrency control. If multiple processes might update the same document simultaneously, use if_seq_no and if_primary_term to prevent lost updates:

POST /shop-orders-demo/_update/ORD-0001?if_seq_no=1&if_primary_term=1

{ "doc": { "order_status": "shipped" } }

If the document has been modified since you last read it (seq_no no longer matches), Elasticsearch returns 409 Conflict instead of silently overwriting. This is the equivalent of optimistic locking in application code.

2.5 Deleting documents

Delete by ID

DELETE /shop-orders-demo/_doc/ORD-0001

Response:

{
  "_index": "shop-orders-demo",
  "_id": "ORD-0001",
  "_version": 4,
  "result": "deleted",
  ...
}

Deleting a non-existent document returns "result": "not_found" and HTTP 404 — no error, just an informative response.

_delete_by_query — delete many documents matching a query

POST /shop-orders-demo/_delete_by_query

{
  "query": {
    "term": { "channel": "store" }
  }
}

The response includes "deleted": <count>, "failures": [], and timing information. Like _update_by_query, this runs a search first, then deletes each match.

Pitfall — deletes are soft until merge. A deleted document is still physically on disk and still consumes disk space until the segment it lives in is merged away. If you delete large amounts of data and need the space back immediately, you can trigger a _forcemerge — but that is an advanced operation with trade-offs, covered in chapter 07 — Performance and scaling. For routine housekeeping, just let Elasticsearch merge in the background.

Pitfall — _delete_by_query and concurrency. If documents matching your query are being modified by another process while the delete runs, some matches may be missed or may fail. Check the failures array in the response and consider using conflicts=proceed to skip version conflicts rather than aborting.

2.6 The Bulk API — batch operations

Round-tripping to Elasticsearch one document at a time is slow. The Bulk API lets you send thousands of operations in a single HTTP request. This is the standard way to load large datasets — including how the shop-orders-demo index is populated.

Format

The request body is newline-delimited JSON (NDJSON). Each operation consists of two lines:

1. An action line — a JSON object with one key: index, create, update, or delete.
2. A source line — the document body (not required for delete).

POST /_bulk
Content-Type: application/x-ndjson

{ "index": { "_index": "shop-orders-demo", "_id": "ORD-0010" } }
{ "order_id": "ORD-0010", "created_at": "2025-04-01T09:00:00Z", "channel": "web", "currency": "CZK", "total_amount": 450.00, "order_status": "delivered", "customer": { "id": "CUST-01", "name": "Petr Svoboda", "loyalty_tier": "standard" }, "returned": false }
{ "create": { "_index": "shop-orders-demo", "_id": "ORD-0011" } }
{ "order_id": "ORD-0011", "created_at": "2025-04-02T11:30:00Z", "channel": "store", "currency": "CZK", "total_amount": 899.00, "order_status": "processing", "customer": { "id": "CUST-02", "name": "Markéta Horáková", "loyalty_tier": "silver" }, "returned": false }
{ "update": { "_index": "shop-orders-demo", "_id": "ORD-0010" } }
{ "doc": { "order_status": "returned", "returned": true } }
{ "delete": { "_index": "shop-orders-demo", "_id": "ORD-0002" } }

Critical — the trailing newline. The body must end with a \n character. In Kibana Dev Tools this is handled automatically. In raw HTTP clients (curl, Postman), you must include it or you will get a parse error.

If all operations share the same index, you can set it in the URL and omit _index from each action line:

POST /shop-orders-demo/_bulk

Reading the bulk response

{
  "took": 12,
  "errors": true,
  "items": [
    { "index":  { "_id": "ORD-0010", "result": "created",  "status": 201 } },
    { "create": { "_id": "ORD-0011", "result": "created",  "status": 201 } },
    { "update": { "_id": "ORD-0010", "result": "updated",  "status": 200 } },
    { "delete": { "_id": "ORD-0002", "result": "not_found","status": 404 } }
  ]
}

Key points:

• "errors": true means at least one item failed — not that everything failed. Check each item individually.
• "status" per item is the HTTP status that would have been returned for a standalone request.
• Successful operations do not include an error key. Failed ones include "error": { "type": ..., "reason": ... }.
• The Bulk API never returns a non-2xx status for item-level failures — the HTTP response itself is 200. Always inspect "errors" and the items array.

Tip — optimal bulk size. A common starting point is 5–15 MB per bulk request or 1,000–5,000 documents, whichever comes first. Too small wastes round-trip overhead; too large can cause memory pressure. Tune based on your document size and cluster capacity.

2.7 Dynamic vs explicit mapping — a first look

Dynamic mapping — Elasticsearch guesses

When you index into an index that does not exist yet, Elasticsearch creates the index automatically and infers a data type for every field based on the first document it sees. You can inspect what it guessed:

GET /shop-orders-demo/_mapping

The response is a detailed schema — every field, its type, and its sub-fields. This is convenient during prototyping, but it has pitfalls.

Pitfall — the first document decides the type. If your first order document has "total_amount": "1299.00" (a string), Elasticsearch maps that field as text and keyword. Later documents that send 1299.00 (a number) will be stored as strings and arithmetic on that field will fail. The fix is to be consistent in your source data — or to define mappings explicitly before ingesting any data.

Pitfall — string fields get mapped as text AND keyword. Dynamic mapping creates a sub-field called .keyword for every string field. customer.name becomes searchable full-text as customer.name, and sortable/aggregatable as customer.name.keyword. This is usually what you want, but it doubles storage for string fields.

Explicit mapping — you define the schema

Create the index with a mapping before ingesting any data:

PUT /my-orders-explicit

{
  "mappings": {
    "properties": {
      "order_id":      { "type": "keyword" },
      "created_at":    { "type": "date" },
      "channel":       { "type": "keyword" },
      "total_amount":  { "type": "double" },
      "order_status":  { "type": "keyword" },
      "customer": {
        "properties": {
          "id":            { "type": "keyword" },
          "name":          { "type": "text", "fields": { "keyword": { "type": "keyword" } } },
          "loyalty_tier":  { "type": "keyword" }
        }
      },
      "returned":  { "type": "boolean" },
      "rating":    { "type": "float" },
      "notes":     { "type": "text" }
    }
  }
}

Now if you try to index "total_amount": "not-a-number", Elasticsearch rejects the document immediately rather than silently storing garbage.

Tip — you can mix both. Set "dynamic": "strict" to reject unknown fields, or "dynamic": false to ignore them silently (fields are stored in _source but not indexed). You can also enable dynamic mapping for some objects and disable it for others. All of this is covered in chapter 04 — Mapping and analyzers.

Labs

Work through these exercises in Kibana Dev Tools against your local Elasticsearch cluster.

Lab 2-A — Index, read, and verify your first document

1. Index a new document into my-first-orders with PUT, using the ID LAB-001. Include at least: order_id, customer_name, total_amount, order_status, and created_at.
2. Verify it was created with GET /_doc/LAB-001.
3. Try to PUT /_create/LAB-001 again with different data. Observe the 409 response.
4. Use HEAD to confirm the document exists, then GET /_source/LAB-001 to see only the source.

Takeaway: You understand the difference between PUT, PUT /_create, and POST, and can read back a document you indexed.

Lab 2-B — Partial update and scripted update

1. Index a document LAB-002 with order_status: "processing" and rating: 3.5.
2. Use POST /_update/LAB-002 with "doc" to change only order_status to "delivered".
3. Confirm rating is still 3.5 — it was not overwritten.
4. Use a Painless script to increment rating by 1.0.
5. Check _version in each response — it should increment with each real change, but not on a noop.

Takeaway: You know when to use full replace vs partial update vs scripted update, and you understand document versioning.

Lab 2-C — Bulk ingest and error handling

1. Send a POST /_bulk request that:
   1. Indexes three new orders (LAB-010, LAB-011, LAB-012).
   2. Uses create for LAB-010 — then run the same bulk body a second time and observe that LAB-010 fails with a 409 while the others succeed.
2. Check "errors": true in the response and find which item failed.
3. Use _mget to fetch all three documents in one call.

Takeaway: You can batch-write documents efficiently and interpret partial failures in the bulk response.

Lab 2-D — Mapping inspection and type pitfall

1. Index a document into a brand-new index my-type-test with "price": "not-a-number" (a string).
2. Call GET /my-type-test/_mapping and see what type ES inferred for price.
3. Try to index a second document with "price": 99.90 (a real number). What happens?
4. Delete the index (DELETE /my-type-test), then recreate it with an explicit mapping where price is double. Repeat steps 1–3 with the explicit index and compare the behaviour.

Takeaway: You have seen first-hand why the first document matters for dynamic mapping, and why explicit mappings are safer for production.

Summary

In the next chapter — 03 — Basic search — you will learn how to find documents using queries rather than fetching them by ID. The CRUD skills from this chapter form the foundation: you need to know how your data got in before you can search it effectively.