Cursor rule

1---
2description: Elasticsearch index mapping and indexing constraints
3globs: app/storage/es.py,app/worker/es_indexer.py
4alwaysApply: false
5---
6 
7# Elasticsearch Mapping & Indexing
8 
9ES is a **derived, rebuildable mirror** of Mongo — never the source of truth.
10The explicit mapping is intentional; document any change in `ARCHITECTURE.md §6`.
11 
12## Mapping is immutable — changes require a reindex
13 
14A field's mapping type cannot change on an existing index. Editing `_MAPPING`
15only affects newly created indices, so any type change means: create a new
16index with the new mapping and reindex from Mongo (the outbox is the source).
17Call this out explicitly in the PR/commit and the architecture doc.
18 
19## Field-type conventions (don't drift)
20 
21- Exact-match / aggregatable fields (`event_type`, `user_id`) → `keyword`.
22- `schema_version` → `integer` (producer-declared event schema version).
23- `source_url` → `keyword` with a `text` sub-field for tokenized search.
24- `metadata` → `flattened`: schemaless keys can't explode the mapping and mixed
25  value types across events can't cause index-time conflicts. The trade-off is
26  that `flattened` leaves are exact-match keyword, not analyzed full-text — which
27  is why metadata values are *also* mirrored into `metadata_text` (below).
28- `metadata_text` → `text` (analyzed). Derived at index time from the metadata
29  leaf values and included in the `/search?q=` query, so full-text search hits
30  terms that appear only inside metadata. Search-only: excluded from `/search`
31  responses (`source_excludes`) and never stored in Mongo (ARCHITECTURE §6).
32 
33## Indexing must tolerate partial failure
34 
35`ensure_mapping()` is idempotent and may run lazily (ES can be down at startup).
36Bulk index with `raise_on_error=False` so one malformed document doesn't discard
37the rest of the batch; log the failures. A transport-level error (ES
38unreachable) **should** propagate so the `EsIndexer` leaves events unindexed and
39retries them next pass — that's the outbox guarantee, not a bug to swallow.
40