AGENTS.md

1# ColumNAR — contributor & agent guide
2 
3Read `README.md` for the architecture first, then this. It covers what is expensive to get wrong.
4 
5## Non-negotiable rules
6 
71. **Binary only.** ColumNAR stores and serves fields at the `BINARY` surface (`addBinaryField` /
8   `getBinary`); the typed shapes (`Numeric`/`SortedNumeric`/`Sorted`/`SortedSet`) throw. There is no
9   delegate format — an unsupported type is an error, not a fallback.
10 
112. **Type-tagged and open.** Every field carries a `ColumnarFieldType` (`columnar.type` attribute).
12   `LONG`/`DOUBLE` are the numeric column today; new types (`STRING`, …) slot in by extending
13   the write dispatch (consumer) and read dispatch (producer) — the field framing is generic.
14 
153. **The integration chooses the encoding.** Encoding is a per-field decision driven by what the
16   integration knows (type, sorted, metric role). Keep that seam open; don't hard-wire one pipeline.
17 
184. **Insertion order is preserved.** The numeric column never sorts or deduplicates; value ordinals
19   stay internal to the presence layer.
20 
215. **Never hold a column on the heap.** Read, write and merge stream one block at a time. Offset
22   tables use `DirectMonotonic` (temp file on write, mapped slice on read); presence uses
23   `IndexedDISI`. Only bounded metadata and one decode block stay in memory.
24 
25## Encoders
26 
27A block is encoded by a `NumericPipeline`: adaptive `BlockTransform`s (delta, offset, GCD — reversible
28in-place transforms that fire only when they shrink the block) then one `BlockTerminal` (FOR
29bit-packing) that serializes the residuals. The default pipeline runs all detection; a field can be
30handed an explicit pipeline to skip it.
31 
32**Adding an encoder** — additive and backward-compatible:
33 
341. Implement `BlockTransform` (adaptive, mutates the `long[]` in place) or `BlockTerminal` (serializes
35   it) with a new, **frozen** `byte` id.
362. Register the id in `NumericPipeline.Registry`.
373. Add it to a pipeline — the default or a per-field one.
38 
39A column records its stage ids in metadata, so old data lists only old ids and a newer reader rebuilds
40the exact pipeline and decodes it unchanged. Never reuse or renumber a shipped id.
41 
42## Versioning
43 
44Each segment stamps `ColumnarFormat.VERSION_CURRENT`; readers accept `[VERSION_START, VERSION_CURRENT]`
45and reject anything newer. Most evolution needs no bump — new encoders, field types and block-bytes
46codecs ride frozen ids recorded per column. Only a change to the metadata *layout* needs a version
47bump; then branch on the header version (returned by `ColumnarCodecUtil.checkHeader`) in the affected
48`readFrom`.
49 
50## Benchmarks & tests
51 
52Ship every format change with round-trip and range/bulk correctness coverage. JMH benchmarks live
53in the `:benchmarks` module (`org.elasticsearch.benchmark.index.codec.columnar`) and compare ColumNAR
54against the TSDB codecs; no results are committed. See `docs/BENCHMARKS.md`.
55 
56## Build & verify
57 
58- `./gradlew :libs:columnar:spotlessApply` — format (no wildcard imports; don't reorder untouched lines).
59- `./gradlew :libs:columnar:test` — tests.
60- Dependencies stay minimal (`lucene-core`, `libs:simdvec`) and need justification. Never depend on
61  `server` — `server` will depend on this library.
62 
63The repo-wide top-level `AGENTS.md` governs formatting, logging, Javadoc, and license headers.
64