Comparison

A · AGENTS.md · elastic/elasticsearch B · AGENTS.md · elastic/elasticsearch

What each file covers, counted
Dimension	Only in A	Only in B	Overlap
Sections	6	6	0%
Commands	2	0	0%
Section tags	3	1	0%

What each file covers

Sections

0 shared · 6 only in A · 6 only in B

− ColumNAR — contributor & agent guide
− Non-negotiable rules
− Encoders
− Versioning
− Benchmarks & tests
− Build & verify
+ Benchmarks
+ Running benchmarks
+ ColumNAR transform benchmarks
+ Single stage + pattern
+ Quick smoke
+ Self-test

Commands

0 shared · 2 only in A · 0 only in B

− ./gradlew :libs:columnar:spotlessApply
− ./gradlew :libs:columnar:test

Section tags

0 shared · 3 only in A · 1 only in B

− build
− do-not
− agent-behaviour
+ test

Line diff

+22 added−54 removed10 unchanged15.6% identical

elastic/elasticsearch · libs/columnar/AGENTS.md

@@ −1 @@

1# ColumNAR — contributor & agent guide

3Read `README.md` for the architecture first, then this. It covers what is expensive to get wrong.

5## Non-negotiable rules

71. **Binary only.** ColumNAR stores and serves fields at the `BINARY` surface (`addBinaryField` /

8 `getBinary`); the typed shapes (`Numeric`/`SortedNumeric`/`Sorted`/`SortedSet`) throw. There is no

9 delegate format — an unsupported type is an error, not a fallback.

112. **Type-tagged and open.** Every field carries a `ColumnarFieldType` (`columnar.type` attribute).

12 `LONG`/`DOUBLE` are the numeric column today; new types (`STRING`, …) slot in by extending

13 the write dispatch (consumer) and read dispatch (producer) — the field framing is generic.

153. **The integration chooses the encoding.** Encoding is a per-field decision driven by what the

16 integration knows (type, sorted, metric role). Keep that seam open; don't hard-wire one pipeline.

184. **Insertion order is preserved.** The numeric column never sorts or deduplicates; value ordinals

19 stay internal to the presence layer.

215. **Never hold a column on the heap.** Read, write and merge stream one block at a time. Offset

22 tables use `DirectMonotonic` (temp file on write, mapped slice on read); presence uses

23 `IndexedDISI`. Only bounded metadata and one decode block stay in memory.

25## Encoders

27A block is encoded by a `NumericPipeline`: adaptive `BlockTransform`s (delta, offset, GCD — reversible

28in-place transforms that fire only when they shrink the block) then one `BlockTerminal` (FOR

29bit-packing) that serializes the residuals. The default pipeline runs all detection; a field can be

30handed an explicit pipeline to skip it.

32**Adding an encoder** — additive and backward-compatible:

341. Implement `BlockTransform` (adaptive, mutates the `long[]` in place) or `BlockTerminal` (serializes

35 it) with a new, **frozen** `byte` id.

362. Register the id in `NumericPipeline.Registry`.

373. Add it to a pipeline — the default or a per-field one.

39A column records its stage ids in metadata, so old data lists only old ids and a newer reader rebuilds

40the exact pipeline and decodes it unchanged. Never reuse or renumber a shipped id.

42## Versioning

44Each segment stamps `ColumnarFormat.VERSION_CURRENT`; readers accept `[VERSION_START, VERSION_CURRENT]`

45and reject anything newer. Most evolution needs no bump — new encoders, field types and block-bytes

46codecs ride frozen ids recorded per column. Only a change to the metadata *layout* needs a version

47bump; then branch on the header version (returned by `ColumnarCodecUtil.checkHeader`) in the affected

48`readFrom`.

50## Benchmarks & tests

52Ship every format change with round-trip and range/bulk correctness coverage. JMH benchmarks live

53in the `:benchmarks` module (`org.elasticsearch.benchmark.index.codec.columnar`) and compare ColumNAR

54against the TSDB codecs; no results are committed. See `docs/BENCHMARKS.md`.

56## Build & verify

58- `./gradlew :libs:columnar:spotlessApply` — format (no wildcard imports; don't reorder untouched lines).

59- `./gradlew :libs:columnar:test` — tests.

60- Dependencies stay minimal (`lucene-core`, `libs:simdvec`) and need justification. Never depend on

61 `server` — `server` will depend on this library.

63The repo-wide top-level `AGENTS.md` governs formatting, logging, Javadoc, and license headers.

elastic/elasticsearch · benchmarks/AGENTS.md

@@ +1 @@

1# Benchmarks

3## Running benchmarks

5Run from the `benchmarks/` directory using the `run` task with `--args`. Always use the fully-qualified class

6name including package to avoid ambiguity. Always pipe through `tee /tmp/bench/<descriptive_name>` using a filename that reflects the task (e.g. `tee /tmp/bench/paged_write`).

8```

9cd benchmarks

10../gradlew run --args "org.elasticsearch.benchmark._nightly.BytesBuilderBenchmark -pdata=1000_ints -pimpl=paged -poperation=write -rf json -rff build/jmh-result.json" | tee /tmp/bench/paged_write

11```

13## ColumNAR transform benchmarks

15```

16cd benchmarks

17../gradlew run --args="EncodeBlockTransformBenchmark" | tee /tmp/bench/encode_transform

18../gradlew run --args="DecodeBlockTransformBenchmark" | tee /tmp/bench/decode_transform

20# Single stage + pattern

21../gradlew run --args="EncodeBlockTransformBenchmark -p stage=splitDelta -p pattern=TSDB_SPLIT" | tee /tmp/bench/encode_splitdelta_tsdb

23# Quick smoke

24../gradlew run --args="EncodeBlockTransformBenchmark -wi 1 -i 1 -f 1 -w 1 -r 1 -p stage=delta -p pattern=MONOTONIC_TIMESTAMPS"

25```

27## Self-test

29Never skip the self-test. Do not pass `-DskipSelfTest=true` or `--test` to `run.sh`. The

30self-test validates correctness across all impl/operation/data combinations and poisons virtual

31dispatch to behave more like production.

@@ −1 +1 @@

1−# ColumNAR — contributor & agent guide

1+# Benchmarks

3−Read `README.md` for the architecture first, then this. It covers what is expensive to get wrong.

3+## Running benchmarks

5−## Non-negotiable rules

5+Run from the `benchmarks/` directory using the `run` task with `--args`. Always use the fully-qualified class

6+name including package to avoid ambiguity. Always pipe through `tee /tmp/bench/<descriptive_name>` using a filename that reflects the task (e.g. `tee /tmp/bench/paged_write`).

7−1. **Binary only.** ColumNAR stores and serves fields at the `BINARY` surface (`addBinaryField` /

8− `getBinary`); the typed shapes (`Numeric`/`SortedNumeric`/`Sorted`/`SortedSet`) throw. There is no

9− delegate format — an unsupported type is an error, not a fallback.

8+```

9+cd benchmarks

10+../gradlew run --args "org.elasticsearch.benchmark._nightly.BytesBuilderBenchmark -pdata=1000_ints -pimpl=paged -poperation=write -rf json -rff build/jmh-result.json" | tee /tmp/bench/paged_write

11+```

1012

11−2. **Type-tagged and open.** Every field carries a `ColumnarFieldType` (`columnar.type` attribute).

12− `LONG`/`DOUBLE` are the numeric column today; new types (`STRING`, …) slot in by extending

13− the write dispatch (consumer) and read dispatch (producer) — the field framing is generic.

13+## ColumNAR transform benchmarks

1414

15−3. **The integration chooses the encoding.** Encoding is a per-field decision driven by what the

16− integration knows (type, sorted, metric role). Keep that seam open; don't hard-wire one pipeline.

15+```

16+cd benchmarks

17+../gradlew run --args="EncodeBlockTransformBenchmark" | tee /tmp/bench/encode_transform

18+../gradlew run --args="DecodeBlockTransformBenchmark" | tee /tmp/bench/decode_transform

1719

18−4. **Insertion order is preserved.** The numeric column never sorts or deduplicates; value ordinals

19− stay internal to the presence layer.

20+# Single stage + pattern

21+../gradlew run --args="EncodeBlockTransformBenchmark -p stage=splitDelta -p pattern=TSDB_SPLIT" | tee /tmp/bench/encode_splitdelta_tsdb

2022

21−5. **Never hold a column on the heap.** Read, write and merge stream one block at a time. Offset

22− tables use `DirectMonotonic` (temp file on write, mapped slice on read); presence uses

23− `IndexedDISI`. Only bounded metadata and one decode block stay in memory.

23+# Quick smoke

24+../gradlew run --args="EncodeBlockTransformBenchmark -wi 1 -i 1 -f 1 -w 1 -r 1 -p stage=delta -p pattern=MONOTONIC_TIMESTAMPS"

25+```

2426

25−## Encoders

27+## Self-test

2628

27−A block is encoded by a `NumericPipeline`: adaptive `BlockTransform`s (delta, offset, GCD — reversible

28−in-place transforms that fire only when they shrink the block) then one `BlockTerminal` (FOR

29−bit-packing) that serializes the residuals. The default pipeline runs all detection; a field can be

30−handed an explicit pipeline to skip it.

31−

32−**Adding an encoder** — additive and backward-compatible:

33−

34−1. Implement `BlockTransform` (adaptive, mutates the `long[]` in place) or `BlockTerminal` (serializes

35− it) with a new, **frozen** `byte` id.

36−2. Register the id in `NumericPipeline.Registry`.

37−3. Add it to a pipeline — the default or a per-field one.

38−

39−A column records its stage ids in metadata, so old data lists only old ids and a newer reader rebuilds

40−the exact pipeline and decodes it unchanged. Never reuse or renumber a shipped id.

41−

42−## Versioning

43−

44−Each segment stamps `ColumnarFormat.VERSION_CURRENT`; readers accept `[VERSION_START, VERSION_CURRENT]`

45−and reject anything newer. Most evolution needs no bump — new encoders, field types and block-bytes

46−codecs ride frozen ids recorded per column. Only a change to the metadata *layout* needs a version

47−bump; then branch on the header version (returned by `ColumnarCodecUtil.checkHeader`) in the affected

48−`readFrom`.

49−

50−## Benchmarks & tests

51−

52−Ship every format change with round-trip and range/bulk correctness coverage. JMH benchmarks live

53−in the `:benchmarks` module (`org.elasticsearch.benchmark.index.codec.columnar`) and compare ColumNAR

54−against the TSDB codecs; no results are committed. See `docs/BENCHMARKS.md`.

55−

56−## Build & verify

57−

58−- `./gradlew :libs:columnar:spotlessApply` — format (no wildcard imports; don't reorder untouched lines).

59−- `./gradlew :libs:columnar:test` — tests.

60−- Dependencies stay minimal (`lucene-core`, `libs:simdvec`) and need justification. Never depend on

61− `server` — `server` will depend on this library.

62−

63−The repo-wide top-level `AGENTS.md` governs formatting, logging, Javadoc, and license headers.

29+Never skip the self-test. Do not pass `-DskipSelfTest=true` or `--test` to `run.sh`. The

30+self-test validates correctness across all impl/operation/data combinations and poisons virtual

31+dispatch to behave more like production.

6432