Comparison

A · AGENTS.md · elastic/elasticsearch B · AGENTS.md · elastic/elasticsearch

What each file covers, counted
Dimension	Only in A	Only in B	Overlap
Sections	6	6	0%
Commands	0	2	0%
Section tags	1	3	0%

What each file covers

Sections

0 shared · 6 only in A · 6 only in B

− Benchmarks
− Running benchmarks
− ColumNAR transform benchmarks
− Single stage + pattern
− Quick smoke
− Self-test
+ ColumNAR — contributor & agent guide
+ Non-negotiable rules
+ Encoders
+ Versioning
+ Benchmarks & tests
+ Build & verify

Commands

0 shared · 0 only in A · 2 only in B

+ ./gradlew :libs:columnar:spotlessApply
+ ./gradlew :libs:columnar:test

Section tags

0 shared · 1 only in A · 3 only in B

− test
+ build
+ do-not
+ agent-behaviour

Line diff

+54 added−22 removed10 unchanged15.6% identical

elastic/elasticsearch · benchmarks/AGENTS.md

@@ −1 @@

1# Benchmarks

3## Running benchmarks

5Run from the `benchmarks/` directory using the `run` task with `--args`. Always use the fully-qualified class

6name including package to avoid ambiguity. Always pipe through `tee /tmp/bench/<descriptive_name>` using a filename that reflects the task (e.g. `tee /tmp/bench/paged_write`).

8```

9cd benchmarks

10../gradlew run --args "org.elasticsearch.benchmark._nightly.BytesBuilderBenchmark -pdata=1000_ints -pimpl=paged -poperation=write -rf json -rff build/jmh-result.json" | tee /tmp/bench/paged_write

11```

13## ColumNAR transform benchmarks

15```

16cd benchmarks

17../gradlew run --args="EncodeBlockTransformBenchmark" | tee /tmp/bench/encode_transform

18../gradlew run --args="DecodeBlockTransformBenchmark" | tee /tmp/bench/decode_transform

20# Single stage + pattern

21../gradlew run --args="EncodeBlockTransformBenchmark -p stage=splitDelta -p pattern=TSDB_SPLIT" | tee /tmp/bench/encode_splitdelta_tsdb

23# Quick smoke

24../gradlew run --args="EncodeBlockTransformBenchmark -wi 1 -i 1 -f 1 -w 1 -r 1 -p stage=delta -p pattern=MONOTONIC_TIMESTAMPS"

25```

27## Self-test

29Never skip the self-test. Do not pass `-DskipSelfTest=true` or `--test` to `run.sh`. The

30self-test validates correctness across all impl/operation/data combinations and poisons virtual

31dispatch to behave more like production.

elastic/elasticsearch · libs/columnar/AGENTS.md

@@ +1 @@

1# ColumNAR — contributor & agent guide

3Read `README.md` for the architecture first, then this. It covers what is expensive to get wrong.

5## Non-negotiable rules

71. **Binary only.** ColumNAR stores and serves fields at the `BINARY` surface (`addBinaryField` /

8 `getBinary`); the typed shapes (`Numeric`/`SortedNumeric`/`Sorted`/`SortedSet`) throw. There is no

9 delegate format — an unsupported type is an error, not a fallback.

112. **Type-tagged and open.** Every field carries a `ColumnarFieldType` (`columnar.type` attribute).

12 `LONG`/`DOUBLE` are the numeric column today; new types (`STRING`, …) slot in by extending

13 the write dispatch (consumer) and read dispatch (producer) — the field framing is generic.

153. **The integration chooses the encoding.** Encoding is a per-field decision driven by what the

16 integration knows (type, sorted, metric role). Keep that seam open; don't hard-wire one pipeline.

184. **Insertion order is preserved.** The numeric column never sorts or deduplicates; value ordinals

19 stay internal to the presence layer.

215. **Never hold a column on the heap.** Read, write and merge stream one block at a time. Offset

22 tables use `DirectMonotonic` (temp file on write, mapped slice on read); presence uses

23 `IndexedDISI`. Only bounded metadata and one decode block stay in memory.

25## Encoders

27A block is encoded by a `NumericPipeline`: adaptive `BlockTransform`s (delta, offset, GCD — reversible

28in-place transforms that fire only when they shrink the block) then one `BlockTerminal` (FOR

29bit-packing) that serializes the residuals. The default pipeline runs all detection; a field can be

30handed an explicit pipeline to skip it.

32**Adding an encoder** — additive and backward-compatible:

341. Implement `BlockTransform` (adaptive, mutates the `long[]` in place) or `BlockTerminal` (serializes

35 it) with a new, **frozen** `byte` id.

362. Register the id in `NumericPipeline.Registry`.

373. Add it to a pipeline — the default or a per-field one.

39A column records its stage ids in metadata, so old data lists only old ids and a newer reader rebuilds

40the exact pipeline and decodes it unchanged. Never reuse or renumber a shipped id.

42## Versioning

44Each segment stamps `ColumnarFormat.VERSION_CURRENT`; readers accept `[VERSION_START, VERSION_CURRENT]`

45and reject anything newer. Most evolution needs no bump — new encoders, field types and block-bytes

46codecs ride frozen ids recorded per column. Only a change to the metadata *layout* needs a version

47bump; then branch on the header version (returned by `ColumnarCodecUtil.checkHeader`) in the affected

48`readFrom`.

50## Benchmarks & tests

52Ship every format change with round-trip and range/bulk correctness coverage. JMH benchmarks live

53in the `:benchmarks` module (`org.elasticsearch.benchmark.index.codec.columnar`) and compare ColumNAR

54against the TSDB codecs; no results are committed. See `docs/BENCHMARKS.md`.

56## Build & verify

58- `./gradlew :libs:columnar:spotlessApply` — format (no wildcard imports; don't reorder untouched lines).

59- `./gradlew :libs:columnar:test` — tests.

60- Dependencies stay minimal (`lucene-core`, `libs:simdvec`) and need justification. Never depend on

61 `server` — `server` will depend on this library.

63The repo-wide top-level `AGENTS.md` governs formatting, logging, Javadoc, and license headers.

@@ −1 +1 @@

1−# Benchmarks

1+# ColumNAR — contributor & agent guide

3−## Running benchmarks

3+Read `README.md` for the architecture first, then this. It covers what is expensive to get wrong.

5−Run from the `benchmarks/` directory using the `run` task with `--args`. Always use the fully-qualified class

6−name including package to avoid ambiguity. Always pipe through `tee /tmp/bench/<descriptive_name>` using a filename that reflects the task (e.g. `tee /tmp/bench/paged_write`).

5+## Non-negotiable rules

8−```

9−cd benchmarks

10−../gradlew run --args "org.elasticsearch.benchmark._nightly.BytesBuilderBenchmark -pdata=1000_ints -pimpl=paged -poperation=write -rf json -rff build/jmh-result.json" | tee /tmp/bench/paged_write

11−```

7+1. **Binary only.** ColumNAR stores and serves fields at the `BINARY` surface (`addBinaryField` /

8+ `getBinary`); the typed shapes (`Numeric`/`SortedNumeric`/`Sorted`/`SortedSet`) throw. There is no

9+ delegate format — an unsupported type is an error, not a fallback.

1210

13−## ColumNAR transform benchmarks

11+2. **Type-tagged and open.** Every field carries a `ColumnarFieldType` (`columnar.type` attribute).

12+ `LONG`/`DOUBLE` are the numeric column today; new types (`STRING`, …) slot in by extending

13+ the write dispatch (consumer) and read dispatch (producer) — the field framing is generic.

1414

15−```

16−cd benchmarks

17−../gradlew run --args="EncodeBlockTransformBenchmark" | tee /tmp/bench/encode_transform

18−../gradlew run --args="DecodeBlockTransformBenchmark" | tee /tmp/bench/decode_transform

15+3. **The integration chooses the encoding.** Encoding is a per-field decision driven by what the

16+ integration knows (type, sorted, metric role). Keep that seam open; don't hard-wire one pipeline.

1917

20−# Single stage + pattern

21−../gradlew run --args="EncodeBlockTransformBenchmark -p stage=splitDelta -p pattern=TSDB_SPLIT" | tee /tmp/bench/encode_splitdelta_tsdb

18+4. **Insertion order is preserved.** The numeric column never sorts or deduplicates; value ordinals

19+ stay internal to the presence layer.

2220

23−# Quick smoke

24−../gradlew run --args="EncodeBlockTransformBenchmark -wi 1 -i 1 -f 1 -w 1 -r 1 -p stage=delta -p pattern=MONOTONIC_TIMESTAMPS"

25−```

21+5. **Never hold a column on the heap.** Read, write and merge stream one block at a time. Offset

22+ tables use `DirectMonotonic` (temp file on write, mapped slice on read); presence uses

23+ `IndexedDISI`. Only bounded metadata and one decode block stay in memory.

2624

27−## Self-test

25+## Encoders

2826

29−Never skip the self-test. Do not pass `-DskipSelfTest=true` or `--test` to `run.sh`. The

30−self-test validates correctness across all impl/operation/data combinations and poisons virtual

31−dispatch to behave more like production.

27+A block is encoded by a `NumericPipeline`: adaptive `BlockTransform`s (delta, offset, GCD — reversible

28+in-place transforms that fire only when they shrink the block) then one `BlockTerminal` (FOR

29+bit-packing) that serializes the residuals. The default pipeline runs all detection; a field can be

30+handed an explicit pipeline to skip it.

31+

32+**Adding an encoder** — additive and backward-compatible:

33+

34+1. Implement `BlockTransform` (adaptive, mutates the `long[]` in place) or `BlockTerminal` (serializes

35+ it) with a new, **frozen** `byte` id.

36+2. Register the id in `NumericPipeline.Registry`.

37+3. Add it to a pipeline — the default or a per-field one.

38+

39+A column records its stage ids in metadata, so old data lists only old ids and a newer reader rebuilds

40+the exact pipeline and decodes it unchanged. Never reuse or renumber a shipped id.

41+

42+## Versioning

43+

44+Each segment stamps `ColumnarFormat.VERSION_CURRENT`; readers accept `[VERSION_START, VERSION_CURRENT]`

45+and reject anything newer. Most evolution needs no bump — new encoders, field types and block-bytes

46+codecs ride frozen ids recorded per column. Only a change to the metadata *layout* needs a version

47+bump; then branch on the header version (returned by `ColumnarCodecUtil.checkHeader`) in the affected

48+`readFrom`.

49+

50+## Benchmarks & tests

51+

52+Ship every format change with round-trip and range/bulk correctness coverage. JMH benchmarks live

53+in the `:benchmarks` module (`org.elasticsearch.benchmark.index.codec.columnar`) and compare ColumNAR

54+against the TSDB codecs; no results are committed. See `docs/BENCHMARKS.md`.

55+

56+## Build & verify

57+

58+- `./gradlew :libs:columnar:spotlessApply` — format (no wildcard imports; don't reorder untouched lines).

59+- `./gradlew :libs:columnar:test` — tests.

60+- Dependencies stay minimal (`lucene-core`, `libs:simdvec`) and need justification. Never depend on

61+ `server` — `server` will depend on this library.

62+

63+The repo-wide top-level `AGENTS.md` governs formatting, logging, Javadoc, and license headers.

3264