CLAUDE.md

1# engine — the content-aware Caveman Engine (commercial, binary-distributed)
2 
3Detect a payload's type → route to a safety-classed compressor → count the token reduction
4→ store the original for recovery. The stable four-call API (`Compress`/`Retrieve`/`Detect`/`Stats`)
5is what the proxy, CLI, SDKs, MCP, and WASM build all share. Everything it reports is
6`inferred`; it never says `verified`.
7 
8## Layout
9- `engine.go` — the `Engine` core: detect → route → compress → token ratio → CCR. `record` mode + miss + parse-fail + not-smaller + no-store all pass through.
10- `result.go` — `Result`/`Options`/`Mode`; unknown mode fails closed to `record`.
11- `detect.go` — content router (`json`/`diff`/`code`/`log`/`search-result`/`html`/`tabular`/`config`/`terminal`/`text`); low confidence → `text`.
12- `listing.go` — strips a read tool's line-number gutter (`1\t{`, `2\t  "unit"`) before Detect and before the compressor, then restores it after. Agents send file *listings*, not files; the gutter is presentation, and left in place it made every guttered payload detect as `text` and compress ~0%. Restoration keeps each surviving line's ORIGINAL number and is declined entirely for transforms that restructure rather than elide (re-encoded JSON), because numbers that describe nothing are worse than none.
13- `safety/` — the S0–S4 registry; S4 is lossy and `RequiresCCR`. Unknown class → fail closed.
14- `tokens/` — `Counter` interface; default is an offline, vocab-embedded BPE tokenizer (o200k_base). `inferred` always.
15- `contextwindow/` — deterministic BM25 context packer with recency/error/priority signals and token-budget accounting.
16- `compressors/` — `Compressor` interface + registry; structural JSON/log/search/diff/text/HTML/table/config/code compressors plus forced-only TOON/tool-schema/tool-schema-annotations/accessibility/repetition paths — `Default()` registers 15. `toolschema-annotations` stays manifest-excluded because no compiled plan routes to it; advertising it would rotate `RegistrySHA256` and fail existing Cave Build locks. A compressor is a pure byte transform.
17- `ccr/` — `~/.caveman/ccr.db` SQLite recovery plus typed native-session store;
18  content-addressed handles, byte-exact `Get`, session scope, dependencies,
19  current/stale state, and Hot/Warm/Cold/Archived lifecycle. Embedded SQLite uses
20  one serialized connection so concurrent hook/repository writes cannot split an
21  in-memory schema or race with `SQLITE_BUSY`.
22- `pixel/` — pxpipe port (MIT, see its NOTICE): text→PNG request compression. Embedded glyph atlases + renderer + profitability gate + per-wire-format transforms (Anthropic/OpenAI/Gemini). S4-lossy, allowlist-gated (`CAVE_PIXEL_MODELS`, default `claude-fable-5,gpt-5.6`), consumed by the proxy's `pixel` mode; never wired into `Detect` and never imported by the WASM build (≈4 MB assets).
23- `evals/` — the local eval harness + fail-closed graders + embedded `fixtures/`. `Run()` is the quality gate.
24- `cmd/caveman-engine/` — the binary the CLI shells out to: `compress | detect | retrieve | stats | registry | toon encode|decode | evals run | pixel render|simulate`. `toon` is the stateless (no-CCR) JSON⇄TOON converter; both directions fail closed.
25 
26## Conventions
27- Build/test: `make product-build PRODUCT=engine` / `make product-test PRODUCT=engine`.
28- A new compressor is a self-contained file in `compressors/` + tests, registered in `Default()`. Forced-only compressors such as `toolschema` and `toon` must not be added to `Detect`.
29- Compressors never count tokens, touch CCR, or hit the network — the engine core does that around them.
30- **The `toolschema` transform is client-side-only today.** Registration makes it forceable by
31  engine callers; it does not make it reachable from managed-gateway traffic. Provider adapters
32  intentionally keep tool arrays in the frozen prompt-cache prefix, and no billed route invokes
33  this compressor. Engine API/CLI callers may force it locally, and `caveman-shrink` is its
34  dedicated product surface; those reductions remain local and `inferred`. Managed gateway has a
35  separate S2 tool-search/deferral path; do not conflate it with compression. Any future gateway
36  route for this transform needs cache-versus-schema cost arithmetic, stable byte-identical prefix
37  output, and an eval gate before activation.
38 
39## Gotchas (honesty invariants — correctness, not style)
40- **fail-closed transforms**: every compressor passes the original through unchanged on any parse problem; the engine also passes through when the result is not smaller in tokens. Reserve **byte-safe** for classes whose `safety.Info.ByteSafe` is true.
41- **elide repetition, never a document**: every unit-dropping compressor (text, log, json, tabular, config, searchresult, terminal) calls `keepNonRedundant` (`compressors/redundancy.go`) before emitting. It keeps one representative of each distinct content class, so a stream of near-identical lines still collapses while a bibliography, a source file or a table of measurements survives whole — and when nothing is left to drop, the not-smaller check passes the payload through and claims nothing. This is a correctness rule, not a tuning knob: eliding a document the agent must enumerate cost the agent bench 5.5x its baseline on 2026-08-06 (`bench/agent/README.md`). Class matching masks digit runs, but only for units carrying at least one word — a wordless unit like `6 ['1973.', 251]` is compared with its digits intact, because there the numbers ARE the content.
42- **an elision marker states only what it verified**: `compressors/invariants.go` enriches the log/tabular/JSON markers with facts computed from the exact units that marker replaced — a field constant across every one of them (`all state=charged`), a varying field enumerated IN FULL with exact counts that sum to the elided count (`status: fulfilled×15 shipped×3 processing×2`, with an `absent×N` bucket for units lacking the field), and a numeric range printed from the ORIGINAL extreme value strings (`range amount=5.00..199.99`). Nothing else. A credential-shaped field name, a value with a space or over 24 bytes, more than five distinct values, and a run in which any unit yields no fields at all are all withheld **entirely** — never shortened, because a partial list reads as complete. When nothing survives, the marker renders byte-identically to the bare `… N rows elided (caveman) …`. An enumeration whose every bucket holds ONE unit is an identifier list, not an invariant, so instead of enumerating it the marker states its COVERAGE — `order_id: 25 distinct, ord-1000..ord-1024`, the exact distinct count plus the byte-wise smallest and largest ORIGINAL value strings, claiming nothing about the gaps between them (this is the only fact that answers "was ord-1043 among the elided rows, or absent from the ledger?", the last legitimate retrieve on the v5 probe). When a coverage run is EXACTLY dense — one shared prefix, fixed-width digits, and `max-min+1` equal to the counted distinct set — it says so instead: `wh-5000..wh-5059 all 60 present`, a verified membership answer that a set-difference task needs and would otherwise retrieve for. Density is counted, never inferred from the endpoints; mixed suffix widths, two prefixes, or a single gap all fall back to the counted form (whose own arithmetic still says how many are missing, never which). A dense entry outlives even the constants when the budget binds; a bounded one goes before them. At most two coverage entries per summary — a record carries several identifier columns and at ~38 bytes each they will name the run three ways and crowd out a 16-byte constant. An enumeration averaging two units or fewer per bucket still prints but is shed first. Summaries are capped at 160 bytes and a quarter of the bytes they replace (floor 64, never over half), shedding identifier-ish enumerations, then ranges, then bounded coverage, then constants, then dense coverage, class enumerations last; one trailing contract line per payload is added only when ≥4 KB was actually dropped. **A run under 3 units is not elided at all** unless collapsing both summarizes and halves its bytes — a one-unit marker plus its recovery handle costs about what the unit cost and reads as a hole. Together this costs ~4 points of ratio on the toolwork corpus (0.83 → 0.79) and is bought on purpose: on 2026-08-08 an agent that could not tell what had been elided made 46 sequential `caveman_retrieve` calls at 3.3x the uncompressed arm; a first fix stating only constants and ranges left the storms at 11–27 calls because it was silent about the one mixed-value field the task turned on; and a second round still leaked retrieves through identifier enums, fact-free small classes, and `{"__caveman_elided__":1}` singletons.
43- **a recovery view is made of whole units**: `retrieve_query.go` narrows a CCR recovery to the units matching a query, and a unit is whatever is self-delimiting — a complete JSON record (never a field line: `"status": "unfulfilled",` is a fragment), a whole CSV record sliced out of the original bytes with the header shown once, a whole log/NDJSON line, or a whole paragraph from non-JSON prose. JSON field names are never provenance: `messages[].content`, `text`, and `system` stay attached to their full object record, because tool output can use the same names as provider requests. Separate JSON arrays have explicit gap sentinels; omitted envelope/scalar bytes get leading/trailing markers. Content it cannot decompose (JSON with no array of records) returns in FULL rather than line-split, and a narrowed view that is not smaller also returns the full original; over-returning is safe, a fragment is not. Every gap — between two returned units, before the first, after the last — prints `… [caveman: non-adjacent] …`, so adjacency in the view never implies adjacency in the original. This is a correctness rule with a measured cost: on 2026-08-08 query mode returned BM25-ranked lines of a pretty-printed orders page, the agent read `"status": "unfulfilled"` (ord-1043's) directly above `"order_id": "ord-1047"`, attributed it to ord-1047, and reported 5 unfulfilled orders where there were 3 — two wrong answers in a 12-task bench run.
44- **one recovery surface, two id spaces**: the CCR store holds compression blob handles (`ccr_…`, `store.Get`) AND the native runtime's typed objects (`store.GetObject`), and the runtime shows the latter to agents as `ccr://<objectID>` when it masks a whole tool output. `Engine.Retrieve` resolves a blob first and falls through to the object table before failing, and `mcp.normalizeRecoveryHandle` accepts `ccr_…`, `<<ccr:…>>`, `ccr:…` and `ccr://…`. An agent copies whatever reference it is shown; a form that does not resolve does not degrade, it loops. Measured 2026-08-08: inventory-mismatch and webhook-delivery-gaps were handed `ccr://<objectID>`, every retrieve answered `cave_unknown_handle`, the failing tool result was itself masked into a fresh pointer — "a new pointer to another pointer, indefinitely" — 27–97 recovery calls, no answer written, both tasks 0/6 while rate-limit-forensics on the same build scored 3/3 at ~35% cheaper.
45- **CCR-or-pass-through**: a lossy (S4) result is only emitted if its original was stored; with no store, the engine fails closed to pass-through.
46- **inferred-only**: ratios are token estimates labeled `inferred`; never `verified`, never re-projected.
47- **fail-closed**: unknown mode → `record`; unknown content type → `text`; unknown grader → `passed:false`.
48- **cgo**: full code compression (Python/JS/TS) needs the tree-sitter build; the cgo-free build compresses Go only. The embedded eval fixtures cover all three under cgo.
49- **boundary**: this is `public/` — never import `cloud/…`. `make check-boundaries` enforces it.
50 
51See ../../CLAUDE.md (root)
52