CLAUDE.md

1# mcp — the Caveman MCP server (commercial Go core + MIT launcher)
2 
3A thin **stdio JSON-RPC** adapter exposing the compression [engine](../engine/CLAUDE.md) as
4five MCP tools to any host (Claude Code, Cursor, …). It owns only the MCP framing; all
5compression is the engine's, linked **in-process** (no subprocess, no drift). Local-only — it
6opens no network connection — and everything it reports is `inferred`, never `verified`.
7 
8## Layout
9- `server.go` — the `Server`: JSON-RPC loop, dispatch, the five tool handlers. Takes an injectable `Engine` interface so the framing is testable without the real compressors.
10- `protocol.go` — JSON-RPC + MCP tool-result types, `toolText`/`toolError` helpers, the exact `tools/list` definitions.
11- `cmd/caveman-mcp/` — binary: opens shared file CCR store (`CAVEMAN_CCR_DB`,
12  else `CAVEMAN_HOME/ccr.db`/`~/.caveman/ccr.db`) so proxy handles resolve across
13  processes, then serves stdin↔stdout. `CAVEMAN_MCP_EPHEMERAL=1` opts into an
14  isolated in-memory store for tests/sessions that do not need proxy recovery.
15- `bin/caveman-mcp.mjs` + `package.json` — the `npx caveman-mcp` launcher that execs the prebuilt Go binary.
16 
17## The five tools (exact names, case-sensitive)
18- `caveman_compress(input)` → compressed text + inferred ratio + `recovery_handle`. Lossy (S4), recoverable, and fail-closed: incompressible/malformed/not-smaller input returns unchanged, `ratio:0`, `recovery_handle:null` — never an error.
19- `caveman_retrieve(recovery_handle)` → the byte-exact original. Unknown handle → `isError:true` + a `cave_snake_code`, never a fabricated payload.
20- `caveman_stats()` → `basis:"inferred"`, `scope:"session"`; the string `verified` never appears.
21- `caveman_toon_encode(input)` → explicit JSON→TOON re-encoding with input/output sizes; returns pass-through plus note when encoding fails.
22- `caveman_toon_decode(input)` → TOON→JSON; invalid TOON returns `isError:true`, never raw input as JSON.
23 
24## These five tools are a prompt-prefix cost, and it is measured
25Registering this server puts all five tool schemas in the wrapped agent's prefix
26on **every** call — **11,060 tokens/call** in the agent bench (~2.22M over its 201
27calls). That is why `caveman wrap` gates the injection behind the `execute.mcp`
28surface knob (`auto` | `marker-only` | `true` | `false`, `packages/cli/src/index.ts`).
29Under a non-auto surface wrap suppresses **both** of its injection sites — the
30`mcp install` write and, for config-file agents like openclaw, the profile's
31`mcp.servers.caveman` overlay — but it never uninstalls a server the user
32installed themselves, so an existing install keeps costing its prefix until
33`caveman tools mcp uninstall <agent>`. The knob never touches recovery honesty on
34the wrap door: `CAVEMAN_RECOVERY` is answered from evidence that this launch is
35really giving the agent `caveman_retrieve`, so the proxy is never told a
36retrieval tool exists when none does. Adding a sixth tool here raises that
37per-call tax for every wrapped agent.
38 
39## Conventions
40- Build/test: `make product-build PRODUCT=mcp` / `make product-test PRODUCT=mcp`.
41- **stdout is the protocol channel** — logs go to stderr only (a dedicated test guards this).
42 
43## Gotchas (honesty invariants)
44- **un-killable transport** — the stdio server survives everything short of EOF (issue #139). Framing is line-delimited: a malformed line is answered `-32700` and the loop RESYNCHRONIZES to the next newline (never `return`); a handler panic is contained by `recover()` → `cave_tool_panicked` (dispatch panics → `cave_internal_error`); JSON-RPC batch arrays are handled per spec (one array response); id-less/`"id":null` requests are notifications and get no reply; and both inbound lines and generated tool output (compress/toon) are size-capped (`cave_payload_too_large`, `maxInboundBytes`/`maxResultBytes`, 16 MiB default) — but `caveman_retrieve` is exempt (`Tool.ExemptResultCap`): recovery returns the byte-exact original and must never fail closed on size, since the shared gateway store has no matching ceiling. A dead server is worse than a slow one — the proxy keeps eliding content that no longer has a `caveman_retrieve` to expand it.
45- **fail-open** — engine error or malformed input → byte-identical pass-through, never a protocol error.
46- **fail-closed** — unknown tool/handle → `isError` + cave_snake_code; unknown JSON-RPC method → `-32601`.
47- **zero-egress** — the adapter imports no `net`/`net/http`/`os/exec`; a test parses the source to enforce it.
48- v1 is **stdio-only**, **string payloads only** (the engine detects type); HTTP transport + `caveman mcp` subcommand are v2.
49- **protocol negotiation must never error.** The adapter implements the 2024-11-05 contract and echoes that version back; a client asking for a newer one gets 2024-11-05 in the initialize result and decides for itself, per the MCP lifecycle. It previously answered `-32602: unsupported protocol version`, which made Claude Code (and every other current client) drop the server — and because `caveman wrap` reads recovery availability from an install-time marker rather than from the live agent, the proxy kept eliding content that no longer had a `caveman_retrieve` to expand it. Declining to echo an unimplemented version is right; refusing to speak is not.
50 
51See ../../CLAUDE.md (root) · ../engine/CLAUDE.md
52 
53## Retrieve anti-storm
54 
55`caveman_retrieve` costs a whole agent turn: the model re-reads the entire conversation prefix, and everything a previous retrieve returned is part of that prefix from then on — so N retrieves cost N turns over a transcript each one grew. A 2026-08-10 read-only sweep of 229 local CaveBench stdout files found 34 recovery sessions and 534 assistant recovery-tool calls: buckets 1 / 2–5 / >5 contained 3 / 16 / 15 sessions, p95 was 118, and max was 143. Eight of the 15 >5-call sessions still passed their exact task grader. Only 3 of 534 normalized `(handle, trimmed query)` pairs repeated exactly; many calls used new handles or pointer chains. The batches mix arms, tasks, and repetitions, so this is descriptive call-shape evidence—not a same-task counterfactual, the managed gateway's final-result population, or validation of a universal cutoff.
56 
57`EngineTools` therefore carries a per-process (= per-session) recovery ledger, and neither rule may ever withhold content the session has not already been given:
58 
591. An identical `(handle, query)` returns a one-line pointer to the answer already verbatim in the transcript, not the bytes again.
602. Past `retrieveStormThreshold` (5) distinct retrieves, the next one returns the handle's **full** stored original instead of a query-narrowed view, and says so. This is preserved historical policy intended to avoid later narrow paging; no paired experiment has validated the threshold, token effect, or task-outcome effect.
61 
62A nil `*retrieveSession` is safe and disables both, so any caller without a session concept keeps the old semantics.
63