CLAUDE.md

1# agentcairn — project guide for Claude Code
2 
3**What this is:** a local-first agent-memory system. An Obsidian **Markdown vault is the source of truth**; a **rebuildable embedded DuckDB index** provides hybrid retrieval. Daemonless: a `cairn` CLI + an on-demand READ_ONLY MCP server. Capture is via **out-of-band transcript ingestion** + an agent `remember` tool — **not** live hooks.
4 
5**Status (2026-06-13):** Shipped and on PyPI — current **0.10.1**. The full loop is implemented
6(`cairn.vault/index/embed/search/ingest/mcp` + CLI + Claude Code plugin). Specs live in `docs/specs/`
7(the original is `2026-06-08-agentcairn-design.md`; later ones cover the config file, the Layer-B LLM
8judge, antecedent resolution, and memory consolidation). Releases follow the cut-a-release ritual
9(CHANGELOG → tag → Trusted-Publishing → GitHub Release).
10 
11**Plugins:** agentcairn ships a **Claude Code plugin**, a **Codex plugin**, and an **Antigravity plugin** — all three reuse the same `using-agentcairn-memory` skill and bundled MCP server. The Codex plugin is discoverable via the Codex plugin marketplace; the Antigravity plugin installs from a local directory (`agy plugin install` takes a dir or a registered marketplace, not a git repo — so `cairn install antigravity --source <plugin dir>`). `cairn install` routes by host kind: for plugin hosts (`claude-code`, `codex`, `antigravity`) it installs the plugin by shelling to the host's own CLI; for MCP hosts (`cursor`, `claude-desktop`, `vscode`, `gemini`) it writes the MCP server config. The Antigravity plugin bundles the MCP server + skill; `cairn install antigravity` (via `agy plugin install`) also migrates away any stale `mcpServers.agentcairn` entry from `~/.gemini/config/mcp_config.json`. Gemini, Cursor, and the other MCP hosts do not have plugins. **Cursor is the exception among MCP hosts:** it has no plugin system, but `cairn install cursor` ALSO installs the `using-agentcairn-memory` skill (alongside writing `~/.cursor/mcp.json`) via Cursor's `Host.skill_dir` (`~/.cursor/skills`), writing `~/.cursor/skills/using-agentcairn-memory/SKILL.md` — so any future host exposing a `skill_dir` can ship the memory skill the same way without being a plugin host.
12 
13**Capture pipeline (ingest):** transcripts from Claude Code, Codex, Antigravity CLI (`~/.gemini/antigravity-cli/brain/<uuid>/.system_generated/logs/transcript.jsonl`), and Cursor (`<CursorUser>/globalStorage/state.vscdb`, SQLite `cursorDiskKV`) are ingested automatically
14(auto-detected via a `HarnessAdapter` seam; `cairn sweep` picks up whichever harnesses are present). Gemini CLI transcript ingestion is not supported (Google is sunsetting Gemini CLI in favour of Antigravity CLI).
15Then: redact → structural candidate selection (authored user turns only) →
16judge (embedding/LLM tier; durability + optional distillation) → gate → distill → **consolidate**
17(semantic dedup + supersession, LLM-tier, fail-safe) → write. Redaction-before-write is mandatory;
18the judged cache is version-stamped (`_JUDGE_CACHE_VERSION`).
19 
20## Locked decisions
21- **Name:** package/org/repo `agentcairn`; **CLI command `cairn`**.
22- **Language:** Python 3.11+. Distribute via `uv`/`uvx`/`pipx`; MCP launched via `uvx agentcairn`.
23- **Index is a disposable cache** — always rebuildable from Markdown (`cairn reindex`); never the source of truth.
24- **Default embedder:** FastEmbed `nomic-embed-text-v1.5` (768-d) — a 5-model sweep settled it (best quality-per-dim; beat the original `bge-small-en-v1.5` 384-d default). Pluggable interface (Ollama/cloud opt-in).
25- **Retrieval:** BM25 + vector + wikilink-graph-boost + recency/importance, fused with RRF (k=60); degradation ladder down to BM25-only ("never silently dead").
26- **Provenance-aware recall:** notes carry `project`/`harness` provenance in frontmatter + index; recall boosts the current project (resolved from an explicit arg or the process cwd) ×1.4 non-lossily, with `--scope project` to hard-filter to it.
27- **Concurrency:** MCP opens DuckDB **READ_ONLY**; one short-lived CLI process is the sole writer. The `.duckdb` lives on **local disk, NOT inside the synced vault folder**.
28- **Markdown contract:** basic-memory conventions (frontmatter `title/type/permalink/tags`; observations `- [category] text #tag (ctx)`; relations `- rel_type [[Target]]`; bare `[[link]]` ⇒ implicit `links_to`) + Dataview-compatible inline fields.
29 
30## Hard parts / constraints (don't relearn the hard way)
31- `cairn.vault` (parse/write Markdown preserving frontmatter order, unresolved forward-refs, and link-rewrite-on-move) is the **hardest** component — build and test it first.
32- **Secret redaction before any write is mandatory** (we write plaintext to disk).
33- **Brute-force cosine is the right default; HNSW is not worth building yet.** Measured 2026-06-15 (`benchmarks/cairn_bench.latency`, see benchmarks/README.md §Retrieval latency): the size-dependent cosine *scan* is <2 ms at ≤1k chunks, ~14 ms at 10k, and only crosses a 100 ms budget at ~100k chunks — typical vaults are hundreds–low-thousands of chunks. A fixed ~39 ms/query is the query-vector Python-list→DuckDB bind (size-independent, HNSW-irrelevant), which dominates recall latency at small scale. Revisit HNSW only as vaults approach ~50k chunks. If HNSW is built: DuckDB VSS HNSW persistence is *experimental* → prefer **in-memory HNSW rebuilt at MCP spawn**. DuckDB-WASM has **no VSS** → a future Obsidian plugin gets BM25+graph only (no in-browser semantic search).
34- **No single headline benchmark number** — vendor LoCoMo/LongMemEval figures are self-reported and disputed; validate on LongMemEval-S + LoCoMo with committed scripts before any comparative claim.
35 
36## The wedge (keep it crisp)
37Closest competitor is `rohitg00/agentmemory` (it already has hybrid+graph, local embeddings, decay, Obsidian *export*). Our narrow-but-real edge: **vault-as-truth (not export) · disposable/rebuildable index · non-lossy by construction · free wikilink graph · daemonless, zero external DB.** If a change blurs this, reconsider it.
38 
39## Tooling (uv — exclusive)
40 
41**Always use `uv`; never use pip, poetry, or global venvs.**
42 
43- Install / sync deps: `uv sync`
44- Run any command: `uv run <cmd>` (e.g. `uv run pytest`, `uv run cairn --help`)
45- Ephemeral tool runs: `uvx <tool>` — the MCP server will be launched as `uvx agentcairn`
46- Contributor setup: `uv sync` then `uv run pre-commit install`
47 
48`uv.lock` is **committed** — it keeps the tool reproducible across machines.
49 
50## Conventions
51- Specs/designs → `docs/specs/` (date-prefixed `YYYY-MM-DD-<topic>-design.md`). No `superpowers/` segment.
52- `README.md` is the public positioning doc — advantages framed as design goals until validated.
53- **License: Apache-2.0.** Start each source file with `# SPDX-License-Identifier: Apache-2.0`; keep `NOTICE` intact and propagate it in distributions; set `license = "Apache-2.0"` in `pyproject.toml` when scaffolding.
54 
55## Open items
56- ~~Validate large-vault HNSW rebuild / MCP cold-start latency~~ — **done 2026-06-15**: brute-force scan stays sub-budget to ~50k chunks; HNSW deferred (see benchmarks/README.md §Retrieval latency).
57- Validate the local embedding baseline on a QA-style metric, not just R@5.
58