AGENTS.md

1# Terminal Compression
2 
3Guidance for the codecs and the compressed page representation
4(`Page.zig`) in this directory. These compress terminal page backing
5memory (`terminal.Page`).
6 
7## Priorities
8 
9When making tradeoffs, in order:
10 
111. **Compression ratio on page-shaped data.** Encoded bytes are retained
12   scrollback memory, and raw `terminal.Page` backing memory is the only
13   thing we actually compress. Ratio on text files or synthetic data is a
14   secondary signal.
152. **Decompression throughput.** Pages are compressed once when they go
16   cold but restored on demand (scrollback access, search, inspection), so
17   restore latency is felt directly.
183. **Compression throughput.** Runs on idle pages in the background; being
19   fast is nice, being slow is tolerable.
20 
21## Testing
22 
23- Targeted tests: `zig build test -Dtest-filter=<codec>`
24- Prefer `zig build test-lib-vt -Dtest-filter=<codec>` when practical;
25  this code ships in libghostty-vt.
26- Codecs must keep building for `wasm32-freestanding` (libghostty-vt):
27  no libc, no `src/simd` (Highway) dependencies. Verify with
28  `zig build -Demit-lib-vt -Dtarget=wasm32-freestanding -Doptimize=ReleaseSmall`.
29- Every codec needs a differential property suite: round-trip identity,
30  an independent format walker, wrong-size output rejection, and
31  corruption/truncation decoding. Keep a light version in normal unit
32  tests and gate the exhaustive version behind an environment variable so
33  the default test suite stays fast.
34 
35## Verifying Correctness
36 
37- Decoders must be memory-safe for arbitrary input bytes. Every blind or
38  wide copy needs a stated margin argument bounding it by the output
39  buffer; keep those arguments in comments next to the code.
40- Writing scratch bytes past a copy's logical end is safe only inside the
41  output buffer, because in-order decoding rewrites them before any match
42  can read them back. Do not weaken the exact-size output contract.
43- When a change should not alter compressor output, prove it: compare
44  encoded sizes (or a sequence-count fingerprint) on the same corpus
45  before and after. Ratio drift is a functional change, not noise.
46 
47## Benchmarking
48 
49- Use `ghostty-bench +page-compression` (see `src/benchmark/AGENTS.md`
50  for the general workflow). Modes: `compress`, `decompress`, `store`,
51  and `report` for ratio.
52- Build: `zig build -Demit-bench -Doptimize=ReleaseFast -Demit-macos-app=false`
53- The most representative corpus is a raw dump of real page backing
54  memory, chunked at the page size (400 KiB on ReleaseFast targets).
55  Supplement with a text corpus and random bytes for worst cases, but
56  weigh page corpora highest per the priorities above. Keep corpora
57  outside the repository and reuse identical files across comparisons.
58- `ghostty-bench +scrollback-compression` measures the PageList
59  transitions around the codec rather than the codec itself.
60- For fast iteration, keep codecs dependent only on `std` so a standalone
61  harness can build them directly with `zig build-exe -O ReleaseFast` and
62  time the codec in-process (report min-of-N, verify round-trips).
63- Measure one change at a time and re-measure the final state; run-to-run
64  noise is a few percent, so re-run before believing small deltas.
65 
66## Performance Notes
67 
68- Real page data decodes as millions of tiny operations (in LZ4: mostly
69  zero literals plus a 4-18 byte match). Per-item overhead dominates, so
70  branch-light fast paths with blind fixed-size copies win.
71- Wide copies are the only SIMD that pays here. Vectorized compares and
72  other wide-stride tricks measured as net losses because matches are
73  short; prefer the simple word loop unless a measurement on page corpora
74  says otherwise.
75- `@memcpy` beats stride loops only for long copies (roughly 64 bytes and
76  up); call overhead loses below that.
77 
78## LZ4 Specific
79 
80- The codec is `lz4.zig`, an allocation-free raw block (not frame)
81  implementation. Blocks do not carry their decoded size; callers supply
82  an exact-size output buffer.
83- Tests: `zig build test -Dtest-filter=lz4`. The differential suite is
84  `lz4_differential.zig`; run the exhaustive version for any codec
85  change:
86  `GHOSTTY_LZ4_SLOW=1 zig build test -Dtest-filter="lz4 differential"`
87- The compressor must keep the standard format restrictions (final five
88  bytes literal, matches start at least twelve bytes before the end) so
89  blocks stay consumable by optimized external decoders. The differential
90  walker checks this.
91