AGENTS.md

1# Benchmarking
2 
3The benchmark tools are split into two roles:
4 
5- `ghostty-gen` generates synthetic input data.
6- `ghostty-bench` consumes existing input data and runs a benchmark.
7 
8## Workflow
9 
10- For timing comparisons, generate data first and benchmark it later.
11- Do not pipe `ghostty-gen` directly into `ghostty-bench` when comparing
12  performance. That mixes generation cost into the measurement and makes
13  branch-to-branch comparisons noisy.
14- Reuse the exact same generated files when comparing revisions.
15- Prefer deterministic generation inputs such as fixed seeds when the
16  generator supports them.
17- Keep large generated benchmark corpora outside the repository unless the
18  change explicitly requires checked-in test data.
19 
20## Running Benchmarks
21 
22- Prefer `hyperfine` to compare benchmark timings.
23- Benchmark the `ghostty-bench` command line, not the generator.
24- Use `ghostty-bench ... --data <path>` with pre-generated files.
25- Run multiple warmups and repeated measurements so branch comparisons are
26  based on medians instead of single runs.
27- When comparing branches, keep all benchmark inputs and CLI flags the same,
28  including terminal dimensions.
29- Never run multiple benchmarks in parallel on the same machine, as they will
30  interfere with each other and produce unreliable results.
31 
32## Building
33 
34- Build benchmark tools with `zig build -Demit-bench -Doptimize=ReleaseFast`.
35- On macOS, add `-Demit-macos-app=false` to avoid building the macOS app.
36- Make sure you specify `-Doptimize=ReleaseFast` when building benchmarks,
37  otherwise the debug build will be very slow and not representative of real
38  performance.
39 
40## Comparing Branches
41 
42- When comparing branches, switch to that branch, build the binary, then
43  rename it e.g. `zig-out/bin/ghostty-bench` to `zig-out/bin/ghostty-bench-branch1`.
44  Replace branch1 with something better.
45- Then switch to the other branch, build it, and rename it to
46  `zig-out/bin/ghostty-bench-branch2`. Replace branch2 with something better.
47- Then run all the benchmarks with `hyperfine` comparing the N binaries
48  we want to.
49