CLAUDE.md

1# verify-baseline-static — triage guide
2 
3Static ISA scanner. Disassembles every instruction in `.text` of a baseline
4Bun binary and flags anything the baseline CPU can't decode. Catches `-march`
5leaks at compile time, before they SIGILL on a user's machine.
6 
7This file is for triaging CI failures. For architecture details see
8`README.md` and inline comments in `src/main.rs` / `src/aarch64.rs`.
9 
10## This is a best-effort check, not a proof
11 
12A PASS here does **not** guarantee the binary is baseline-safe, and a FAIL
13does not guarantee a real bug. Treat it as a sensitive smoke detector, not an
14oracle. The emulator phase (`scripts/verify-baseline.ts`) is the complementary
15check — together they catch most things; neither alone is bulletproof.
16 
17**Out of scope entirely (tool will never find these):**
18 
19- **JIT-emitted code.** JSC compiles JS/WASM to machine code at runtime; none
20  of it exists in `.text` at scan time. If the JIT backend emits post-
21  baseline instructions on a baseline CPU, this tool is blind to it. The
22  emulator's `--jit-stress` path covers this.
23- **Dynamically loaded code.** N-API addons, FFI callees, dlopen'd shared
24  libs. Scanner only reads the `bun-profile` binary.
25- **Gate correctness.** The tool does not verify that a CPUID gate actually
26  checks the right bits. It trusts the allowlist. Feature ceilings catch the
27  "code grew new features, gate wasn't updated" case, but a gate that was
28  wrong from the start (checks AVX, uses AVX2) passes silently if the
29  ceiling says `[AVX, AVX2]`.
30 
31**In scope but may miss:**
32 
33- x64 linear-sweep may desync on data-in-`.text` and skip real instructions
34  that follow. Variable-length x86 encoding makes perfect code/data
35  separation undecidable (`README.md:53-59`). aarch64 is more reliable
36  (fixed-width words, `$d` mapping symbols mark data), but a missing mapping
37  symbol can still hide a hit.
38- Instructions deliberately ignored (TZCNT/XGETBV on x64, hint-space PAC/BTI
39  on aarch64) could theoretically be misused; we assume the compiler's idiom
40  is the only one.
41 
42**Can report false violations:**
43 
44- Data bytes in `.text` that happen to form a valid post-baseline encoding.
45  Rare on ELF (LLVM puts tables in `.rodata`), common on Windows PE (MSVC
46  inlines jump tables). See `README.md:61-74`.
47 
48When in doubt, the emulator is ground truth: `qemu -cpu Nehalem` and hit the
49code path. SIGILL = real bug. No SIGILL = either gated or a data-in-text
50false positive.
51 
52## Which builds run this
53 
54See `needsBaselineVerification()` in `.buildkite/ci.mjs`:
55 
56| Target                                                         | Allowlist file              |
57| -------------------------------------------------------------- | --------------------------- |
58| `linux-x64`, `linux-x64-musl`                                  | `allowlist-x64.txt`         |
59| `windows-x64`                                                  | `allowlist-x64-windows.txt` |
60| `linux-aarch64`, `linux-aarch64-musl`, `linux-aarch64-android` | `allowlist-aarch64.txt`     |
61 
62x64 baseline = Nehalem (`-march=nehalem`). aarch64 baseline = `armv8-a+crc`.
63Every x64 build is baseline (there is no separate `-baseline` variant).
64 
65## Reproduce a CI failure locally
66 
67The scanner runs on the _CI-built_ `-profile` artifact. You can't reproduce by
68building locally unless you build with the exact baseline toolchain. Download
69the artifact instead.
70 
711. Get `<triplet>-profile.zip` from the failing build's `build-bun` step
72   (Artifacts tab in Buildkite). Triplets look like `bun-linux-x64`,
73   `bun-linux-aarch64-musl`, `bun-windows-x64`.
74 
752. Build and run the scanner (host arch is irrelevant — the scanner reads the
76   binary's headers, it doesn't execute it):
77 
78```sh
79   cargo build --release --manifest-path scripts/verify-baseline-static/Cargo.toml
80 
81   # Linux x64 baseline
82   ./scripts/verify-baseline-static/target/release/verify-baseline-static \
83     --binary bun-linux-x64-profile/bun-profile \
84     --allowlist scripts/verify-baseline-static/allowlist-x64.txt
85 
86   # Linux aarch64
87   ./scripts/verify-baseline-static/target/release/verify-baseline-static \
88     --binary bun-linux-aarch64-profile/bun-profile \
89     --allowlist scripts/verify-baseline-static/allowlist-aarch64.txt
90 
91   # Windows x64 baseline (PDB auto-discovered at <binary>.pdb)
92   ./scripts/verify-baseline-static/target/release/verify-baseline-static \
93     --binary bun-windows-x64-profile/bun-profile.exe \
94     --allowlist scripts/verify-baseline-static/allowlist-x64-windows.txt
95```
96 
97**Never scan the stripped release binary.** It has no `.symtab` (ELF) / no
98`.pdb` (PE), so every hit becomes `<no-symbol@addr>` and nothing matches the
99allowlist.
100 
101## Reading the output
102 
103```
104VIOLATIONS (would SIGILL on Nehalem):
105 
106  _ZN7simdutf7haswell14implementation17some_new_functionEPKcm  [AVX, AVX2]  (42 insns)
107    0x0000a1b2c3  Vpbroadcastb  (AVX2)
108    0x0000a1b2d7  Vpshufb  (AVX)
109    0x0000a1b2ee  Vpcmpeqb  (AVX)
110    ... 39 more
111 
112ALLOWLISTED (suppressed, runtime-dispatched):
113  ...
114  -- 550 symbols, 18234 instructions total
115 
116STALE ALLOWLIST ENTRIES (no matching symbol found — remove these?):
117  _ZN7simdutf7haswell14implementation13old_gone_funcEPKcm
118 
119SUMMARY:
120  violations:  1 symbols, 42 instructions
121  allowlisted: 550 symbols
122  stale allowlist entries: 1
123  FAIL
124```
125 
126- Violation line format: `symbol  [FEAT, ...]  (N insns)`. Copy the symbol
127  name exactly when allowlisting — it's compared post-canonicalization.
128- Feature names are iced-x86's `CpuidFeature` Debug names (x64) or the strings
129  in `src/aarch64.rs:44-54` (aarch64). They must match the allowlist brackets
130  character-for-character.
131- `STALE` entries are informational, not an error. One allowlist covers both
132  glibc and musl; a symbol LTO'd away on one libc shows STALE on the other.
133 
134## Triage: is this an allowlist entry or a real bug?
135 
136The tool found post-baseline instructions in some symbol. Two possibilities:
137 
138**A. Runtime-dispatched.** The symbol only runs after a CPUID/HWCAP gate
139decides the CPU supports it. This is fine — allowlist it.
140 
141**B. Not gated.** A `-march` flag leaked into a translation unit that's always
142executed. Real bug, will SIGILL on baseline hardware. Fix the compile flags.
143 
144### Deciding which
145 
146**Identify the dependency.** Demangle the symbol (`c++filt`, or recognize the
147prefix: `_ZN7simdutf` = simdutf, `_ZN3bun` + `N_AVX2`/`N_SVE` = Bun's Highway
148code, `_RNv` + `memchr` = Rust memchr, etc). Search the allowlist for that
149dependency — if neighbors are there under an existing `# Gate: ...` header,
150this is almost certainly (A).
151 
152**Find the gate.** Grep for the symbol name (unmangled) in the dependency's
153source. Trace up to the caller — there should be a CPUID check, a dispatcher
154table, an HWCAP test. Known patterns:
155 
156| Dependency                            | Gate                                                        | Where                                      |
157| ------------------------------------- | ----------------------------------------------------------- | ------------------------------------------ |
158| simdutf                               | `set_best()` — CPUID first call, cached atomic ptr          | `vendor/` or WebKit's bundled copy         |
159| Highway (Bun)                         | `HWY_DYNAMIC_DISPATCH` → `hwy::SupportedTargets()`          | `src/jsc/bindings/highway_strings.cpp`     |
160| BoringSSL                             | `OPENSSL_ia32cap_P` global, set at init                     | `vendor/boringssl/crypto/cpu_intel.c`      |
161| zstd                                  | `ZSTD_cpuid()`                                              | `vendor/zstd/lib/common/cpu.h`             |
162| libdeflate                            | `libdeflate_init_x86_cpu_features()` / `HWCAP_ASIMDDP`      | `vendor/libdeflate/lib/x86/cpu_features.c` |
163| Rust `memchr`                         | `is_x86_feature_detected!()`                                | (via lolhtml dep)                          |
164| compiler-rt outline-atomics (aarch64) | `__aarch64_have_lse_atomics` (= `AT_HWCAP & HWCAP_ATOMICS`) | compiler-rt builtin                        |
165 
166**If no gate exists:** (B). Usually a subbuild that picked up host
167`-march=native` instead of the pinned `-march=nehalem` / `-mcpu=cortex-a53`.
168Fix that dep's compile flags in `scripts/build/deps/`. Confirm with the
169emulator (the ground-truth check):
170 
171```sh
172qemu-x86_64 -cpu Nehalem ./bun-profile &lt;code path that hits it&gt;   # x64 → SIGILL = bug
173qemu-aarch64 -cpu cortex-a53 ./bun-profile &lt;code path&gt;             # aarch64
174```
175 
176### Data-in-`.text` false positives (x64, mostly Windows)
177 
178Linear-sweep decode means data bytes in `.text` can happen to form a valid
179instruction encoding. LLVM puts tables in `.rodata` so ELF builds are usually
180clean; MSVC inlines jump tables and `static const` arrays into `.text`.
181 
182Signs of a false positive:
183 
184- Symbol is a lookup table or a function you _know_ contains no SIMD.
185- Reported instruction count is tiny (1–3) inside an otherwise-non-SIMD symbol.
186- `objdump -d` around the reported address shows `ret` then byte soup — no
187  stack frame setup, no control flow leading to it.
188 
189If confirmed: allowlist the symbol as a **blanket pass** (bare name, no
190`[...]` bracket). The reported features are misdecoded data bytes whose
191values move with link layout, not gated code, so a ceiling has nothing to
192bound and just re-flakes on the next layout that decodes differently. Note
193the reason in the group comment.
194 
195## Adding an allowlist entry
196 
197Append the symbol to the appropriate file. Group with its neighbors under the
198existing `# Gate: ...` header; if no existing group matches, add one:
199 
200```
201# ----------------------------------------------------------------------------
202# <dependency> <variant>. Gate: <what checks CPUID/HWCAP>.
203# (N symbols)
204# ----------------------------------------------------------------------------
205symbol_name_exactly_as_the_tool_printed_it  [FEAT1, FEAT2]
206```
207 
208**Use a feature ceiling** (`[...]`) for gated code. A blanket pass (no
209brackets) defeats the "did the gate get updated when the dep grew AVX-512?"
210check (`src/main.rs:616-621`). List exactly the features the tool reported;
211that's what the gate currently checks. The exceptions are confirmed
212data-in-.text misdecodes (previous section) and `<no-symbol@...>` padding
213(below): there is no gate to drift past, so blanket-pass those.
214 
215**x64 feature names** (iced-x86 Debug strings — must match exactly):
216`AVX`, `AVX2`, `FMA`, `FMA4`, `BMI1`, `BMI2`, `MOVBE`, `ADX`, `RDRAND`,
217`AES`, `PCLMULQDQ`, `VAES`, `VPCLMULQDQ`, `SHA`, `AVX512F`, `AVX512BW`,
218`AVX512DQ`, `AVX512VL`, `AVX512_VBMI`, `AVX512_VBMI2`, `AVX512_VNNI`,
219`AVX512_VPOPCNTDQ`, `AVX512_FP16`, `AVX_VNNI`, …
220 
221**aarch64 feature names:** `LSE`, `SVE`, `RCPC`, `DotProd`, `JSCVT`, `RDM`,
222`PAC(non-hint)`.
223 
224### Special symbol forms
225 
226**Rust v0 mangling — `<rust-hash>`.** Rust symbols contain a crate-hash
227(`Cs[base62]_`) that changes across target triples and toolchains. The tool
228canonicalizes both sides (`src/main.rs:196-227`), so allowlist entries should
229use `<rust-hash>` in place of the hash:
230 
231```
232# Tool reports:
233  _RNvMNtNtNtNtCs5QMN7YRSXc3_6memchr4arch6x86_644avx26memchrNtB2_3One13find_raw_avx2  [AVX, AVX2]
234# Allowlist as:
235  _RNvMNtNtNtNt<rust-hash>6memchr4arch6x86_644avx26memchrNtB2_3One13find_raw_avx2  [AVX, AVX2]
236```
237 
238Either form works (the tool canonicalizes both before comparing), but
239`<rust-hash>` survives toolchain bumps.
240 
241**Windows `<lib:NAME.lib>`.** When PDB has no per-function record for a hit
242(stripped CRT objects, anonymized staticlib helpers), the tool falls back to
243section-contribution attribution: the linker-map "which `.lib` did this byte
244come from" data. These attributions are stable across link layout changes.
245Allowlist them literally:
246 
247```
248<lib:lolhtml.lib>  [AVX, AVX2]
249```
250 
251**`<no-symbol@0x...>`** — the address fell in padding between functions or the
252binary is stripped. If you see these for every violation, you're scanning the
253wrong binary (use `-profile`). If it's just one or two, it's usually inter-
254function padding that decoded as something; investigate with `objdump -d`
255around that address and, if it's genuinely junk, add a brief `# padding at
256<addr range>` comment with a blanket-pass entry.
257 
258### PDB coverage drift (Windows)
259 
260A function may get an `S_LPROC32` record (real mangled name) on one toolchain
261and fall through to `<lib:...>` on another. If the same code flips between
262forms across CI runs, allowlist both.
263 
264## Deliberately ignored (not reported even if found)
265 
266See `src/main.rs:94-135` and `src/aarch64.rs`:
267 
268- **TZCNT** (x64) — decodes as REP BSF on pre-BMI1; LLVM preloads dest with
269  operand-width so the `src==0` case matches. (LZCNT is NOT ignored —
270  `BSR` ≠ `LZCNT` for nonzero inputs and LLVM never emits it for Nehalem.)
271- **XGETBV** (x64) — needed by every AVX gate; a stray one SIGILLs at
272  startup so the emulator catches it trivially.
273- **ENDBR64 (CET_IBT), RDSSP/INCSSP (CET_SS hint-space subset)** (x64) —
274  NOP-encoded on pre-CET by design. The rest of CET_SS (WRSSD/RSTORSSP/
275  SETSSBSY etc.) IS flagged — dedicated opcode slots that #UD on pre-CET.
276- **PACIASP/AUTIASP/BTI** (aarch64) — HINT-space, architecturally NOP on
277  pre-PAC CPUs. (`LDRAA`/`LDRAB` are _not_ HINT-space and _are_ reported.)
278- **3DNow!, SMM, Cyrix, VIA, RTM/TSX** (x64) — no toolchain targeting x86-64
279  emits these without explicit intrinsics. When their encodings show up
280  (`0f 0f` 3DNow!, `C7/C6 F8` XBEGIN/XABORT), it's data.
281