Windsurf rules

1---
2name: run-benchmark
3model: haiku
4description: "Run skill quality benchmarks from specs/benchmarks/ definitions — N-run with/without-skill delta grading, train/validation split, pass@k + benchmark.json reports. Use before and after evolve-skill to prove quality changes are improvements, not regressions."
5---
6 
7# Run Benchmark
8 
9> **HARD GATE** — Do NOT use benchmark scores to declare a skill "good" or "bad" in isolation. Benchmarks measure relative quality vs. a baseline — they catch regressions, they do not certify correctness.
10 
11Reads benchmark definitions from `specs/benchmarks/`, executes each scenario's grader with and without the skill loaded, and writes a structured `pass@k` report with delta grading that `evolve-skill` consumes.
12 
13## With/Without-Skill Delta Grading
14 
15Every scenario runs N times (default 3) in two modes: **with** the skill loaded and **without** (bare agent with only CLAUDE.md). The delta `Δ = pass@k_with − pass@k_without` isolates the skill's causal contribution. A negative delta is a regression flag.
16 
17## Train/Validation Split
18 
19Benchmark definitions partition scenarios into two sets:
20 
21| Set | Tag | Purpose |
22|-----|-----|---------|
23| **Train** | `split: train` | Development scenarios — used while iterating. Hitting 100% on train is expected. |
24| **Validation** | `split: validation` | Held-out scenarios — the real quality signal. Overfitting train while validation stagnates is a design smell. |
25 
26`pass@k` is reported separately for train and validation. Validation score is authoritative; train score is iteration guidance only.
27 
28## Usage
29 
30```bash
31bash scripts/run-benchmark.sh <skill-name>           # benchmark single skill
32bash scripts/run-benchmark.sh --all                  # benchmark all with definitions
33bash scripts/run-benchmark.sh <skill-name> --baseline # pin results as baseline
34```
35 
36## Process
37 
381. **Locate definition** — Read `specs/benchmarks/<skill>.yaml`. If absent, stop with message.
39 
402. **Partition scenarios** — Split by `split` field (`train` → iteration, `validation` → authoritative, default: `validation`).
41 
423. **Run each scenario (N-run delta)** — For each scenario, run grader N times (default 3, configurable via `runs:`):
43   - **Without skill:** Agent with only CLAUDE.md/CONVENTIONS.md
44   - **With skill:** Agent with the skill under test active
45   - Code grader: `bash -c <command>`, exit 0 → PASS. Timeout: 15s.
46   - Rubric grader: yes/no per criterion, ≥ 80% yes → PASS.
47   - Record: `{scenario_id: {with: [P/F,...], without: [P/F,...]}}`
48 
494. **Calculate scores** — Per split (train, validation) and mode (with, without):
50   - `pass@k = sum(weight × pass_rate) / sum(weights)` where `pass_rate = passes/runs`
51   - `Δ = pass@k_with − pass@k_without` — causal contribution
52   - Round to 2 decimal places
53 
545. **Write benchmark.json** to `specs/benchmarks/reports/benchmark-<skill>.json`:
55```json
56   {"skill":"survey-context","run_date":"2026-06-22","runs_per_scenario":3,"train":{"with_skill":0.92,"without_skill":0.67,"delta":0.25,"scenarios":["s01","s02"]},"validation":{"with_skill":0.83,"without_skill":0.60,"delta":0.23,"scenarios":["s03","s04","s05"]}}
57```
58 
596. **Write YAML report** to `specs/benchmarks/reports/BENCHMARK-<skill>-<YYYY-MM-DD>.yaml`:
60```yaml
61   skill: survey-context
62   run_date: "2026-06-22"
63   runs_per_scenario: 3
64   train:
65     pass_at_k_with: 0.92
66     pass_at_k_without: 0.67
67     delta: 0.25
68   validation:
69     pass_at_k_with: 0.83
70     pass_at_k_without: 0.60
71     delta: 0.23
72   scenarios:
73     - id: s01
74       split: train
75       with_pass_rate: 1.0
76       without_pass_rate: 0.67
77       delta: 0.33
78       weight: 1.0
79```
80 
817. **Baseline** (`--baseline`) — Copy to `BASELINE-<skill>.yaml` + `baseline-<skill>.json`.
82 
838. **Compare to baseline** — `IMPROVED: Δ 0.17 → 0.25` / `REGRESSION: Δ 0.25 → 0.17 — do NOT ship` / `STABLE`.
84 
859. **Delta threshold gate** — Validation Δ < 0.0 blocks release. Δ < 0.05 warns (marginal). Min meaningful threshold: 0.05.
86