Windsurf rules

1---
2name: run-evals
3model: sonnet
4description: "Eval-Driven Development — define capability and regression evals before building; code graders use verify commands, model graders use explicit rubrics; log pass@k. Use before develop-tdd on new features, or when measuring agent capability over runs."
5---
6 
7# Run Evals
8 
9> **HARD GATE** — Define evals before implementation. Code graders = runnable `verify:` commands; model graders = explicit rubric with pass/fail criteria.
10 
11## Process
12 
131. Name the capability under test (one sentence).
142. Write `specs/EVALS-<feature>.md` with:
15   - **Capability evals** (does it do the job?)
16   - **Regression evals** (did we break anything?)
173. Assign grader type per eval: `code` (shell verify) or `model` (rubric).
184. Assign **strictness tier** per eval (graduated promotion — e45s37):
19 
20   | Tier | Meaning | Promotion rule |
21   |------|---------|------------------|
22   | `EXPERIMENTAL` | New eval, may flake | Not gating |
23   | `USUALLY_PASSES` | Stable in dev; ≥2/3 recent runs pass | Blocks BUILD only when combined with ALWAYS_PASSES suite |
24   | `ALWAYS_PASSES` | Zero tolerance; required for release | Any single failure blocks BUILD and merge |
25 
26   Promote: `EXPERIMENTAL → USUALLY_PASSES` after 3 consecutive passes; `USUALLY_PASSES → ALWAYS_PASSES` after 5 consecutive passes with zero flakes documented in `specs/state.yaml`.
27 
285. Run evals; log results table with pass@k (e.g. 3/3 runs) and tier per eval.
296. Block BUILD phase until all `ALWAYS_PASSES` evals pass at agreed k. `USUALLY_PASSES` failures warn; `EXPERIMENTAL` failures log only.
30 
31## Artefact
32 
33`specs/verifications/eNNsYY-eval-report.md` — see [REFERENCE.md](REFERENCE.md) for template. Eval reports are stored alongside verification evidence in `specs/verifications/`, keyed by story ID for traceability.
34 
35## Verify
36 
37→ verify: `test -d specs/benchmarks && test -f specs/benchmarks/SCHEMA.md`
38 
39 
40<!-- story: e02s01 -->
41 
42---
43 
44# Run Evals — Reference
45 
46## Strictness tiers (e45s37)
47 
48Add a `tier:` column to each eval row:
49 
50| Tier | Gate behaviour |
51|------|----------------|
52| `EXPERIMENTAL` | Log only — does not block |
53| `USUALLY_PASSES` | Warn on failure; blocks only when paired with failing `ALWAYS_PASSES` |
54| `ALWAYS_PASSES` | Hard block on any failure |
55 
56## EVALS template
57 
58```markdown
59# EVALS: <feature>
60 
61## Capability
62| ID | Eval | Grader | Tier | verify / rubric |
63|----|------|--------|------|-----------------|
64| C1 | ... | code | ALWAYS_PASSES | `verify: npm test -- <file>` |
65| C2 | ... | model | USUALLY_PASSES | Rubric: [ ] criterion A [ ] criterion B |
66 
67## Regression
68| ID | Eval | Grader | verify / rubric |
69|----|------|--------|-----------------|
70| R1 | Full suite passes | code | `verify: npm test` |
71 
72## Results
73| Run | C1 | C2 | R1 | pass@k |
74|-----|----|----|-----|--------|
75| 1 | PASS | PASS | PASS | 3/3 |
76```
77 
78## pass@k
79 
80Run capability evals k times (default k=3). Ship when all k pass or document known flake in `specs/state.yaml` `handoff.open_decisions`.
81