AGENTS.md

1# SDP-Eval-Core v1 Runtime Contract
2 
3## Context Objects
4- **Case**: Primary input object defining a single evaluation case (YAML)
5- **Result**: Primary output object containing case execution outcome
6- **Transcripts**: JSONL format with role+content per line
7 
8## Environment
9- No required environment variables
10- File system access required for transcript and case file I/O
11- Case files located in `evals/cases/` or configurable directory
12 
13## Configuration
14- Case definition format: YAML with fields:
15  - `name`: case identifier
16  - `skill`: associated skill name
17  - `input_transcript`: path to JSONL transcript
18  - `forbidden_patterns`: patterns that must NOT appear in output
19  - `required_patterns`: patterns that MUST appear in output
20  - `verdict`: expected outcome ("PASS" or "FAIL")
21 
22## Dependencies
23- External: `gopkg.in/yaml.v3` for YAML case parsing
24- Stdlib: `encoding/json` for JSONL transcript parsing
25 
26## Transcript Format
27- JSONL (one JSON object per line)
28- Each line: `{"role": "...", "content": "..."}` or nested message structure
29- Roles: `assistant` messages are extracted for evaluation
30 
31## Evidence-Mismatch Metric (v1)
32- Replaces hallucination rate per IIP council decision
33- Measures governance-decision accuracy
34- Computed as: (mismatched decisions) / (total decisions)
35- Lower rates indicate better alignment between decisions and evidence
36