CLAUDE.md

1# CLAUDE.md - Direct Benchmark Harness
2 
3This file provides guidance to Claude Code when working with the direct benchmark harness.
4 
5## Overview
6 
7The Direct Benchmark Harness is a high-performance testing framework for AutoGPT that directly instantiates agents without HTTP server overhead. It enables parallel execution of multiple strategy/model configurations.
8 
9## Quick Reference
10 
11All commands run from the `classic/` directory (parent of this directory):
12 
13```bash
14# Install (one-time setup)
15cd classic
16poetry install
17 
18# Run benchmarks
19poetry run direct-benchmark run
20 
21# Run specific strategies and models
22poetry run direct-benchmark run \
23    --strategies one_shot,rewoo \
24    --models claude,openai \
25    --parallel 4
26 
27# Run a single test
28poetry run direct-benchmark run \
29    --strategies one_shot \
30    --tests ReadFile
31 
32# List available challenges
33poetry run direct-benchmark list-challenges
34 
35# List model presets
36poetry run direct-benchmark list-models
37 
38# List strategies
39poetry run direct-benchmark list-strategies
40```
41 
42## CLI Options
43 
44### Run Command
45 
46| Option | Short | Description |
47|--------|-------|-------------|
48| `--strategies` | `-s` | Comma-separated strategies (one_shot, rewoo, plan_execute, reflexion, tree_of_thoughts) |
49| `--models` | `-m` | Comma-separated model presets (claude, openai, etc.) |
50| `--categories` | `-c` | Filter by challenge categories |
51| `--skip-category` | `-S` | Exclude categories |
52| `--tests` | `-t` | Filter by test names |
53| `--attempts` | `-N` | Number of times to run each challenge |
54| `--parallel` | `-p` | Maximum parallel runs (default: 4) |
55| `--timeout` | | Per-challenge timeout in seconds (default: 300) |
56| `--cutoff` | | Alias for --timeout |
57| `--no-cutoff` | `--nc` | Disable time limit |
58| `--max-steps` | | Maximum steps per challenge (default: 50) |
59| `--maintain` | | Run only regression tests |
60| `--improve` | | Run only non-regression tests |
61| `--explore` | | Run only never-beaten challenges |
62| `--no-dep` | | Ignore challenge dependencies |
63| `--workspace` | | Workspace root directory |
64| `--challenges-dir` | | Path to challenges directory |
65| `--reports-dir` | | Path to reports directory |
66| `--keep-answers` | | Keep answer files for debugging |
67| `--quiet` | `-q` | Minimal output |
68| `--verbose` | `-v` | Detailed per-challenge output |
69| `--json` | | JSON output for CI/scripting |
70| `--ci` | | CI mode: no live display, shows completion blocks (auto-enabled when CI env var is set or not a TTY) |
71| `--fresh` | | Clear all saved state and start fresh (don't resume) |
72| `--retry-failures` | | Re-run only the challenges that failed in previous run |
73| `--reset-strategy` | | Reset saved results for specific strategy (can repeat) |
74| `--reset-model` | | Reset saved results for specific model (can repeat) |
75| `--reset-challenge` | | Reset saved results for specific challenge (can repeat) |
76| `--debug` | | Enable debug output |
77 
78### State Management Commands
79```bash
80# Show current state
81poetry run direct-benchmark state show
82 
83# Clear all state
84poetry run direct-benchmark state clear
85 
86# Reset specific strategy/model/challenge
87poetry run direct-benchmark state reset --strategy reflexion
88poetry run direct-benchmark state reset --model claude-thinking-25k
89poetry run direct-benchmark state reset --challenge ThreeSum
90```
91 
92## Available Strategies
93 
94- `one_shot` - Single-pass reasoning (default)
95- `rewoo` - Reasoning with observations
96- `plan_execute` - Plan then execute
97- `reflexion` - Self-reflection loop
98- `tree_of_thoughts` - Multiple reasoning paths
99 
100## Available Model Presets
101 
102### Claude
103- `claude` - sonnet-4 smart, haiku fast
104- `claude-smart` - sonnet-4 for both
105- `claude-fast` - haiku for both
106- `claude-opus` - opus smart, sonnet fast
107- `claude-opus-only` - opus for both
108 
109### Claude with Extended Thinking
110- `claude-thinking-10k` - 10k thinking tokens
111- `claude-thinking-25k` - 25k thinking tokens
112- `claude-thinking-50k` - 50k thinking tokens
113- `claude-opus-thinking` - opus with 25k thinking
114- `claude-opus-thinking-50k` - opus with 50k thinking
115 
116### OpenAI
117- `openai` - gpt-4o smart, gpt-4o-mini fast
118- `openai-smart` - gpt-4o for both
119- `openai-fast` - gpt-4o-mini for both
120- `gpt5` - gpt-5 smart, gpt-4o fast
121- `gpt5-only` - gpt-5 for both
122 
123### OpenAI Reasoning Models
124- `o1`, `o1-mini` - o1 variants
125- `o1-low`, `o1-medium`, `o1-high` - o1 with reasoning effort
126- `o3-low`, `o3-medium`, `o3-high` - o3 with reasoning effort
127- `gpt5-low`, `gpt5-medium`, `gpt5-high` - gpt-5 with reasoning effort
128 
129## Directory Structure
130 
131```
132direct_benchmark/
133├── pyproject.toml           # Poetry config
134├── README.md                 # User documentation
135├── CLAUDE.md                 # This file
136├── .gitignore
137└── direct_benchmark/
138    ├── __init__.py
139    ├── __main__.py           # CLI entry point
140    ├── models.py             # Pydantic models, presets
141    ├── harness.py            # Main orchestrator
142    ├── runner.py             # AgentRunner (single agent lifecycle)
143    ├── parallel.py           # ParallelExecutor (concurrent runs)
144    ├── challenge_loader.py   # Load challenges from JSON
145    ├── evaluator.py          # Evaluate outputs vs ground truth
146    ├── report.py             # Report generation
147    └── ui.py                 # Rich UI components
148```
149 
150## Architecture
151 
152### Execution Flow
153 
154```
155CLI args → HarnessConfig
156    ↓
157BenchmarkHarness.run()
158    ↓
159ChallengeLoader.load_all() → list[Challenge]
160    ↓
161ParallelExecutor.execute_matrix(configs × challenges × attempts)
162    ↓
163[Parallel with semaphore limiting to N concurrent]
164    ↓
165AgentRunner.run_challenge():
166  1. Create temp workspace
167  2. Copy input artifacts to agent workspace
168  3. Create AppConfig with strategy/model
169  4. create_agent() - direct instantiation
170  5. Run agent loop until finish/timeout
171  6. Collect output files
172    ↓
173Evaluator.evaluate() - check against ground truth
174    ↓
175ReportGenerator - write reports
176```
177 
178### Key Components
179 
180**AgentRunner** (`runner.py`)
181- Manages single agent lifecycle for one challenge
182- Creates isolated temp workspace per run
183- Copies input artifacts to `{workspace}/.autogpt/agents/{agent_id}/workspace/`
184- Instantiates agent directly via `create_agent()`
185- Runs agent loop: `propose_action()` → `execute()` until finish/timeout
186 
187**ParallelExecutor** (`parallel.py`)
188- Manages concurrent execution with asyncio semaphore
189- Supports multiple attempts per challenge
190- Reports progress via callbacks
191 
192**Evaluator** (`evaluator.py`)
193- String matching (should_contain/should_not_contain)
194- Python script execution
195- Pytest execution
196 
197**ReportGenerator** (`report.py`)
198- Per-config `report.json` files (compatible with agbenchmark format)
199- Comparison reports across all configs
200 
201## Report Format
202 
203Reports are generated in `./reports/` with format:
204```
205reports/
206├── {timestamp}_{strategy}_{model}/
207│   └── report.json
208└── strategy_comparison_{timestamp}.json
209```
210 
211## Dependencies
212 
213- `autogpt-forge` - Core agent framework
214- `autogpt` - Original AutoGPT agent
215- `click` - CLI framework
216- `pydantic` - Data models
217- `rich` - Terminal UI
218 
219## Key Differences from agbenchmark
220 
221| agbenchmark | direct_benchmark |
222|-------------|-----------------|
223| `subprocess.Popen` + HTTP server | Direct `create_agent()` |
224| HTTP/REST via Agent Protocol | Direct `propose_action()`/`execute()` |
225| Sequential (one config at a time) | Parallel via asyncio semaphore |
226| Port-based isolation | Workspace-based isolation |
227| `agbenchmark run` CLI | Direct JSON parsing |
228 
229## Common Tasks
230 
231### Run Full Benchmark Suite
232```bash
233poetry run direct-benchmark run \
234    --strategies one_shot,rewoo,plan_execute \
235    --models claude \
236    --parallel 8
237```
238 
239### Compare Strategies
240```bash
241poetry run direct-benchmark run \
242    --strategies one_shot,rewoo,plan_execute,reflexion \
243    --models claude \
244    --tests ReadFile,WriteFile,ThreeSum
245```
246 
247### Debug a Failing Test
248```bash
249poetry run direct-benchmark run \
250    --strategies one_shot \
251    --tests FailingTest \
252    --keep-answers \
253    --verbose
254```
255 
256### Resume / Incremental Runs
257The benchmark automatically saves progress and resumes from where it left off.
258State is saved to `.benchmark_state.json` in the reports directory.
259 
260```bash
261# Run benchmarks - will resume from last run automatically
262poetry run direct-benchmark run \
263    --strategies one_shot,reflexion \
264    --models claude
265 
266# Start fresh (clear all saved state)
267poetry run direct-benchmark run --fresh \
268    --strategies one_shot,reflexion \
269    --models claude
270 
271# Reset specific strategy and re-run
272poetry run direct-benchmark run \
273    --reset-strategy reflexion \
274    --strategies one_shot,reflexion \
275    --models claude
276 
277# Reset specific model and re-run
278poetry run direct-benchmark run \
279    --reset-model claude-thinking-25k \
280    --strategies one_shot \
281    --models claude,claude-thinking-25k
282 
283# Retry only the failures from the last run
284poetry run direct-benchmark run --retry-failures \
285    --strategies one_shot,reflexion \
286    --models claude
287```
288 
289### CI/Scripting Mode
290```bash
291# JSON output (parseable)
292poetry run direct-benchmark run --json
293 
294# CI mode - shows completion blocks without Live display
295# Auto-enabled when CI=true env var is set or stdout is not a TTY
296poetry run direct-benchmark run --ci
297```
298