CLAUDE.md

1# CLAUDE.md
2 
3This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4 
5## Quick Reference
6 
7All commands run from the `classic/` directory (parent of this directory):
8 
9```bash
10# Run forge agent server (port 8000)
11poetry run python -m forge
12 
13# Run forge tests
14poetry run pytest forge/tests/
15poetry run pytest forge/tests/ --cov=forge
16poetry run pytest -k test_name
17```
18 
19## Entry Point
20 
21`__main__.py` → loads `.env` → configures logging → starts Uvicorn with hot-reload on port 8000
22 
23The app is created in `app.py`:
24```python
25agent = ForgeAgent(database=database, workspace=workspace)
26app = agent.get_agent_app()
27```
28 
29## Directory Structure
30 
31```
32forge/
33├── __main__.py               # Entry: uvicorn server startup
34├── app.py                    # FastAPI app creation
35├── agent/                    # Core agent framework
36│   ├── base.py               # BaseAgent abstract class
37│   ├── forge_agent.py        # Reference implementation
38│   ├── components.py         # AgentComponent base classes
39│   └── protocols.py          # Protocol interfaces
40├── agent_protocol/           # Agent Protocol standard
41│   ├── agent.py              # ProtocolAgent mixin
42│   ├── api_router.py         # FastAPI routes
43│   └── database/             # Task/step persistence
44├── command/                  # Command system
45│   ├── command.py            # Command class
46│   ├── decorator.py          # @command decorator
47│   └── parameter.py          # CommandParameter
48├── components/               # Built-in components
49│   ├── action_history/       # Track & summarize actions
50│   ├── code_executor/        # Python & shell execution
51│   ├── context/              # File/folder context
52│   ├── file_manager/         # File operations
53│   ├── git_operations/       # Git commands
54│   ├── image_gen/            # DALL-E & SD
55│   ├── system/               # Core directives + finish
56│   ├── user_interaction/     # User prompts
57│   ├── watchdog/             # Loop detection
58│   └── web/                  # Search & Selenium
59├── config/                   # Configuration models
60├── llm/                      # LLM integration
61│   └── providers/            # OpenAI, Anthropic, Groq, etc.
62├── file_storage/             # Storage abstraction
63│   ├── base.py               # FileStorage ABC
64│   ├── local.py              # LocalFileStorage
65│   ├── s3.py                 # S3FileStorage
66│   └── gcs.py                # GCSFileStorage
67├── models/                   # Core data models
68├── content_processing/       # Text/HTML utilities
69├── logging/                  # Structured logging
70└── json/                     # JSON parsing utilities
71```
72 
73## Core Abstractions
74 
75### BaseAgent (`agent/base.py`)
76 
77Abstract base for all agents. Generic over proposal type.
78 
79```python
80class BaseAgent(Generic[AnyProposal], metaclass=AgentMeta):
81    def __init__(self, settings: BaseAgentSettings)
82```
83 
84**Must Override:**
85```python
86async def propose_action(self) -> AnyProposal
87async def execute(self, proposal: AnyProposal, user_feedback: str) -> ActionResult
88async def do_not_execute(self, denied_proposal: AnyProposal, user_feedback: str) -> ActionResult
89```
90 
91**Key Methods:**
92```python
93async def run_pipeline(protocol_method, *args, retry_limit=3) -> list
94# Executes protocol across all matching components with retry logic
95 
96def dump_component_configs(self) -> str  # Serialize configs to JSON
97def load_component_configs(self, json: str)  # Restore configs
98```
99 
100**Configuration (`BaseAgentConfiguration`):**
101```python
102fast_llm: ModelName = "gpt-3.5-turbo-16k"
103smart_llm: ModelName = "gpt-4"
104big_brain: bool = True              # Use smart_llm
105cycle_budget: Optional[int] = 1     # Steps before approval needed
106send_token_limit: Optional[int]     # Prompt token budget
107```
108 
109### Component System (`agent/components.py`)
110 
111**AgentComponent** - Base for all components:
112```python
113class AgentComponent(ABC):
114    _run_after: list[type[AgentComponent]] = []
115    _enabled: bool | Callable[[], bool] = True
116    _disabled_reason: str = ""
117 
118    def run_after(self, *components) -> Self  # Set execution order
119    def enabled(self) -> bool                  # Check if active
120```
121 
122**ConfigurableComponent** - Components with Pydantic config:
123```python
124class ConfigurableComponent(Generic[BM]):
125    config_class: ClassVar[type[BM]]  # Set in subclass
126 
127    @property
128    def config(self) -> BM  # Get/create config from env
129```
130 
131**Component Discovery:**
1321. Agent assigns components: `self.foo = FooComponent()`
1332. `AgentMeta.__call__` triggers `_collect_components()`
1343. Components are topologically sorted by `run_after` dependencies
1354. Disabled components skipped during pipeline execution
136 
137### Protocols (`agent/protocols.py`)
138 
139Protocols define what components CAN do:
140 
141```python
142class DirectiveProvider(AgentComponent):
143    def get_constraints(self) -> Iterator[str]
144    def get_resources(self) -> Iterator[str]
145    def get_best_practices(self) -> Iterator[str]
146 
147class CommandProvider(AgentComponent):
148    def get_commands(self) -> Iterator[Command]
149 
150class MessageProvider(AgentComponent):
151    def get_messages(self) -> Iterator[ChatMessage]
152 
153class AfterParse(AgentComponent, Generic[AnyProposal]):
154    def after_parse(self, result: AnyProposal) -> None
155 
156class AfterExecute(AgentComponent):
157    def after_execute(self, result: ActionResult) -> None
158 
159class ExecutionFailure(AgentComponent):
160    def execution_failure(self, error: Exception) -> None
161```
162 
163**Pipeline execution:**
164```python
165results = await self.run_pipeline(CommandProvider.get_commands)
166# Iterates all components implementing CommandProvider
167# Collects all yielded Commands
168# Handles retries on ComponentEndpointError
169```
170 
171## LLM Providers (`llm/providers/`)
172 
173### MultiProvider
174 
175Routes to correct provider based on model name:
176 
177```python
178class MultiProvider:
179    async def create_chat_completion(
180        self,
181        model_prompt: list[ChatMessage],
182        model_name: ModelName,
183        **kwargs
184    ) -> ChatModelResponse
185 
186    async def get_available_chat_models(self) -> Sequence[ChatModelInfo]
187```
188 
189### Supported Models
190 
191```python
192# OpenAI
193OpenAIModelName.GPT3, GPT3_16k, GPT4, GPT4_32k, GPT4_TURBO, GPT4_O
194 
195# Anthropic
196AnthropicModelName.CLAUDE3_OPUS, CLAUDE3_SONNET, CLAUDE3_HAIKU
197AnthropicModelName.CLAUDE3_5_SONNET, CLAUDE3_5_SONNET_v2, CLAUDE3_5_HAIKU
198AnthropicModelName.CLAUDE4_SONNET, CLAUDE4_OPUS, CLAUDE4_5_OPUS
199 
200# Groq
201GroqModelName.LLAMA3_8B, LLAMA3_70B, MIXTRAL_8X7B
202```
203 
204### Key Types
205 
206```python
207class ChatMessage(BaseModel):
208    role: Role  # USER, SYSTEM, ASSISTANT, TOOL, FUNCTION
209    content: str
210 
211class AssistantFunctionCall(BaseModel):
212    name: str
213    arguments: dict[str, Any]
214 
215class ChatModelResponse(BaseModel):
216    completion_text: str
217    function_calls: list[AssistantFunctionCall]
218```
219 
220## File Storage (`file_storage/`)
221 
222Abstract interface for file operations:
223 
224```python
225class FileStorage(ABC):
226    def open_file(self, path, mode="r", binary=False) -> IO
227    def read_file(self, path, binary=False) -> str | bytes
228    async def write_file(self, path, content) -> None
229    def list_files(self, path=".") -> list[Path]
230    def list_folders(self, path=".", recursive=False) -> list[Path]
231    def delete_file(self, path) -> None
232    def exists(self, path) -> bool
233    def clone_with_subroot(self, subroot) -> FileStorage
234```
235 
236**Implementations:** `LocalFileStorage`, `S3FileStorage`, `GCSFileStorage`
237 
238## Command System (`command/`)
239 
240### @command Decorator
241 
242```python
243@command(
244    names=["greet", "hello"],
245    description="Greet a user",
246    parameters={
247        "name": JSONSchema(type=JSONSchema.Type.STRING, required=True),
248        "greeting": JSONSchema(type=JSONSchema.Type.STRING, required=False),
249    },
250)
251def greet(self, name: str, greeting: str = "Hello") -> str:
252    return f"{greeting}, {name}!"
253```
254 
255### Providing Commands
256 
257```python
258class MyComponent(CommandProvider):
259    def get_commands(self) -> Iterator[Command]:
260        yield self.greet  # Decorated method becomes Command
261```
262 
263## Built-in Components
264 
265| Component | Protocols | Purpose |
266|-----------|-----------|---------|
267| `SystemComponent` | DirectiveProvider, MessageProvider, CommandProvider | Core directives, `finish` command |
268| `FileManagerComponent` | DirectiveProvider, CommandProvider | read/write/list files |
269| `CodeExecutorComponent` | CommandProvider | Python & shell execution (Docker) |
270| `WebSearchComponent` | DirectiveProvider, CommandProvider | DuckDuckGo & Google search |
271| `WebPlaywrightComponent` | DirectiveProvider, CommandProvider | Browser automation (Playwright) |
272| `ActionHistoryComponent` | MessageProvider, AfterParse, AfterExecute | Track & summarize history |
273| `WatchdogComponent` | AfterParse | Loop detection, LLM switching |
274| `ContextComponent` | MessageProvider, CommandProvider | Keep files in prompt context |
275| `ImageGeneratorComponent` | CommandProvider | DALL-E, Stable Diffusion |
276| `GitOperationsComponent` | CommandProvider | Git commands |
277| `UserInteractionComponent` | CommandProvider | `ask_user` command |
278 
279## Configuration
280 
281### BaseAgentSettings
282 
283```python
284class BaseAgentSettings(SystemSettings):
285    agent_id: str
286    ai_profile: AIProfile          # name, role, goals
287    directives: AIDirectives       # constraints, resources, best_practices
288    task: str
289    config: BaseAgentConfiguration
290```
291 
292### UserConfigurable Fields
293 
294```python
295class MyConfig(SystemConfiguration):
296    api_key: SecretStr = UserConfigurable(from_env="API_KEY", exclude=True)
297    max_retries: int = UserConfigurable(default=3, from_env="MAX_RETRIES")
298 
299config = MyConfig.from_env()  # Load from environment
300```
301 
302## Agent Protocol (`agent_protocol/`)
303 
304REST API for task-based interaction:
305 
306```
307POST /ap/v1/agent/tasks              # Create task
308GET  /ap/v1/agent/tasks              # List tasks
309GET  /ap/v1/agent/tasks/{id}         # Get task
310POST /ap/v1/agent/tasks/{id}/steps   # Execute step
311GET  /ap/v1/agent/tasks/{id}/steps   # List steps
312GET  /ap/v1/agent/tasks/{id}/artifacts  # List artifacts
313```
314 
315**ProtocolAgent mixin** provides these endpoints + database persistence.
316 
317## Testing
318 
319**Fixtures** (`conftest.py`):
320- `storage` - Temporary LocalFileStorage
321 
322Run from the `classic/` directory:
323```bash
324poetry run pytest forge/tests/                    # All forge tests
325poetry run pytest forge/tests/ --cov=forge        # With coverage
326```
327 
328**Note**: Tests requiring API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY) will be skipped if not set.
329 
330## Creating a Custom Component
331 
332```python
333from forge.agent.components import AgentComponent, ConfigurableComponent
334from forge.agent.protocols import CommandProvider
335from forge.command import command
336from forge.models.json_schema import JSONSchema
337 
338class MyConfig(BaseModel):
339    setting: str = "default"
340 
341class MyComponent(CommandProvider, ConfigurableComponent[MyConfig]):
342    config_class = MyConfig
343 
344    def get_commands(self) -> Iterator[Command]:
345        yield self.my_command
346 
347    @command(
348        names=["mycmd"],
349        description="Do something",
350        parameters={"arg": JSONSchema(type=JSONSchema.Type.STRING, required=True)},
351    )
352    def my_command(self, arg: str) -> str:
353        return f"Result: {arg}"
354```
355 
356## Creating a Custom Agent
357 
358```python
359from forge.agent.forge_agent import ForgeAgent
360 
361class MyAgent(ForgeAgent):
362    def __init__(self, database, workspace):
363        super().__init__(database, workspace)
364        self.my_component = MyComponent()
365 
366    async def propose_action(self) -> ActionProposal:
367        # 1. Collect directives
368        constraints = await self.run_pipeline(DirectiveProvider.get_constraints)
369        resources = await self.run_pipeline(DirectiveProvider.get_resources)
370 
371        # 2. Collect commands
372        commands = await self.run_pipeline(CommandProvider.get_commands)
373 
374        # 3. Collect messages
375        messages = await self.run_pipeline(MessageProvider.get_messages)
376 
377        # 4. Build prompt and call LLM
378        response = await self.llm_provider.create_chat_completion(
379            model_prompt=messages,
380            model_name=self.config.smart_llm,
381            functions=function_specs_from_commands(commands),
382        )
383 
384        # 5. Parse and return proposal
385        return ActionProposal(
386            thoughts=response.completion_text,
387            use_tool=response.function_calls[0],
388            raw_message=AssistantChatMessage(content=response.completion_text),
389        )
390```
391 
392## Key Patterns
393 
394### Component Ordering
395```python
396self.component_a = ComponentA()
397self.component_b = ComponentB().run_after(self.component_a)
398```
399 
400### Conditional Enabling
401```python
402self.search = WebSearchComponent()
403self.search._enabled = bool(os.getenv("GOOGLE_API_KEY"))
404self.search._disabled_reason = "No Google API key"
405```
406 
407### Pipeline Retry Logic
408- `ComponentEndpointError` → retry same component (3x)
409- `EndpointPipelineError` → restart all components (3x)
410- `ComponentSystemError` → restart all pipelines
411 
412## Key Files Reference
413 
414| Purpose | Location |
415|---------|----------|
416| Entry point | `__main__.py` |
417| FastAPI app | `app.py` |
418| Base agent | `agent/base.py` |
419| Reference agent | `agent/forge_agent.py` |
420| Components base | `agent/components.py` |
421| Protocols | `agent/protocols.py` |
422| LLM providers | `llm/providers/` |
423| File storage | `file_storage/` |
424| Commands | `command/` |
425| Built-in components | `components/` |
426| Agent Protocol | `agent_protocol/` |
427