AGENTS.md

1# Backend
2 
3This file provides guidance to coding agents when working with the backend.
4 
5## Essential Commands
6 
7To run something with Python package dependencies you MUST use `poetry run ...`.
8 
9```bash
10# Install dependencies
11poetry install
12 
13# Run database migrations
14poetry run prisma migrate dev
15 
16# Start all services (database, redis, rabbitmq, clamav)
17docker compose up -d
18 
19# Run the backend as a whole
20poetry run app
21 
22# Run tests
23poetry run test
24 
25# Run specific test
26poetry run pytest path/to/test_file.py::test_function_name
27 
28# Run block tests (tests that validate all blocks work correctly)
29poetry run pytest backend/blocks/test/test_block.py -xvs
30 
31# Run tests for a specific block (e.g., GetCurrentTimeBlock)
32poetry run pytest 'backend/blocks/test/test_block.py::test_available_blocks[GetCurrentTimeBlock]' -xvs
33 
34# Lint and format
35# prefer format if you want to just "fix" it and only get the errors that can't be autofixed
36poetry run format  # Black + isort
37poetry run lint    # ruff
38```
39 
40More details can be found in @TESTING.md
41 
42### Creating/Updating Snapshots
43 
44When you first write a test or when the expected output changes:
45 
46```bash
47poetry run pytest path/to/test.py --snapshot-update
48```
49 
50⚠️ **Important**: Always review snapshot changes before committing! Use `git diff` to verify the changes are expected.
51 
52## Architecture
53 
54- **API Layer**: FastAPI with REST and WebSocket endpoints
55- **Database**: PostgreSQL with Prisma ORM, includes pgvector for embeddings
56- **Queue System**: RabbitMQ for async task processing
57- **Execution Engine**: Separate executor service processes agent workflows
58- **Authentication**: JWT-based with Supabase integration
59- **Security**: Cache protection middleware prevents sensitive data caching in browsers/proxies
60 
61## Code Style
62 
63- **Top-level imports only** — no local/inner imports (lazy imports only for heavy optional deps like `openpyxl`)
64- **Absolute imports** — use `from backend.module import ...` for cross-package imports. Single-dot relative (`from .sibling import ...`) is acceptable for sibling modules within the same package (e.g., blocks). Avoid double-dot relative imports (`from ..parent import ...`) — use the absolute path instead
65- **No duck typing** — no `hasattr`/`getattr`/`isinstance` for type dispatch; use typed interfaces/unions/protocols
66- **Pydantic models** over dataclass/namedtuple/dict for structured data
67- **No linter suppressors** — no `# type: ignore`, `# noqa`, `# pyright: ignore`; fix the type/code
68- **List comprehensions** over manual loop-and-append
69- **Early return** — guard clauses first, avoid deep nesting
70- **f-strings vs printf syntax in log statements** — Use `%s` for deferred interpolation in `debug` statements, f-strings elsewhere for readability: `logger.debug("Processing %s items", count)`, `logger.info(f"Processing {count} items")`
71- **Sanitize error paths** — `os.path.basename()` in error messages to avoid leaking directory structure
72- **TOCTOU awareness** — avoid check-then-act patterns for file access and credit charging
73- **`Security()` vs `Depends()`** — use `Security()` for auth deps to get proper OpenAPI security spec
74- **Redis pipelines** — `transaction=True` for atomicity on multi-step operations
75- **`max(0, value)` guards** — for computed values that should never be negative
76- **SSE protocol** — `data:` lines for frontend-parsed events (must match Zod schema), `: comment` lines for heartbeats/status
77- **File length** — keep files under ~300 lines; if a file grows beyond this, split by responsibility (e.g. extract helpers, models, or a sub-module into a new file). Never keep appending to a long file.
78- **Function length** — keep functions under ~40 lines; extract named helpers when a function grows longer. Long functions are a sign of mixed concerns, not complexity.
79- **Top-down ordering** — define the main/public function or class first, then the helpers it uses below. A reader should encounter high-level logic before implementation details.
80 
81## Testing Approach
82 
83- Uses pytest with snapshot testing for API responses
84- Test files are colocated with source files (`*_test.py`)
85- Mock at boundaries — mock where the symbol is **used**, not where it's **defined**
86- After refactoring, update mock targets to match new module paths
87- Use `AsyncMock` for async functions (`from unittest.mock import AsyncMock`)
88 
89### Test-Driven Development (TDD)
90 
91When fixing a bug or adding a feature, write the test **before** the implementation:
92 
93```python
94# 1. Write a failing test marked xfail
95@pytest.mark.xfail(reason="Bug #1234: widget crashes on empty input")
96def test_widget_handles_empty_input():
97    result = widget.process("")
98    assert result == Widget.EMPTY_RESULT
99 
100# 2. Run it — confirm it fails (XFAIL)
101# poetry run pytest path/to/test.py::test_widget_handles_empty_input -xvs
102 
103# 3. Implement the fix
104 
105# 4. Remove xfail, run again — confirm it passes
106def test_widget_handles_empty_input():
107    result = widget.process("")
108    assert result == Widget.EMPTY_RESULT
109```
110 
111This catches regressions and proves the fix actually works. **Every bug fix should include a test that would have caught it.**
112 
113## Database Schema
114 
115Key models (defined in `schema.prisma`):
116 
117- `User`: Authentication and profile data
118- `AgentGraph`: Workflow definitions with version control
119- `AgentGraphExecution`: Execution history and results
120- `AgentNode`: Individual nodes in a workflow
121- `StoreListing`: Marketplace listings for sharing agents
122 
123## Environment Configuration
124 
125- **Backend**: `.env.default` (defaults) → `.env` (user overrides)
126 
127## Common Development Tasks
128 
129### Adding a new block
130 
131Follow the comprehensive [Block SDK Guide](@../../docs/platform/block-sdk-guide.md) which covers:
132 
133- Provider configuration with `ProviderBuilder`
134- Block schema definition
135- Authentication (API keys, OAuth, webhooks)
136- Testing and validation
137- File organization
138 
139Quick steps:
140 
1411. Create new file in `backend/blocks/`
1422. Configure provider using `ProviderBuilder` in `_config.py`
1433. Inherit from `Block` base class
1444. Define input/output schemas using `BlockSchema`
1455. Implement async `run` method
1466. Generate unique block ID using `uuid.uuid4()`
1477. Test with `poetry run pytest backend/blocks/test/test_block.py`
148 
149Note: when making many new blocks analyze the interfaces for each of these blocks and picture if they would go well together in a graph-based editor or would they struggle to connect productively?
150ex: do the inputs and outputs tie well together?
151 
152If you get any pushback or hit complex block conditions check the new_blocks guide in the docs.
153 
154#### Handling files in blocks with `store_media_file()`
155 
156When blocks need to work with files (images, videos, documents), use `store_media_file()` from `backend.util.file`. The `return_format` parameter determines what you get back:
157 
158| Format | Use When | Returns |
159|--------|----------|---------|
160| `"for_local_processing"` | Processing with local tools (ffmpeg, MoviePy, PIL) | Local file path (e.g., `"image.png"`) |
161| `"for_external_api"` | Sending content to external APIs (Replicate, OpenAI) | Data URI (e.g., `"data:image/png;base64,..."`) |
162| `"for_block_output"` | Returning output from your block | Smart: `workspace://` in CoPilot, data URI in graphs |
163 
164**Examples:**
165 
166```python
167# INPUT: Need to process file locally with ffmpeg
168local_path = await store_media_file(
169    file=input_data.video,
170    execution_context=execution_context,
171    return_format="for_local_processing",
172)
173# local_path = "video.mp4" - use with Path/ffmpeg/etc
174 
175# INPUT: Need to send to external API like Replicate
176image_b64 = await store_media_file(
177    file=input_data.image,
178    execution_context=execution_context,
179    return_format="for_external_api",
180)
181# image_b64 = "data:image/png;base64,iVBORw0..." - send to API
182 
183# OUTPUT: Returning result from block
184result_url = await store_media_file(
185    file=generated_image_url,
186    execution_context=execution_context,
187    return_format="for_block_output",
188)
189yield "image_url", result_url
190# In CoPilot: result_url = "workspace://abc123"
191# In graphs:  result_url = "data:image/png;base64,..."
192```
193 
194**Key points:**
195 
196- `for_block_output` is the ONLY format that auto-adapts to execution context
197- Always use `for_block_output` for block outputs unless you have a specific reason not to
198- Never hardcode workspace checks - let `for_block_output` handle it
199 
200### Modifying the API
201 
2021. Update route in `backend/api/features/`
2032. Add/update Pydantic models in same directory
2043. Write tests alongside the route file
2054. Run `poetry run test` to verify
206 
207## Workspace & Media Files
208 
209**Read [Workspace & Media Architecture](../../docs/platform/workspace-media-architecture.md) when:**
210- Working on CoPilot file upload/download features
211- Building blocks that handle `MediaFileType` inputs/outputs
212- Modifying `WorkspaceManager` or `store_media_file()`
213- Debugging file persistence or virus scanning issues
214 
215Covers: `WorkspaceManager` (persistent storage with session scoping), `store_media_file()` (media normalization pipeline), and responsibility boundaries for virus scanning and persistence.
216 
217## Security Implementation
218 
219### Cache Protection Middleware
220 
221- Located in `backend/api/middleware/security.py`
222- Default behavior: Disables caching for ALL endpoints with `Cache-Control: no-store, no-cache, must-revalidate, private`
223- Uses an allow list approach - only explicitly permitted paths can be cached
224- Cacheable paths include: static assets (`static/*`, `_next/static/*`), health checks, public store pages, documentation
225- Prevents sensitive data (auth tokens, API keys, user data) from being cached by browsers/proxies
226- To allow caching for a new endpoint, add it to `CACHEABLE_PATHS` in the middleware
227- Applied to both main API server and external API applications
228