Two files, one repository
bytedance/deer-flow ships 2 formats across 24 indexed files. The question worth asking is whether the second one says anything the first does not.
| Dimension | Shared | Only in A | Only in B | Overlap |
|---|---|---|---|---|
| Sections | 0 | 14 | 1 | 0% |
| Commands | 0 | 26 | 1 | 0% |
| Section tags | 0 | 5 | 3 | 0% |
What each file covers
Sections
0 shared · 14 only in A · 1 only in B- − AGENTS.md
- − What is DeerFlow
- − Service Topology
- − Repository Map
- − Commands: Root vs. Module
- − Backend (see backend/AGENTS.md for the full set)
- − Frontend (see frontend/AGENTS.md for the full set)
- − Prerequisites before `make dev`
- − Run a single test
- − Backend (pytest); run one file or one test function
- − Frontend (rstest)
- − Logs
- − Where to Go Next
- − Cross-Cutting Conventions
- + Subagent System (`packages/harness/deerflow/subagents/`)
Commands
0 shared · 26 only in A · 1 only in B- − make setup
- − make doctor
- − make support-bundle
- − make config
- − make check
- − make install
- − make extension-install SOURCE=...
- − make extension-list
- − make extension-enable NAME=...
- − make extension-disable NAME=...
- − make extension-remove NAME=...
- − make dev
- − make start
- − make stop
- − make up / down
- − make docker-start / docker-stop / docker-logs
- − make extension-*
- − make
- − make help
- − pnpm
- − pnpm.cmd
- − make docker-logs
- − docker compose -f docker/... logs -f <svc>
- − make format
- − pnpm check
- − ruff format --check
- + task()
Section tags
0 shared · 5 only in A · 3 only in B- − setup
- − test
- − lint-format
- − code-style
- − do-not
- + architecture
- + dependencies
- + monorepo
Line diff
bytedance/deer-flow · AGENTS.md
@@ −1 @@
1# AGENTS.md
2
3This file provides guidance to AI coding agents (Claude Code, Codex, and others) when working with code in this repository. It is the source of truth; the sibling `CLAUDE.md` imports it via `@AGENTS.md`.
4
5It is the **monorepo orientation layer**: it maps the whole repo and points to the
6module guides that own the depth. For anything inside a module, read that module's
7guide rather than expecting full detail here:
8
9- **[backend/AGENTS.md](backend/AGENTS.md)** — backend depth: harness/app split, agent &
10 middleware chain, sandbox, MCP, skills, memory, IM channels, persistence/migrations,
11 config system, test layout.
12- **[frontend/AGENTS.md](frontend/AGENTS.md)** — frontend depth: Next.js App Router layout,
13 thread/streaming data flow, code style, commands.
14
15## What is DeerFlow
16
17DeerFlow is a LangGraph-based AI super-agent system with a full-stack architecture. The
18backend runs a "super agent" with sandboxed execution, persistent memory, subagent
19delegation, and extensible tools (built-in, MCP, community), all per-thread isolated. The
20frontend is a Next.js chat UI. External IM platforms (Feishu, Slack, Telegram, Discord,
21DingTalk) bridge into the same agent through the Gateway.
22
23## Service Topology
24
25A single `make dev` / Docker stack runs four cooperating services:
26
27| Service | Port | Role |
28| --------------- | ------ | ------------------------------------------------------------------- |
29| **Nginx** | `2026` | Unified reverse-proxy entry point — open this in the browser |
30| **Gateway API** | `8001` | FastAPI REST API + embedded LangGraph-compatible agent runtime |
31| **Frontend** | `3000` | Next.js web interface |
32| **Provisioner** | `8002` | Optional — only when sandbox is configured for provisioner/K8s mode |
33
34Nginx is the single public entry: it serves the frontend and proxies `/api/langgraph/*`
35to the Gateway's LangGraph runtime, rewriting it to Gateway's native `/api/*` routes; all
36other `/api/*` go straight to the Gateway REST routers. See
37[backend/AGENTS.md](backend/AGENTS.md) for the runtime and router detail.
38It compresses HTML and configured textual assets, while deliberately leaving SSE,
39fonts, images, audio, and video uncompressed at the proxy layer.
40
41Both compose files publish that entry as `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"`
42— **loopback by default**, matching the README's documented deployment model. A bare
43`"${PORT}:2026"` binds `0.0.0.0`, which does not.
44The root `PORT` value is Docker ingress configuration only; local orchestration pins
45Next.js to `3000` so loading `.env` cannot make `make dev` wait on the wrong port.
46Nginx itself listens `default_server` on IPv4+IPv6 and the
47Gateway binds `0.0.0.0:8001` inside the container on purpose — both are container-
48internal; the published nginx port is the entire external surface, and the Gateway's
49`8001` is deliberately not published. Any new published port needs an explicit bind
50address; `backend/tests/test_compose_default_bind_host.py` pins this for every service
51in both compose files.
52
53## Repository Map
54
55```
56deer-flow/
57├── Makefile # Root orchestration: drives the full stack (dev/start/stop, docker, setup)
58├── config.example.yaml # Template → copy to config.yaml (gitignored) at repo root
59├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
60├── backend/ # Python backend — see backend/AGENTS.md
61│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
62│ ├── extensions/sources/ # Deployable snapshots of locally installed Python extensions
63│ ├── packages/extension-api/ # deerflow-extension-api package (import: deerflow_extension_api.*) — public extension contract
64│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
65│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
66├── frontend/ # Next.js frontend (pnpm) — see frontend/AGENTS.md
67├── docker/ # docker-compose files, nginx config, provisioner
68├── skills/ # Agent skills: public/ (committed), custom/ (gitignored)
69│ # Managed integration skill packs are global at .deer-flow/integrations/skills/{provider}/
70│ # Integration credentials and enabled state remain per-user
71├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
72├── examples/deerflow-extension-example/ # Standalone package demonstrating all extension contribution kinds
73├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
74├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
75└── docs/ # Cross-cutting docs, plans, and design notes
76```
77
78Third-party extensions are loaded from a top-level `plugins:` list in `config.yaml`
79(operator-controlled on purpose — that list causes code to be imported, so it is deliberately
80kept out of the API-writable `extensions_config.json`). Packaged extensions can contribute
81middleware, task lifecycle, system-model observers, Gateway services, and FastAPI HTTP
82routers; the [reference extension](examples/deerflow-extension-example/) demonstrates all
83five. Manage them with `deerflow extensions install/list/enable/disable/remove` or the root
84`make extension-*` wrappers. Every mutation requires a Gateway restart, and both build
85hooks and extension code execute with Gateway privileges, so only trusted operator sources
86belong in this path. The manager transaction, accepted source forms, lock discipline, and
87contribution contract live in
88[the extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md).
89
90Runtime config lives at the **repo root**: copy `config.example.yaml` → `config.yaml`
91(main app config) and `extensions_config.example.json` → `extensions_config.json` (MCP
92servers + skills). Both real files are gitignored and may be edited at runtime via the
93Gateway API. Config schema and resolution order are documented in
94[backend/AGENTS.md](backend/AGENTS.md).
95
96Skill quality review note:
97- `skills/public/skill-reviewer/` is the built-in read-only skill quality reviewer.
98 It uses the harness-layer `review_skill_package` tool and contracts in
99 `contracts/skill_review/`. Model-visible review data is compact and
100 tag-neutralized; full raw payloads stay in tool artifacts. See
101 [backend/AGENTS.md](backend/AGENTS.md) for the non-activation, SkillScan, and
102 `skill-creator` ownership boundaries.
103
104Scheduled-task note:
105- The scheduled-task MVP adds a workspace page at `/workspace/scheduled-tasks` plus a background scheduler service gated by `config.yaml -> scheduler.enabled`.
106- Scheduled background runs are intentionally non-interactive: they execute through the normal run lifecycle, but the lead-agent toolset excludes `ask_clarification` when `context.non_interactive=true`. The key is honored only for internally-authenticated callers (the scheduler launch path); client-supplied `context.non_interactive` is dropped.
107
108## Commands: Root vs. Module
109
110**Root `make` targets drive the whole stack** (run from the repo root):
111
112```bash
113make setup # Interactive setup wizard (recommended for new users)
114make doctor # Check configuration and system requirements
115make support-bundle # Generate redacted troubleshooting summary, AI issue draft, and optional zip
116make config # Generate local config files from the examples
117make check # Check that required tools are installed
118make install # Install all dependencies (frontend + backend + pre-commit hooks)
119make extension-install SOURCE=... # Install and enable a trusted Python extension
120make extension-list # List configured Python extensions
121make extension-enable NAME=... # Enable an installed extension (restart required)
122make extension-disable NAME=... # Disable without uninstalling (restart required)
123make extension-remove NAME=... # Remove package and config entry (restart required)
124make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
125make start # Start all services in production mode (local, optimized)
126make stop # Stop all running services
127make up / down # Build/stop the production Docker stack (browser at localhost:2026)
128make docker-start / docker-stop / docker-logs # Docker development environment
129```
130
131Production startup uses the image's pre-built Python environment with `uv run
132--no-sync`, gives the Gateway a real `/health` probe, and makes `make up` wait
133for that probe before printing its success banner. A readiness failure must
134surface Compose status and recent Gateway logs instead of claiming the stack is
135running.
136
137Docker log and restart commands resolve `DEER_FLOW_ROOT` from the current
138checkout before invoking Compose, matching the start and stop commands.
139
140Run `make help` for the full list.
141
142**Per-module commands drive a single module** (run inside that module):
143
144```bash
145# Backend (see backend/AGENTS.md for the full set)
146cd backend && make dev # Gateway API with reload (port 8001)
147cd backend && make test # Backend test suite
148cd backend && make lint # ruff check
149cd backend && make format # ruff format
150
151# Frontend (see frontend/AGENTS.md for the full set)
152cd frontend && pnpm dev # Dev server with Turbopack (port 3000)
153cd frontend && pnpm check # Lint + type check (run before committing)
154cd frontend && pnpm test # Unit tests
155```
156
157Rule of thumb: **root `make` = the full application**; **`backend/Makefile` and `frontend/`
158(`pnpm`) = per-module work.**
159
160Host-side pnpm consumers, including the root/frontend Makefiles and local diagnostic scripts, must run through `scripts/pnpm.py`. Diagnostic scripts resolve the runner and frontend directory to absolute paths before changing the child process working directory, so they remain independent of the caller's current directory. The runner preserves direct `pnpm`/`pnpm.cmd` priority, falls back to `corepack pnpm`, and is invoked from `frontend/` so Corepack honors the package-manager version pinned by that project.
161
162### Prerequisites before `make dev`
163
164`make dev` does **not** generate config files. First-time setup order:
165
166```bash
167make config # copy config.example.yaml -> config.yaml and extensions_config.example.json -> extensions_config.json (both gitignored)
168make install # install frontend + backend deps and pre-commit hooks
169make dev # then start everything
170```
171
172Without `config.yaml` present, services fail to boot. `config.yaml` / `extensions_config.json`
173may be edited at runtime via the Gateway API but are gitignored, so never commit them.
174
175### Run a single test
176
177```bash
178# Backend (pytest); run one file or one test function
179cd backend && python -m pytest tests/test_compose_default_bind_host.py -q
180cd backend && python -m pytest tests/path/to/test.py::test_func -q
181
182# Frontend (rstest)
183cd frontend && pnpm rstest run <pattern> # e.g. pnpm rstest run my-component
184```
185
186### Logs
187
188- Docker stack: `make docker-logs` (or `docker compose -f docker/... logs -f <svc>`).
189- Local `make dev`: each service logs to its own terminal pane. Frontend Turbopack
190 errors surface in the browser console at `localhost:3000`; backend tracebacks appear
191 in the Gateway terminal.
192
193## Where to Go Next
194
195- Backend work → **[backend/AGENTS.md](backend/AGENTS.md)**
196- Frontend work → **[frontend/AGENTS.md](frontend/AGENTS.md)**
197- Setup & install → **[Install.md](Install.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**
198- Project overview & usage → **[README.md](README.md)** (translations: `README_zh.md`,
199 `README_ja.md`, `README_fr.md`, `README_ru.md`)
200- Security policy → **[SECURITY.md](SECURITY.md)**
201- Changes → **[CHANGELOG.md](CHANGELOG.md)**
202- Cutting a release → **[RELEASING.md](RELEASING.md)**
203
204## Cross-Cutting Conventions
205
206These apply repo-wide; module guides own the module-specific detail.
207
208- **Documentation update policy** — keep docs in sync with code: update `README.md` for
209 user-facing changes and the relevant `AGENTS.md` for development/architecture changes in
210 the same change set.
211- **Test-driven development** — features and bug fixes ship with tests. Backend tests live
212 in `backend/tests/` (TDD is mandatory there; see [backend/AGENTS.md](backend/AGENTS.md));
213 frontend tests live in `frontend/tests/`.
214- **Format before pushing** — run `make format` (backend) / `pnpm check` (frontend). Backend
215 CI enforces `ruff format --check`, so formatting must be clean before a push.
216- **Version sources must stay in lockstep** — a release version must match identically in
217 `backend/pyproject.toml`, `frontend/package.json`, and `deploy/helm/deer-flow/Chart.yaml`
218 (`version` + `appVersion`). Pushing a `v*` git tag triggers CI that runs
219 `scripts/verify_versions.sh` and **blocks all publishing** if any source drifts. Before
220 bumping a version, run `scripts/bump_version.sh <ver>` (aligns all four at once) and
221 `scripts/verify_versions.sh <ver>` to catch drift early. See [RELEASING.md](RELEASING.md).
222- **Don't edit `CLAUDE.md`** — it only contains `@AGENTS.md`. All agent guidance changes
223 belong here in `AGENTS.md`; `CLAUDE.md` is a thin import shim.
224
bytedance/deer-flow · backend/packages/harness/deerflow/subagents/AGENTS.md
@@ +1 @@
1### Subagent System (`packages/harness/deerflow/subagents/`)
2
3**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
4**Benefit-based routing policy**: Enabling subagents exposes delegation as an optimization, not a default response to complexity. The lead prompt defaults to direct execution and permits `task` only when parallel latency, specialist capability, or context-isolation benefit clearly exceeds startup, duplicate-discovery, synthesis, state-conflict, and side-effect costs. Inter-agent output dependencies and overlapping mutable state are hard vetoes for parallel dispatch, while duplicate discovery and a cheap direct path remain costs rather than categorical vetoes; a bounded sequential chain may run in one subagent when specialist or context-isolation benefit clearly wins. Parallel scopes must be independent and non-overlapping, the lead uses the fewest useful subagents, and every later batch is re-evaluated while retaining any within-batch parallel benefit. When the enforced per-response limit is 1, the rendered prompt removes parallel and multi-batch benefit guidance and permits delegation only for material specialist or context-isolation benefit. Keep this policy aligned across `lead_agent/prompt.py`, the `task` tool description, and both built-in role descriptions; routing regressions are pinned in `tests/test_subagent_routing_prompt.py`, `tests/test_subagent_prompt_security.py`, and `tests/test_lead_agent_prompt.py`.
5**User-scoped Skills**: Subagents resolve their configured skills through `get_or_new_user_skill_storage(user_id)` using the parent runtime identity, with `DEFAULT_USER_ID` only when no identity is available. This keeps custom-skill shadowing and visibility aligned with the lead agent instead of reading the global-only catalog.
6**Date context (#4781)**: Every built-in subagent execution registers `SubagentDateContextMiddleware` immediately before `SystemMessageCoalescingMiddleware`. Its one-time `before_agent` hook adds a hidden framework-owned `SystemMessage` containing only `<current_date>` before the first model call; it does not read `AppConfig.memory`, call the memory manager, rewrite the task `HumanMessage`, or inherit the lead agent's frozen-conversation/midnight lifecycle. The coalescer merges that reminder with the subagent's static prompt so strict providers still receive exactly one leading `SystemMessage`. The lead-only `DynamicContextMiddleware` registration and its date, optional-memory, and midnight-update behavior remain unchanged.
7**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
8**Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 1-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box)
9**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result. `task_started` carries the resolved effective model name. The per-subagent `SubagentTokenCollector` publishes a cumulative usage snapshot to the shared `SubagentResult` after every completed LLM response; the next `task_running` event carries that snapshot, so collapsed workspace cards can update without re-accounting parent-run totals. Terminal ToolMessage metadata (`subagent_model_name`, `subagent_token_usage`) and the persisted `subagent.end` event retain the model/usage after reload; absent provider usage stays absent rather than being estimated as zero.
10**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
11**Handled LLM failures**: `LLMErrorHandlingMiddleware` deliberately converts provider/model exceptions into an `AIMessage` so the graph can end cleanly, stamping `additional_kwargs.deerflow_error_fallback=true` plus error metadata. Clean graph termination does not imply subagent success: `SubagentExecutor` inspects the last assistant message at terminalization and maps a marked fallback to `SubagentStatus.FAILED`, which then emits `task_failed` and the existing structured `subagent_error`. Only the marker is authoritative — error-looking assistant prose without it remains a normal completed result, so neither the executor nor frontend parses display text as a status protocol.
12**Guardrail caps & `stop_reason` (#3875 Phase 2)**: three independent axes can end a subagent run early, and all now surface *why* through one additive field rather than a new status enum. **Turn axis**: `recursion_limit` on the subagent `run_config` equals `max_turns`, so exhausting the turn budget raises `GraphRecursionError` from `agent.astream`; `executor.py::_aexecute` catches it specifically (before the generic `except Exception`). **Token axis**: `TokenBudgetMiddleware` is attached per-agent via `build_subagent_runtime_middlewares` from `subagents.token_budget` (default `max_tokens` **coupled to `summarization.enabled`** — 1,000,000 when subagent summarization is on, 2,000,000 when off, warn at 0.7, hard-stop at 1.0; a user-set budget always wins regardless of the switch — #3875 Phase 3; a backstop against a subagent that burns tokens on trivial work). It does *not* raise: at the hard-stop threshold it strips the in-flight turn's tool calls, forces `finish_reason="stop"`, and lets the run complete naturally with a final answer. **Loop axis**: `LoopDetectionMiddleware` (attached at the same point) catches repeated identical tool-call sets — or one tool *type* called many times with varying args — and its hard-stop likewise strips `tool_calls` and forces a final answer without raising, recording `loop_capped`. Each guard exposes its cap on a per-`run_id` `consume_stop_reason(run_id)` accessor; `_aexecute` collects **every** middleware with that method (duck-typed via `hasattr`, so the executor has no import coupling to the guard classes) and surfaces the first non-`None` reason — adding a future guard needs no executor change. **Surfacing**: whichever axis fired, `_aexecute` stamps a normal status plus an additive reason — `completed` + `stop_reason=token_capped|turn_capped|loop_capped` when a usable final answer (or partial recovered from the last streamed chunk via `_extract_final_result` → `utils/messages.py::message_content_to_text`, returning a `"No response Generated"` sentinel when no text survived) was produced; `failed` + `stop_reason=turn_capped` when nothing usable survived. `SubagentResult.stop_reason` flows through `task_tool.py::_task_result_command` → `format_subagent_result_message` (renders `Task Succeeded (capped: ...)` / `Task failed (capped: ...)`) and `make_subagent_additional_kwargs`, which stamps the additive `subagent_stop_reason` key alongside the normal `subagent_status`. **Why additive, not an enum**: a new status value would break v1 consumers; an optional field is ignored by older frontends and ledger readers, so the cross-language contract (`contracts/subagent_status_contract.json` v2 + `subagents/status_contract.py` + `frontend/.../subtask-result.ts`, pinned by `test_status_values_match_contract` / `test_stop_reason_values_match_contract`) stays backward-compatible. The durable delegation ledger captures `stop_reason` onto the entry and renders model-facing guidance ("hit a guardrail cap with a partial result; reuse it, retry tighter, or raise the per-agent budget (`max_turns` / `token_budget`)") so the lead reuses a capped completion knowingly instead of mistaking it for a clean one. (Phase 1 shipped this surfacing as a `MAX_TURNS_REACHED` status enum in #3949; Phase 2 replaced that enum with the additive `stop_reason` field per the agreed design — the `max_turns_reached` status value and `SubagentStatus.MAX_TURNS_REACHED` are gone.)
13**Context compaction (#3875 Phase 3, #4039)**: subagents inherit `DeerFlowSummarizationMiddleware` via `build_subagent_runtime_middlewares`, gated on the **same** `summarization.enabled` switch the lead reads (one config covers both chains; trigger/keep/model/prompt come from the shared `summarization` config so they cannot drift). The subagent builder attaches `DurableContextMiddleware` immediately before summarization, using the same skills path/read-tool settings as the lead chain. Compaction stores the generated summary in `ThreadState.summary_text` rather than as a `messages` item; the durable-context wrapper therefore projects it into the next model request as guarded hidden human data. This is required when a message-count keep policy preserves only an assistant tool-call plus its tool results: without the injected summary the next request begins with assistant/tool history and strict OpenAI-compatible providers can reject it. Because `DurableContextMiddleware` inserts a second `SystemMessage(authority_contract)` after the subagent's leading system prompt, the builder also appends `SystemMessageCoalescingMiddleware` innermost (mirroring the lead chain, appended after the optional summarization middleware so it is unconditionally last) to merge every `SystemMessage` into one leading `system_message` — otherwise the durable fix would trade #4039's assistant-first HTTP 400 for a duplicate-system 400 on the same strict backends (#4040). The factory is called with `skip_memory_flush=True` on the subagent path: the lead's `memory_flush_hook` (attached when `memory.enabled`) flushes pre-compaction messages into durable memory keyed by `thread_id`, and subagents share the parent's `thread_id`, so without skipping the hook a subagent's internal turns would pollute the **parent** thread's durable memory. Placement differs from the lead chain (lead appends summarization *before* the guard trio; subagent appends it *after*) — benign because the middleware implements only `before_model` (compaction) with no `after_model`/`consume_stop_reason`, so it cannot disturb the Phase 2 guard-cap stop-reason channel. Compaction rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, which shrinks `len(messages)` below the step-capture cursor mid-run; `capture_new_step_messages` (see Step capture below) resets the cursor to the new tail on contraction so steps appended after the compaction point are not silently dropped.
14**Step capture & persistence (#3779)**: `executor.py` captures both assistant turns (`AIMessage`) **and** tool outputs (`ToolMessage`) via `subagents/step_events.py::capture_new_step_messages`, which walks the *newly-appended tail* of each `stream_mode="values"` chunk (not just `messages[-1]`) so a multi-tool-call turn — where LangGraph's `ToolNode` appends several `ToolMessage`s in one super-step — keeps every tool output instead of dropping all but the last. `runtime/runs/worker.py::_SubagentEventBuffer` additionally persists these `task_*` custom events to the `RunEventStore` as `subagent.start`/`subagent.step`/`subagent.end` (`category="subagent"`, `task_id` in `metadata`). It **batches** writes via `put_batch` (flushing on a terminal `subagent.end`, at `FLUSH_THRESHOLD` events, and in the worker's `finally`) rather than one `put()` per step, since `put()` is a documented low-frequency path (per-thread advisory lock per call) and a deep subagent (`max_turns=150`) emits hundreds of steps on the hot stream loop. `subagent_run_event` rejects malformed chunks that lack a non-empty `task_id`; running chunks additionally require a non-negative integer `message_index` and a message object, so persisted records always satisfy the required lifecycle envelope. `build_subagent_step` caps both the per-step `text` and each tool call's serialized `args` at `SUBAGENT_STEP_MAX_CHARS` (flagged `truncated` / `args_truncated`) so a large `write_file`/`bash` payload can't produce an unbounded row. The dedicated category keeps them out of `list_messages` (the thread feed) while `list_events` returns them for the frontend's fetch-on-expand backfill. `list_events` accepts `task_id` (filters on `metadata["task_id"]` — SQL-side in `DbRunEventStore` via `event_metadata["task_id"].as_string()`, in-memory in the JSONL/memory stores) plus an `after_seq` forward cursor, so the card pages through one subagent's steps without the run-wide `limit` truncating the tail (no schema migration: the filter rides the existing run-scoped index). `step_events.py` is a pure, unit-tested layer (`build_subagent_step` / `subagent_run_event`). **History contraction (#3875 Phase 3)**: `capture_new_step_messages` assumes append-only growth, but `DeerFlowSummarizationMiddleware` rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, shrinking `len(messages)` below the cursor mid-run. On contraction (`total < processed_count`) the cursor resets to the new tail; `capture_step_message`'s id/content dedup prevents re-emitting pre-compaction steps, so steps appended after the compaction point are still captured instead of being dropped until `total` overtakes the stale cursor.
15**Deferred MCP tools** (if `tool_search.enabled`): `SubagentExecutor._build_initial_state` applies the subagent name allow/deny list and assembly-time authorization before calling the shared `assemble_deferred_tools`, appends the `tool_search` tool, injects the `<available-deferred-tools>` section into the subagent's `SystemMessage`, and threads the setup to `_create_agent`, which attaches `McpRoutingMiddleware` (when PR1 routing metadata matches deferred tools) before `DeferredToolFilterMiddleware` through `build_subagent_runtime_middlewares(...)`. Runtime skill policy is intentionally later and dynamic: `tool_search` may disclose/promote catalog metadata, but `SkillToolPolicyMiddleware` still removes or blocks any promoted business tool omitted by the active skill. Subagents thus withhold full MCP schemas until promotion, same as the lead agent; each task run gets a fresh `ThreadState` so promotion is isolated per run
16**Checkpointer isolation**: Subagent graphs are compiled with `checkpointer=False` to avoid inheriting the parent run's checkpointer, since subagents are one-shot and never resume.
17**Checkpoint lineage / stream isolation**: `_aexecute` deliberately omits checkpoint-coordinate keys (`thread_id`, `checkpoint_ns`, `checkpoint_id`, `checkpoint_map`) from the child `RunnableConfig`. LangGraph must inherit those coordinates from the copied parent ContextVar so the delegated graph retains a non-root subgraph namespace; explicitly re-supplying even the same parent `thread_id` starts a new root lineage on LangGraph 1.2.6+ and can route child AI/tool frames into the parent `messages` stream. DeerFlow business components still receive the parent `thread_id` through `runtime.context`, which is the preferred lookup path for sandbox, middleware, and attribution code. Regression coverage in `tests/test_subagent_executor.py::TestSubagentCheckpointLineage` keeps the invocation-contract assertion active on every supported version and version-gates the production-shaped parent-stream test to LangGraph 1.2.6+, where the leak exists.
18
19**Isolated-loop callback boundary**: sync delegation from an active event loop and `execute_async()` copy the ambient ContextVars into the persistent subagent loop so checkpoint lineage, user identity, tracing context, tags, metadata, and LangGraph's namespaced message-stream handler survive. Before submission, `_copy_isolated_subagent_context()` copies the callback manager/list and removes only handlers marked `deerflow_loop_bound`; `RunJournal` carries that marker because it owns parent-loop tasks and a SQL store/pool. LangGraph merges inherited callbacks with the child run's explicit `SubagentTokenCollector`/tracing callbacks, so letting `RunJournal` cross loops causes duplicate accounting and `Future attached to a different loop` failures, while dropping the whole callback chain silently removes child token frames. Do not replace the boundary with a blank `Context`; the inherited checkpoint namespace and framework stream callback are required by the stream-isolation contract above.
20
@@ −1 +1 @@
1−# AGENTS.md
1+### Subagent System (`packages/harness/deerflow/subagents/`)
22
3−This file provides guidance to AI coding agents (Claude Code, Codex, and others) when working with code in this repository. It is the source of truth; the sibling `CLAUDE.md` imports it via `@AGENTS.md`.
3+**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
4+**Benefit-based routing policy**: Enabling subagents exposes delegation as an optimization, not a default response to complexity. The lead prompt defaults to direct execution and permits `task` only when parallel latency, specialist capability, or context-isolation benefit clearly exceeds startup, duplicate-discovery, synthesis, state-conflict, and side-effect costs. Inter-agent output dependencies and overlapping mutable state are hard vetoes for parallel dispatch, while duplicate discovery and a cheap direct path remain costs rather than categorical vetoes; a bounded sequential chain may run in one subagent when specialist or context-isolation benefit clearly wins. Parallel scopes must be independent and non-overlapping, the lead uses the fewest useful subagents, and every later batch is re-evaluated while retaining any within-batch parallel benefit. When the enforced per-response limit is 1, the rendered prompt removes parallel and multi-batch benefit guidance and permits delegation only for material specialist or context-isolation benefit. Keep this policy aligned across `lead_agent/prompt.py`, the `task` tool description, and both built-in role descriptions; routing regressions are pinned in `tests/test_subagent_routing_prompt.py`, `tests/test_subagent_prompt_security.py`, and `tests/test_lead_agent_prompt.py`.
5+**User-scoped Skills**: Subagents resolve their configured skills through `get_or_new_user_skill_storage(user_id)` using the parent runtime identity, with `DEFAULT_USER_ID` only when no identity is available. This keeps custom-skill shadowing and visibility aligned with the lead agent instead of reading the global-only catalog.
6+**Date context (#4781)**: Every built-in subagent execution registers `SubagentDateContextMiddleware` immediately before `SystemMessageCoalescingMiddleware`. Its one-time `before_agent` hook adds a hidden framework-owned `SystemMessage` containing only `<current_date>` before the first model call; it does not read `AppConfig.memory`, call the memory manager, rewrite the task `HumanMessage`, or inherit the lead agent's frozen-conversation/midnight lifecycle. The coalescer merges that reminder with the subagent's static prompt so strict providers still receive exactly one leading `SystemMessage`. The lead-only `DynamicContextMiddleware` registration and its date, optional-memory, and midnight-update behavior remain unchanged.
7+**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
8+**Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 1-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box)
9+**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result. `task_started` carries the resolved effective model name. The per-subagent `SubagentTokenCollector` publishes a cumulative usage snapshot to the shared `SubagentResult` after every completed LLM response; the next `task_running` event carries that snapshot, so collapsed workspace cards can update without re-accounting parent-run totals. Terminal ToolMessage metadata (`subagent_model_name`, `subagent_token_usage`) and the persisted `subagent.end` event retain the model/usage after reload; absent provider usage stays absent rather than being estimated as zero.
10+**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
11+**Handled LLM failures**: `LLMErrorHandlingMiddleware` deliberately converts provider/model exceptions into an `AIMessage` so the graph can end cleanly, stamping `additional_kwargs.deerflow_error_fallback=true` plus error metadata. Clean graph termination does not imply subagent success: `SubagentExecutor` inspects the last assistant message at terminalization and maps a marked fallback to `SubagentStatus.FAILED`, which then emits `task_failed` and the existing structured `subagent_error`. Only the marker is authoritative — error-looking assistant prose without it remains a normal completed result, so neither the executor nor frontend parses display text as a status protocol.
12+**Guardrail caps & `stop_reason` (#3875 Phase 2)**: three independent axes can end a subagent run early, and all now surface *why* through one additive field rather than a new status enum. **Turn axis**: `recursion_limit` on the subagent `run_config` equals `max_turns`, so exhausting the turn budget raises `GraphRecursionError` from `agent.astream`; `executor.py::_aexecute` catches it specifically (before the generic `except Exception`). **Token axis**: `TokenBudgetMiddleware` is attached per-agent via `build_subagent_runtime_middlewares` from `subagents.token_budget` (default `max_tokens` **coupled to `summarization.enabled`** — 1,000,000 when subagent summarization is on, 2,000,000 when off, warn at 0.7, hard-stop at 1.0; a user-set budget always wins regardless of the switch — #3875 Phase 3; a backstop against a subagent that burns tokens on trivial work). It does *not* raise: at the hard-stop threshold it strips the in-flight turn's tool calls, forces `finish_reason="stop"`, and lets the run complete naturally with a final answer. **Loop axis**: `LoopDetectionMiddleware` (attached at the same point) catches repeated identical tool-call sets — or one tool *type* called many times with varying args — and its hard-stop likewise strips `tool_calls` and forces a final answer without raising, recording `loop_capped`. Each guard exposes its cap on a per-`run_id` `consume_stop_reason(run_id)` accessor; `_aexecute` collects **every** middleware with that method (duck-typed via `hasattr`, so the executor has no import coupling to the guard classes) and surfaces the first non-`None` reason — adding a future guard needs no executor change. **Surfacing**: whichever axis fired, `_aexecute` stamps a normal status plus an additive reason — `completed` + `stop_reason=token_capped|turn_capped|loop_capped` when a usable final answer (or partial recovered from the last streamed chunk via `_extract_final_result` → `utils/messages.py::message_content_to_text`, returning a `"No response Generated"` sentinel when no text survived) was produced; `failed` + `stop_reason=turn_capped` when nothing usable survived. `SubagentResult.stop_reason` flows through `task_tool.py::_task_result_command` → `format_subagent_result_message` (renders `Task Succeeded (capped: ...)` / `Task failed (capped: ...)`) and `make_subagent_additional_kwargs`, which stamps the additive `subagent_stop_reason` key alongside the normal `subagent_status`. **Why additive, not an enum**: a new status value would break v1 consumers; an optional field is ignored by older frontends and ledger readers, so the cross-language contract (`contracts/subagent_status_contract.json` v2 + `subagents/status_contract.py` + `frontend/.../subtask-result.ts`, pinned by `test_status_values_match_contract` / `test_stop_reason_values_match_contract`) stays backward-compatible. The durable delegation ledger captures `stop_reason` onto the entry and renders model-facing guidance ("hit a guardrail cap with a partial result; reuse it, retry tighter, or raise the per-agent budget (`max_turns` / `token_budget`)") so the lead reuses a capped completion knowingly instead of mistaking it for a clean one. (Phase 1 shipped this surfacing as a `MAX_TURNS_REACHED` status enum in #3949; Phase 2 replaced that enum with the additive `stop_reason` field per the agreed design — the `max_turns_reached` status value and `SubagentStatus.MAX_TURNS_REACHED` are gone.)
13+**Context compaction (#3875 Phase 3, #4039)**: subagents inherit `DeerFlowSummarizationMiddleware` via `build_subagent_runtime_middlewares`, gated on the **same** `summarization.enabled` switch the lead reads (one config covers both chains; trigger/keep/model/prompt come from the shared `summarization` config so they cannot drift). The subagent builder attaches `DurableContextMiddleware` immediately before summarization, using the same skills path/read-tool settings as the lead chain. Compaction stores the generated summary in `ThreadState.summary_text` rather than as a `messages` item; the durable-context wrapper therefore projects it into the next model request as guarded hidden human data. This is required when a message-count keep policy preserves only an assistant tool-call plus its tool results: without the injected summary the next request begins with assistant/tool history and strict OpenAI-compatible providers can reject it. Because `DurableContextMiddleware` inserts a second `SystemMessage(authority_contract)` after the subagent's leading system prompt, the builder also appends `SystemMessageCoalescingMiddleware` innermost (mirroring the lead chain, appended after the optional summarization middleware so it is unconditionally last) to merge every `SystemMessage` into one leading `system_message` — otherwise the durable fix would trade #4039's assistant-first HTTP 400 for a duplicate-system 400 on the same strict backends (#4040). The factory is called with `skip_memory_flush=True` on the subagent path: the lead's `memory_flush_hook` (attached when `memory.enabled`) flushes pre-compaction messages into durable memory keyed by `thread_id`, and subagents share the parent's `thread_id`, so without skipping the hook a subagent's internal turns would pollute the **parent** thread's durable memory. Placement differs from the lead chain (lead appends summarization *before* the guard trio; subagent appends it *after*) — benign because the middleware implements only `before_model` (compaction) with no `after_model`/`consume_stop_reason`, so it cannot disturb the Phase 2 guard-cap stop-reason channel. Compaction rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, which shrinks `len(messages)` below the step-capture cursor mid-run; `capture_new_step_messages` (see Step capture below) resets the cursor to the new tail on contraction so steps appended after the compaction point are not silently dropped.
14+**Step capture & persistence (#3779)**: `executor.py` captures both assistant turns (`AIMessage`) **and** tool outputs (`ToolMessage`) via `subagents/step_events.py::capture_new_step_messages`, which walks the *newly-appended tail* of each `stream_mode="values"` chunk (not just `messages[-1]`) so a multi-tool-call turn — where LangGraph's `ToolNode` appends several `ToolMessage`s in one super-step — keeps every tool output instead of dropping all but the last. `runtime/runs/worker.py::_SubagentEventBuffer` additionally persists these `task_*` custom events to the `RunEventStore` as `subagent.start`/`subagent.step`/`subagent.end` (`category="subagent"`, `task_id` in `metadata`). It **batches** writes via `put_batch` (flushing on a terminal `subagent.end`, at `FLUSH_THRESHOLD` events, and in the worker's `finally`) rather than one `put()` per step, since `put()` is a documented low-frequency path (per-thread advisory lock per call) and a deep subagent (`max_turns=150`) emits hundreds of steps on the hot stream loop. `subagent_run_event` rejects malformed chunks that lack a non-empty `task_id`; running chunks additionally require a non-negative integer `message_index` and a message object, so persisted records always satisfy the required lifecycle envelope. `build_subagent_step` caps both the per-step `text` and each tool call's serialized `args` at `SUBAGENT_STEP_MAX_CHARS` (flagged `truncated` / `args_truncated`) so a large `write_file`/`bash` payload can't produce an unbounded row. The dedicated category keeps them out of `list_messages` (the thread feed) while `list_events` returns them for the frontend's fetch-on-expand backfill. `list_events` accepts `task_id` (filters on `metadata["task_id"]` — SQL-side in `DbRunEventStore` via `event_metadata["task_id"].as_string()`, in-memory in the JSONL/memory stores) plus an `after_seq` forward cursor, so the card pages through one subagent's steps without the run-wide `limit` truncating the tail (no schema migration: the filter rides the existing run-scoped index). `step_events.py` is a pure, unit-tested layer (`build_subagent_step` / `subagent_run_event`). **History contraction (#3875 Phase 3)**: `capture_new_step_messages` assumes append-only growth, but `DeerFlowSummarizationMiddleware` rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, shrinking `len(messages)` below the cursor mid-run. On contraction (`total < processed_count`) the cursor resets to the new tail; `capture_step_message`'s id/content dedup prevents re-emitting pre-compaction steps, so steps appended after the compaction point are still captured instead of being dropped until `total` overtakes the stale cursor.
15+**Deferred MCP tools** (if `tool_search.enabled`): `SubagentExecutor._build_initial_state` applies the subagent name allow/deny list and assembly-time authorization before calling the shared `assemble_deferred_tools`, appends the `tool_search` tool, injects the `<available-deferred-tools>` section into the subagent's `SystemMessage`, and threads the setup to `_create_agent`, which attaches `McpRoutingMiddleware` (when PR1 routing metadata matches deferred tools) before `DeferredToolFilterMiddleware` through `build_subagent_runtime_middlewares(...)`. Runtime skill policy is intentionally later and dynamic: `tool_search` may disclose/promote catalog metadata, but `SkillToolPolicyMiddleware` still removes or blocks any promoted business tool omitted by the active skill. Subagents thus withhold full MCP schemas until promotion, same as the lead agent; each task run gets a fresh `ThreadState` so promotion is isolated per run
16+**Checkpointer isolation**: Subagent graphs are compiled with `checkpointer=False` to avoid inheriting the parent run's checkpointer, since subagents are one-shot and never resume.
17+**Checkpoint lineage / stream isolation**: `_aexecute` deliberately omits checkpoint-coordinate keys (`thread_id`, `checkpoint_ns`, `checkpoint_id`, `checkpoint_map`) from the child `RunnableConfig`. LangGraph must inherit those coordinates from the copied parent ContextVar so the delegated graph retains a non-root subgraph namespace; explicitly re-supplying even the same parent `thread_id` starts a new root lineage on LangGraph 1.2.6+ and can route child AI/tool frames into the parent `messages` stream. DeerFlow business components still receive the parent `thread_id` through `runtime.context`, which is the preferred lookup path for sandbox, middleware, and attribution code. Regression coverage in `tests/test_subagent_executor.py::TestSubagentCheckpointLineage` keeps the invocation-contract assertion active on every supported version and version-gates the production-shaped parent-stream test to LangGraph 1.2.6+, where the leak exists.
418
5−It is the **monorepo orientation layer**: it maps the whole repo and points to the
6−module guides that own the depth. For anything inside a module, read that module's
7−guide rather than expecting full detail here:
8−
9−- **[backend/AGENTS.md](backend/AGENTS.md)** — backend depth: harness/app split, agent &
10− middleware chain, sandbox, MCP, skills, memory, IM channels, persistence/migrations,
11− config system, test layout.
12−- **[frontend/AGENTS.md](frontend/AGENTS.md)** — frontend depth: Next.js App Router layout,
13− thread/streaming data flow, code style, commands.
14−
15−## What is DeerFlow
16−
17−DeerFlow is a LangGraph-based AI super-agent system with a full-stack architecture. The
18−backend runs a "super agent" with sandboxed execution, persistent memory, subagent
19−delegation, and extensible tools (built-in, MCP, community), all per-thread isolated. The
20−frontend is a Next.js chat UI. External IM platforms (Feishu, Slack, Telegram, Discord,
21−DingTalk) bridge into the same agent through the Gateway.
22−
23−## Service Topology
24−
25−A single `make dev` / Docker stack runs four cooperating services:
26−
27−| Service | Port | Role |
28−| --------------- | ------ | ------------------------------------------------------------------- |
29−| **Nginx** | `2026` | Unified reverse-proxy entry point — open this in the browser |
30−| **Gateway API** | `8001` | FastAPI REST API + embedded LangGraph-compatible agent runtime |
31−| **Frontend** | `3000` | Next.js web interface |
32−| **Provisioner** | `8002` | Optional — only when sandbox is configured for provisioner/K8s mode |
33−
34−Nginx is the single public entry: it serves the frontend and proxies `/api/langgraph/*`
35−to the Gateway's LangGraph runtime, rewriting it to Gateway's native `/api/*` routes; all
36−other `/api/*` go straight to the Gateway REST routers. See
37−[backend/AGENTS.md](backend/AGENTS.md) for the runtime and router detail.
38−It compresses HTML and configured textual assets, while deliberately leaving SSE,
39−fonts, images, audio, and video uncompressed at the proxy layer.
40−
41−Both compose files publish that entry as `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"`
42−— **loopback by default**, matching the README's documented deployment model. A bare
43−`"${PORT}:2026"` binds `0.0.0.0`, which does not.
44−The root `PORT` value is Docker ingress configuration only; local orchestration pins
45−Next.js to `3000` so loading `.env` cannot make `make dev` wait on the wrong port.
46−Nginx itself listens `default_server` on IPv4+IPv6 and the
47−Gateway binds `0.0.0.0:8001` inside the container on purpose — both are container-
48−internal; the published nginx port is the entire external surface, and the Gateway's
49−`8001` is deliberately not published. Any new published port needs an explicit bind
50−address; `backend/tests/test_compose_default_bind_host.py` pins this for every service
51−in both compose files.
52−
53−## Repository Map
54−
55−```
56−deer-flow/
57−├── Makefile # Root orchestration: drives the full stack (dev/start/stop, docker, setup)
58−├── config.example.yaml # Template → copy to config.yaml (gitignored) at repo root
59−├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
60−├── backend/ # Python backend — see backend/AGENTS.md
61−│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
62−│ ├── extensions/sources/ # Deployable snapshots of locally installed Python extensions
63−│ ├── packages/extension-api/ # deerflow-extension-api package (import: deerflow_extension_api.*) — public extension contract
64−│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
65−│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
66−├── frontend/ # Next.js frontend (pnpm) — see frontend/AGENTS.md
67−├── docker/ # docker-compose files, nginx config, provisioner
68−├── skills/ # Agent skills: public/ (committed), custom/ (gitignored)
69−│ # Managed integration skill packs are global at .deer-flow/integrations/skills/{provider}/
70−│ # Integration credentials and enabled state remain per-user
71−├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
72−├── examples/deerflow-extension-example/ # Standalone package demonstrating all extension contribution kinds
73−├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
74−├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
75−└── docs/ # Cross-cutting docs, plans, and design notes
76−```
77−
78−Third-party extensions are loaded from a top-level `plugins:` list in `config.yaml`
79−(operator-controlled on purpose — that list causes code to be imported, so it is deliberately
80−kept out of the API-writable `extensions_config.json`). Packaged extensions can contribute
81−middleware, task lifecycle, system-model observers, Gateway services, and FastAPI HTTP
82−routers; the [reference extension](examples/deerflow-extension-example/) demonstrates all
83−five. Manage them with `deerflow extensions install/list/enable/disable/remove` or the root
84−`make extension-*` wrappers. Every mutation requires a Gateway restart, and both build
85−hooks and extension code execute with Gateway privileges, so only trusted operator sources
86−belong in this path. The manager transaction, accepted source forms, lock discipline, and
87−contribution contract live in
88−[the extensions guide](backend/packages/harness/deerflow/extensions/AGENTS.md).
89−
90−Runtime config lives at the **repo root**: copy `config.example.yaml` → `config.yaml`
91−(main app config) and `extensions_config.example.json` → `extensions_config.json` (MCP
92−servers + skills). Both real files are gitignored and may be edited at runtime via the
93−Gateway API. Config schema and resolution order are documented in
94−[backend/AGENTS.md](backend/AGENTS.md).
95−
96−Skill quality review note:
97−- `skills/public/skill-reviewer/` is the built-in read-only skill quality reviewer.
98− It uses the harness-layer `review_skill_package` tool and contracts in
99− `contracts/skill_review/`. Model-visible review data is compact and
100− tag-neutralized; full raw payloads stay in tool artifacts. See
101− [backend/AGENTS.md](backend/AGENTS.md) for the non-activation, SkillScan, and
102− `skill-creator` ownership boundaries.
103−
104−Scheduled-task note:
105−- The scheduled-task MVP adds a workspace page at `/workspace/scheduled-tasks` plus a background scheduler service gated by `config.yaml -> scheduler.enabled`.
106−- Scheduled background runs are intentionally non-interactive: they execute through the normal run lifecycle, but the lead-agent toolset excludes `ask_clarification` when `context.non_interactive=true`. The key is honored only for internally-authenticated callers (the scheduler launch path); client-supplied `context.non_interactive` is dropped.
107−
108−## Commands: Root vs. Module
109−
110−**Root `make` targets drive the whole stack** (run from the repo root):
111−
112−```bash
113−make setup # Interactive setup wizard (recommended for new users)
114−make doctor # Check configuration and system requirements
115−make support-bundle # Generate redacted troubleshooting summary, AI issue draft, and optional zip
116−make config # Generate local config files from the examples
117−make check # Check that required tools are installed
118−make install # Install all dependencies (frontend + backend + pre-commit hooks)
119−make extension-install SOURCE=... # Install and enable a trusted Python extension
120−make extension-list # List configured Python extensions
121−make extension-enable NAME=... # Enable an installed extension (restart required)
122−make extension-disable NAME=... # Disable without uninstalling (restart required)
123−make extension-remove NAME=... # Remove package and config entry (restart required)
124−make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
125−make start # Start all services in production mode (local, optimized)
126−make stop # Stop all running services
127−make up / down # Build/stop the production Docker stack (browser at localhost:2026)
128−make docker-start / docker-stop / docker-logs # Docker development environment
129−```
130−
131−Production startup uses the image's pre-built Python environment with `uv run
132−--no-sync`, gives the Gateway a real `/health` probe, and makes `make up` wait
133−for that probe before printing its success banner. A readiness failure must
134−surface Compose status and recent Gateway logs instead of claiming the stack is
135−running.
136−
137−Docker log and restart commands resolve `DEER_FLOW_ROOT` from the current
138−checkout before invoking Compose, matching the start and stop commands.
139−
140−Run `make help` for the full list.
141−
142−**Per-module commands drive a single module** (run inside that module):
143−
144−```bash
145−# Backend (see backend/AGENTS.md for the full set)
146−cd backend && make dev # Gateway API with reload (port 8001)
147−cd backend && make test # Backend test suite
148−cd backend && make lint # ruff check
149−cd backend && make format # ruff format
150−
151−# Frontend (see frontend/AGENTS.md for the full set)
152−cd frontend && pnpm dev # Dev server with Turbopack (port 3000)
153−cd frontend && pnpm check # Lint + type check (run before committing)
154−cd frontend && pnpm test # Unit tests
155−```
156−
157−Rule of thumb: **root `make` = the full application**; **`backend/Makefile` and `frontend/`
158−(`pnpm`) = per-module work.**
159−
160−Host-side pnpm consumers, including the root/frontend Makefiles and local diagnostic scripts, must run through `scripts/pnpm.py`. Diagnostic scripts resolve the runner and frontend directory to absolute paths before changing the child process working directory, so they remain independent of the caller's current directory. The runner preserves direct `pnpm`/`pnpm.cmd` priority, falls back to `corepack pnpm`, and is invoked from `frontend/` so Corepack honors the package-manager version pinned by that project.
161−
162−### Prerequisites before `make dev`
163−
164−`make dev` does **not** generate config files. First-time setup order:
165−
166−```bash
167−make config # copy config.example.yaml -> config.yaml and extensions_config.example.json -> extensions_config.json (both gitignored)
168−make install # install frontend + backend deps and pre-commit hooks
169−make dev # then start everything
170−```
171−
172−Without `config.yaml` present, services fail to boot. `config.yaml` / `extensions_config.json`
173−may be edited at runtime via the Gateway API but are gitignored, so never commit them.
174−
175−### Run a single test
176−
177−```bash
178−# Backend (pytest); run one file or one test function
179−cd backend && python -m pytest tests/test_compose_default_bind_host.py -q
180−cd backend && python -m pytest tests/path/to/test.py::test_func -q
181−
182−# Frontend (rstest)
183−cd frontend && pnpm rstest run <pattern> # e.g. pnpm rstest run my-component
184−```
185−
186−### Logs
187−
188−- Docker stack: `make docker-logs` (or `docker compose -f docker/... logs -f <svc>`).
189−- Local `make dev`: each service logs to its own terminal pane. Frontend Turbopack
190− errors surface in the browser console at `localhost:3000`; backend tracebacks appear
191− in the Gateway terminal.
192−
193−## Where to Go Next
194−
195−- Backend work → **[backend/AGENTS.md](backend/AGENTS.md)**
196−- Frontend work → **[frontend/AGENTS.md](frontend/AGENTS.md)**
197−- Setup & install → **[Install.md](Install.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**
198−- Project overview & usage → **[README.md](README.md)** (translations: `README_zh.md`,
199− `README_ja.md`, `README_fr.md`, `README_ru.md`)
200−- Security policy → **[SECURITY.md](SECURITY.md)**
201−- Changes → **[CHANGELOG.md](CHANGELOG.md)**
202−- Cutting a release → **[RELEASING.md](RELEASING.md)**
203−
204−## Cross-Cutting Conventions
205−
206−These apply repo-wide; module guides own the module-specific detail.
207−
208−- **Documentation update policy** — keep docs in sync with code: update `README.md` for
209− user-facing changes and the relevant `AGENTS.md` for development/architecture changes in
210− the same change set.
211−- **Test-driven development** — features and bug fixes ship with tests. Backend tests live
212− in `backend/tests/` (TDD is mandatory there; see [backend/AGENTS.md](backend/AGENTS.md));
213− frontend tests live in `frontend/tests/`.
214−- **Format before pushing** — run `make format` (backend) / `pnpm check` (frontend). Backend
215− CI enforces `ruff format --check`, so formatting must be clean before a push.
216−- **Version sources must stay in lockstep** — a release version must match identically in
217− `backend/pyproject.toml`, `frontend/package.json`, and `deploy/helm/deer-flow/Chart.yaml`
218− (`version` + `appVersion`). Pushing a `v*` git tag triggers CI that runs
219− `scripts/verify_versions.sh` and **blocks all publishing** if any source drifts. Before
220− bumping a version, run `scripts/bump_version.sh <ver>` (aligns all four at once) and
221− `scripts/verify_versions.sh <ver>` to catch drift early. See [RELEASING.md](RELEASING.md).
222−- **Don't edit `CLAUDE.md`** — it only contains `@AGENTS.md`. All agent guidance changes
223− belong here in `AGENTS.md`; `CLAUDE.md` is a thin import shim.
19+**Isolated-loop callback boundary**: sync delegation from an active event loop and `execute_async()` copy the ambient ContextVars into the persistent subagent loop so checkpoint lineage, user identity, tracing context, tags, metadata, and LangGraph's namespaced message-stream handler survive. Before submission, `_copy_isolated_subagent_context()` copies the callback manager/list and removes only handlers marked `deerflow_loop_bound`; `RunJournal` carries that marker because it owns parent-loop tasks and a SQL store/pool. LangGraph merges inherited callbacks with the child run's explicit `SubagentTokenCollector`/tracing callbacks, so letting `RunJournal` cross loops causes duplicate accounting and `Future attached to a different loop` failures, while dropping the whole callback chain silently removes child token frames. Do not replace the boundary with a blank `Context`; the inherited checkpoint namespace and framework stream callback are required by the stream-isolation contract above.
22420
