AGENTS.md

1# e2e/AGENTS.md
2 
3Follow the root `AGENTS.md` first. This package owns user-level end-to-end smoke tests and Playwright UI automation only.
4 
5For the current coverage posture, recent hardening work, grouped-run status, and known intentional gaps, see [`docs/testing/e2e-coverage/status.md`](../docs/testing/e2e-coverage/status.md). For the invariants new and repaired UI tests must hold under the sharded full pool, see [UI test stability rules](#ui-test-stability-rules) below.
6 
7## Directory layout
8 
9- `specs/`: highest-ROI, long-running core business capability regressions suitable for PR or release gating. Each spec should describe one nearly orthogonal product capability chain, such as main dialog generation, Pet, Orbit, or packaged runtime. Keep this layer small and expand it only when a core capability deserves always-on signal.
10- `tests/`: broader user-level end-to-end coverage and local hotspot checks that intentionally span app/package/resource boundaries. Prefer adding tests here when a repeated or high-risk local capability naturally falls out of a core spec. Do not build a speculative coverage matrix before the core spec needs it.
11- `tests/scripts/`: behavior-contract coverage for root operational scripts whose regressions affect install, CI, or release flows. Keep fixtures hermetic and runnable through e2e Vitest; do not put `*.test.ts` siblings directly under root `scripts/`.
12- `ui/`: flat Playwright UI automation test files only. Keep helpers, resources, and non-Playwright harnesses out of this directory.
13- `resources/`: declarative resources for e2e suites, such as Playwright UI scenario lists.
14- `lib/fake-agents.ts`: shared fake local agent CLI harness used by UI and pure-inspect daemon specs.
15- `lib/timeouts.ts`: CI-scaled timeout constants (`T.short`, `T.medium`, `T.long`, `T.xlong`). Import as `{ T }` from `@/timeouts`. Use these instead of hardcoded millisecond values in UI tests.
16- `lib/tools-dev/`: framework-neutral tools-dev runtime lifecycle. It owns namespace/path construction, port reservation, `tools-dev ... --json` execution, status/log/check reads, URL construction, and start/stop semantics. It must not import Vitest or Playwright.
17- `lib/playwright/suite.ts`: Playwright-only suite assembly. It provides the worker-scoped tools-dev fixture, dynamic `baseURL`, and failure attachments. UI tests import `test`/`expect` from `@/playwright/suite`.
18- `lib/vitest/suite.ts`: Vitest-only suite assembly. It composes the neutral tools-dev runtime with report creation, scratch preservation, and Vitest assertions for non-UI smoke suites.
19- `lib/playwright/mock-factory.ts`: shared Playwright mock helpers. `applyStandardMocks(page)` seeds localStorage and intercepts `/api/agents` and `/api/app-config` with standard daemon/mock-agent fixtures. Use in `beforeEach` for tests that do not need a custom agent or protocol setup.
20- `lib/vitest/`: Vitest-specific atomic helpers only. Helpers describe actions such as mock servers, HTTP calls, and reports; tools-dev lifecycle belongs in `lib/tools-dev/` and is only composed through `lib/vitest/suite.ts`.
21- `lib/vitest/report.ts`: the report boundary. Specs save curated output through `report.save(<relpath>, <blob>)` or `report.json(<relpath>, value)`; release workflows should consume only the final report path, not its internal file layout.
22- `createSmokeSuite(...).with.*`: suite-owned lifecycle composition from `@/vitest/suite`. Prefer this shape for namespace-bound resources such as `suite.with.toolsDev(...)` so specs keep business workflow code in the foreground.
23- Temporary e2e Vitest env/PATH mutations, AMR fake endpoint URLs, and packaged smoke default namespaces belong behind `@/vitest/suite` helpers such as `suite.with.env(...)`, `suite.with.pathEntry(...)`, `suite.amr`, and `resolvePackagedSmokeNamespace(...)`. Do not hand-roll save/restore blocks or fixed localhost ports in individual specs.
24- `lib/playwright/`: Playwright-specific fixtures, resource accessors, route helpers, and UI actions.
25- `scripts/playwright.ts`: Playwright auxiliary subcommands such as artifact cleanup; it must not wrap `playwright test`.
26 
27## Spec and test model
28 
29- Start from `specs/`: define orthogonal long-form core capabilities first, then let supporting `tests/` and `lib/` grow from those chains.
30- `specs/` should read as business/system workflows, for example `dialog/main.spec.ts`, `orbit/run.spec.ts`, or `pet/main.spec.ts`.
31- `tests/` should pin reusable local hotspots, such as `tools-dev/inspect.test.ts`, provider mocks, report lifecycle, artifact file shape, or namespace cleanup.
32- High-confidence infrastructure checks may be added to `tests/` before a full core spec exists, but most tests should be extracted only after a spec proves the local hotspot matters.
33- Treat `tests/` as maintainable support material, not permanent coverage inventory. Merge, split, shrink, or delete tests as product capabilities evolve.
34- Keep new non-UI e2e smoke chains pure inspect by default. Do not use Playwright for these chains; use daemon/web APIs, sidecar IPC, tools-dev/tools-pack inspect, logs, reports, and screenshots when available.
35- External service dependencies must use temporary server-level mocks. Do not rely on real API keys, real provider accounts, or UI-level route patching for core e2e smoke.
36- Every atomic suite must run in an isolated namespace. Successful suites should keep only curated reports and high-value artifacts, then clean process/runtime scratch. Failed suites should preserve runtime scratch, logs, mock requests, screenshots, and report pointers for diagnosis.
37 
38## UI test stability rules
39 
40These invariants complement the coverage posture in
41[`docs/testing/e2e-coverage/status.md`](../docs/testing/e2e-coverage/status.md);
42this section is the source of truth for how a UI test must be written to stay
43green under the sharded full pool.
44 
45The `ui-extended-main` full pool (`workflow_dispatch` with `suite=full`)
46executes every non-visual functional shade in one generically sharded matrix
47(`visual-*.test.ts` is excluded by the config's `testIgnore` and runs in its
48own lane): arbitrary P0/P1/P2 interleavings, contiguous shard slices that
49start mid-file, an isolated tools-dev runtime per Playwright worker
50(`nproc / 2`, so two on the `ui_hot` runner) with
51`OD_PLAYWRIGHT_FULLY_PARALLEL=1`, and slow CI runners. It is the only lane
52that runs the whole non-visual `ui` suite together — every P1/P2 shade, plus
53the P0 cases no merge lane covers: `ci.yml`'s `ui_p0` runs only the files
54listed in a `uiP0Groups` group, and `playwright_critical` only its own
55`@critical` file matrix, so a `[P0]`/`@critical` tag does not enroll a new
56file (the P0 cases in `automations-page.test.ts` and `home-hero-rail.test.ts`,
57for instance, run nowhere but the full pool). Two order hazards then hide from
58narrower runs: within-file interleaving (the tests of one file racing under
59fully-parallel workers) and cross-file carry-over (the worker-scoped tools-dev
60runtime — `suite.ts`, `scope: 'worker'` — retaining daemon/config/project
61state between the files a worker runs in sequence). Merge lanes touch each in
62part — `playwright_critical` runs fully-parallel, `ui_p0`'s single-worker
63multi-file groups accumulate carry-over — but only the full pool exercises the
64whole suite interleaved with mid-file shards, so treat it as the acceptance
65gate. New and repaired UI tests must hold the following invariants.
66 
67The merge-gated `workspace-restoration` group also runs fully-parallel across
68its two worker-isolated tools-dev runtimes. Its cases must remain independent
69within the file as well as across files.
70 
71- **Keep browser witnesses at cross-layer boundaries.** Before adding a UI
72  case, identify which assertions already belong to component or runtime tests
73  and which transition uniquely requires the running browser product. Extend
74  an existing browser workflow when it already owns the same project,
75  conversation, file, or retry setup; do not repeat that full setup only to
76  reassert a lower-layer state invariant. Retain one browser witness for each
77  distinct cross-layer transition, and keep its narrower ownership tests
78  explicit enough that future consolidation does not weaken coverage.
79- **Order independence is the contract.** Each test performs its own complete
80  setup — whatever that file's model requires — and never relies on a
81  predecessor's side effects; any contiguous subset of the suite must pass
82  with the rest absent. There is no universal setup list, and the axis that
83  matters is test-scoped browser mocks versus worker-scoped daemon state: the
84  daemon/data root is shared across a worker's tests (`suite.ts`), while
85  localStorage seeds and `applyStandardMocks` route interception are per-test.
86  Real-daemon specs reset config and create real projects; entry-surface
87  specs route-mock the same endpoints and may also create real daemon
88  projects. Match the file's existing model rather than adding route stubs a
89  core smoke chain forbids. Do not add
90  `test.describe.configure({ mode: 'serial' })`: a serial group is atomic
91  within one shard (it cannot be split across the matrix) and adds
92  skip-after-failure, which floors the pool's wall time. Running a file's
93  standalone halves — `--shard=1/2` and `--shard=2/2` with
94  `OD_PLAYWRIGHT_FULLY_PARALLEL=1` — checks within-file split-independence, but
95  each is a fresh process, so it cannot expose the cross-file, same-worker
96  carry-over above; run the whole `ui` folder (one worker runtime spans the
97  files) to surface that, or the full pool, which additionally stresses it
98  with fully-parallel mid-file shards.
99- **Treat a retry-only pass as a signal, not flake.** CI retries a failed
100  functional test once (the visual config sets `retries: 0`), and the retry
101  runs after the Playwright worker restarts with a fresh tools-dev runtime — so a first attempt that failed on a dirty
102  predecessor state can pass on the clean retry and leave the run green. A
103  test that only ever fails on its first attempt is telling you something: it
104  may be carried-over predecessor state, worker-startup instability, or an
105  in-test async-readiness race (the settle and readiness rules below). Don't
106  wave it through — reproduce it locally (CI keeps only the failed roots' paths
107  and the retry's trace, not the failed attempt's runtime) and fix the
108  actual cause.
109- **Settle async surfaces before interacting.** Late-resolving fetches
110  re-render the home surface and a remount silently resets transient UI
111  state: the projects list can remount the templates reveal container. Arm a
112  best-effort response waiter before the navigation or reload that triggers the
113  fetch (entry-chrome-flows's `gotoEntryHome` waits on `/api/projects`, but
114  swallows its own timeout, so it only narrows the race — and the many other
115  same-named helpers do not arm it at all). The load-bearing half is requiring
116  the observed state to survive a settle window rather than trusting the first
117  observation, as `revealHomeTemplates` does by retrying on reveal regression.
118- **An enabled control is not a ready control.** A precondition that arrives
119  over a stream is a distinct gate from a remount reset. Agents load through an
120  incremental `/api/agents` SSE stream, and a BYOK-OpenCode run checks that
121  agent's availability before it will POST: until the stream publishes,
122  `ProjectView` rejects the submit with a visible unavailable error instead of
123  creating a run — even though `chat-send` already reports enabled and the fake
124  runtime env makes availability possible. This pre-POST check is
125  BYOK-OpenCode-specific, not a universal composer invariant, but the pattern
126  generalizes: when a control's readiness depends on a streamed precondition,
127  wait for that signal — ideally a side-effect-free one — rather than for
128  `toBeEnabled`. The BYOK spec in `ui/real-daemon-run.test.ts` instead retries
129  the submit itself until the stream catches up (its `sendPrompt` reports
130  whether a create-run request was issued at all); that is safe only because
131  its oracle tolerates the repeated rejected attempts, and a submit that
132  persisted state on each try would need an idempotent oracle or a
133  non-mutating readiness wait.
134- **Never force-click into gated containers.** An `inert` container swallows
135  force-clicks with no error, and `isVisible()` is true for content inside a
136  collapsed reveal container. Route gallery interactions through the
137  reveal-aware helpers and treat actionability (hit-target), not visibility,
138  as the readiness signal.
139- **Hermetic dependencies only.** CI runners have no host binaries and no
140  provider accounts, so agent availability must come from the harness, by one
141  of two mechanisms. Route-mocked specs intercept `/api/agents` with
142  `routeAgents` / `fulfillAgentsRoute` (via `applyStandardMocks`); that mock
143  must serve both the JSON and the `?stream=1` SSE shape including the
144  terminal `done` event — without it the streamed availability lands
145  transiently and is then cleared when the client rejects the incomplete
146  stream.
147  Real-daemon specs instead run a fake CLI through `createFakeAgentRuntimes` +
148  `agentCliEnv` — and daemon detection resolves `byok-opencode` through the
149  `opencode` agent's env. A spec that only passes where a real CLI happens to
150  be installed is broken.
151- **Oracles assert the running product's observable behavior.** When a spec
152  goes stale, realign it against the current product, not the spec's own
153  history. A merged product PR is not proof its oracle is verified: what runs
154  on the merge path is governed by registration, not priority tag — the
155  `ui_p0` group files and the `@merge-extra` P1 subset (`playwright_critical`
156  is a mutually-exclusive PR fallback — `run_playwright_critical` is
157  `&& !runUiP0` — so `@critical` does not add coverage on the merge queue). A
158  non-visual P1 case outside those also runs in the manual `p0p1` lane; a
159  non-visual P2 case, or a P0 case in a file no group lists, runs only in the
160  full pool (visual tests have their own lane). Confirm a realigned oracle in
161  the lane that actually executes it, not by the PR merging.
162- **Name your failure causes.** A long wait should fail with a diagnosis, not
163  an opaque timeout. `sendPrompt` in `ui/real-daemon-run.test.ts` tracks
164  whether the create-run request was ever issued, so its failure separates "no
165  request left the page" (a composer/overlay gate) from "no accepted response
166  arrived" — enough to point at the right layer in a CI-only reproduction.
167  Assert gate preconditions explicitly for the same reason. A hang is usually a
168  missing gate, not a too-short budget — diagnose the state the wait targets
169  before raising it.
170 
171## Naming and tools
172 
173- `specs/` files must be `*.spec.ts`; `tests/` files must be `*.test.ts`.
174- Prefer directory hierarchy over long file names. Basenames should normally be three words or fewer, such as `main.spec.ts`, `run.spec.ts`, `inspect.test.ts`, or `report.test.ts`.
175- `ui/` files must be flat `*.test.ts` Playwright tests. Do not add subdirectories, TSX, Vitest, jsdom, Testing Library, or React harness tests under `ui/`.
176- `ui/` tests must import runtime-bound `test`/`expect` from `@/playwright/suite`; use `@playwright/test` only for type imports or low-level helper modules that do not own test lifecycle.
177- E2E Vitest tests use Node APIs; do not add JSX/TSX, jsdom, or browser-component tests under `specs/` or `tests/`.
178- Web component/runtime tests belong in `apps/web/tests/`, not `e2e/ui/`.
179- E2E tests may validate cross-app/resource consistency, but must not treat one app's private implementation as a shared helper for another app. Keep test-only helpers local to `e2e/lib/` or promote reusable logic to a pure package such as `packages/contracts`.
180- E2E imports may use `@/*` for `lib/*`; keep this alias local to the e2e package.
181 
182## Commands
183 
184Run commands from this directory:
185 
186```bash
187pnpm test specs/mac.spec.ts
188pnpm test tests/tools-dev/inspect.test.ts
189pnpm test specs
190pnpm test tests
191pnpm test:p0
192pnpm test:p0p1
193pnpm test:ui:p0
194pnpm test:ui:p0p1
195pnpm typecheck
196pnpm exec tsx scripts/playwright.ts clean
197pnpm exec playwright test -c playwright.config.ts --list
198pnpm exec playwright test -c playwright.config.ts
199```
200 
201Use a specific file path when validating a single case. Do not add root e2e aliases or extra package scripts for individual cases.
202 
203Case-level priority tags use test-name prefixes: `[P0]`, `[P1]`, `[P2]`.
204 
205Playwright UI runs use one tools-dev daemon/web/data root per Playwright worker. The single-worker fallback is `--workers=1` (or `OD_PLAYWRIGHT_WORKERS=1`); do not reintroduce a shared daemon/web runtime mode.
206