AGENTS.md

1# AGENTS.md
2 
3Agent entry point for WorldMonitor. Read this first, then follow links for depth.
4 
5## What This Project Is
6 
7Real-time global intelligence dashboard. TypeScript SPA (Vite + Preact) with 186 top-level TypeScript component files, 80+ Vercel Edge API endpoint entries, a Tauri desktop app with Node.js sidecar, and a Railway relay service. Aggregates geopolitics, military, finance, climate, cyber, maritime, and aviation data across 37 freshness-tracked source groups.
8 
9## Repository Map
10 
11```
12.
13├── src/                    # Browser SPA (TypeScript, class-based components)
14│   ├── app/                # App orchestration (data-loader, refresh-scheduler, panel-layout)
15│   ├── bootstrap/          # Startup/recovery (chunk reload, deferred Sentry, SW update)
16│   ├── components/         # 186 top-level TypeScript component files
17│   ├── config/             # Variant configs, panel/layer definitions, market symbols
18│   ├── features/           # Self-contained feature surfaces (stock research room)
19│   ├── services/           # Business logic (237 service modules and domain directories)
20│   ├── shared/             # Cross-cutting helpers (premium paths, registries, staleness)
21│   ├── embed/              # Embeddable widget loader
22│   ├── styles/             # Global CSS (layers, themes, panel styles)
23│   ├── shims/              # Runtime shims (child-process for sidecar)
24│   ├── data/               # Static JSON datasets (conservation, renewable, happiness)
25│   ├── e2e/                # Map test harnesses (consumed by Playwright specs)
26│   ├── types/              # TypeScript type definitions
27│   ├── utils/              # Shared utilities (circuit-breaker, theme, URL state, DOM)
28│   ├── workers/            # Web Workers (analysis, ML/ONNX, vector DB)
29│   ├── generated/          # Proto-generated client/server stubs (DO NOT EDIT)
30│   ├── locales/            # i18n translation files
31│   └── App.ts              # Main application entry
32├── api/                    # Vercel Edge Functions (plain JS, self-contained)
33│   ├── _*.js               # Shared helpers (CORS, rate-limit, API key, relay)
34│   ├── health.js           # Health check endpoint
35│   ├── bootstrap.js        # Bulk data hydration endpoint
36│   └── <domain>/           # Domain-specific endpoints (aviation/, climate/, etc.)
37├── server/                 # Server-side shared code (used by Edge Functions)
38│   ├── _shared/            # Redis, rate-limit, LLM, caching, response headers
39│   ├── gateway.ts          # Domain gateway factory (CORS, auth, cache tiers)
40│   ├── router.ts           # Route matching
41│   └── worldmonitor/       # Domain handlers (mirrors proto service structure)
42├── proto/                  # Protobuf definitions (sebuf framework)
43│   ├── buf.yaml            # Buf configuration
44│   └── worldmonitor/       # Service definitions with HTTP annotations
45├── shared/                 # Cross-platform data (JSON configs for markets, RSS domains)
46├── data/                   # Static data (telegram channels, OREF threat translations, gamma irradiators)
47├── public/                 # Static assets served as-is (favicons, textures, .well-known, llms.txt)
48├── scripts/                # Seed scripts, build helpers, data fetchers
49├── src-tauri/              # Tauri desktop shell (Rust + Node.js sidecar)
50│   └── sidecar/            # Node.js sidecar API server
51├── consumer-prices-core/   # Consumer-price scrapers (Playwright, per-country baskets; Railway/Docker)
52├── workers/                # Cloudflare Workers (edge CORS preflight for api.worldmonitor.app)
53├── tests/                  # Unit/integration tests (node:test runner)
54├── e2e/                    # Playwright E2E specs
55├── pro-test/               # Standalone Pro QA app (separate package)
56├── docs/                   # Mintlify documentation site
57│   └── solutions/          # Documented solutions to past problems (bugs, patterns, practices) — YAML frontmatter (module, tags, problem_type)
58├── docker/                 # Docker build for Railway services
59├── deploy/                 # Deployment configs (nginx)
60├── CONCEPTS.md             # Shared domain vocabulary (entities, named processes, status concepts)
61└── blog-site/              # Static blog (built into public/blog/)
62```
63 
64## How to Run
65 
66```bash
67npm ci                   # Deterministic install (also runs blog-site postinstall)
68npm run dev              # Start Vite dev server (full variant)
69npm run dev:tech         # Start tech-only variant
70npm run dev:energy       # Start energy-security variant
71npm run typecheck        # tsc --noEmit (strict mode)
72npm run typecheck:api    # Typecheck API layer separately
73npm run test:data        # Run unit/integration tests
74npm run test:sidecar     # Run sidecar + API handler tests
75npm run test:e2e         # Run all Playwright E2E tests
76make generate            # Regenerate proto stubs + per-service & unified OpenAPI specs (requires buf + sebuf v0.11.1 plugins)
77npm run worktree:bootstrap          # Fresh worktree: link local env files + npm ci with tmp cache
78npm run worktree:bootstrap:test-only # Fresh docs/test worktree: same, but npm ci --ignore-scripts
79npm run worktree:env                # Link ignored local env files only
80```
81 
82## Fresh Worktree Bootstrap
83 
84Worktrees usually start without ignored local state. When creating or entering one:
85 
861. Start from `origin/main` or the requested base, not a dirty local branch.
872. Run `npm run worktree:bootstrap` before typecheck/tests. The helper links ignored `.env.local` / `.env` from the main worktree when Git can infer it, and installs deps with `npm ci --cache /tmp/worldmonitor-npm-cache`.
883. If only docs/test tooling is needed and native postinstall work is unnecessary, use `npm run worktree:bootstrap:test-only`.
894. If live credentials are unavailable, do not fabricate secrets. Run the non-credentialed checks you can and report the credential gate explicitly.
90 
91Env rules:
92 
93- Link only `.env.local` and `.env`. Never copy or link `.env.vercel-backup` or `.env.vercel-export`; the pre-push guard blocks those files even as symlinks.
94- Override env source discovery with `WM_ENV_SOURCE=/path/to/worldmonitor npm run worktree:env` when the main worktree cannot be inferred.
95- `.env*` files are ignored local state. Do not add, print, or summarize secret values.
96 
97Validation hygiene:
98 
99- Prefer `npm ci` over `npm install` in fresh worktrees. Use `npm_config_cache=/tmp/worldmonitor-npm-cache` for `npx` or install commands if cache ownership errors appear.
100- After bootstrap or pre-push, run `git status --short`. If dependency bootstrap changed lockfiles you did not intend to edit, remove those incidental changes before finalizing.
101- After install, prefer local tools such as `./node_modules/.bin/tsx --test ...` for focused TypeScript tests when `npx` is flaky.
102 
103## Architecture Rules
104 
105### Dependency Direction
106 
107```
108types -> config -> services -> components -> app -> App.ts
109```
110 
111- `types/` has zero internal imports
112- `config/` imports only from `types/`
113- `services/` imports from `types/` and `config/`
114- `components/` imports from all above
115- `app/` orchestrates components and services
116 
117### API Layer Constraints
118 
119- `api/*.js` are Vercel Edge Functions: **self-contained JS only**
120- They CANNOT import from `../src/` or `../server/` (different runtime)
121- Only same-directory `_*.js` helpers and npm packages
122- Enforced by `tests/edge-functions.test.mjs` and pre-push hook esbuild check
123 
124### Server Layer
125 
126- `server/` code is bundled INTO Edge Functions at deploy time via gateway
127- `server/_shared/` contains Redis client, rate limiting, LLM helpers
128- `server/worldmonitor/<domain>/` has RPC handlers matching proto services
129- All handlers use `cachedFetchJson()` for Redis caching with stampede protection
130 
131### Proto Contract Flow
132 
133```
134proto/ definitions -> buf generate -> src/generated/{client,server}/ -> handlers wire up
135```
136 
137- GET fields need `(sebuf.http.query)` annotation
138- `repeated string` fields need `parseStringArray()` in handler
139- `int64` maps to `string` in TypeScript
140- CI checks proto freshness via `.github/workflows/proto-check.yml`
141 
142## Variant System
143 
144The app ships multiple variants with different panel/layer configurations:
145 
146- `full` (default): All features
147- `tech`: Technology-focused subset
148- `finance`: Financial markets focus
149- `commodity`: Commodity markets focus
150- `happy`: Positive news only
151- `energy`: Energy security, chokepoints, oil/gas, and disruption timelines
152 
153Variant is set via `VITE_VARIANT` env var. Config lives in `src/config/variants/`.
154 
155## Key Patterns
156 
157### Adding a New API Endpoint
158 
1591. Define proto message in `proto/worldmonitor/<domain>/`
1602. Add RPC with `(sebuf.http.config)` annotation
1613. Run `make generate`
1624. Create handler in `server/worldmonitor/<domain>/`
1635. Wire handler in domain's `handler.ts`
1646. Use `cachedFetchJson()` for caching, include request params in cache key
165 
166### Adding a New Panel
167 
1681. Create `src/components/MyPanel.ts` extending `Panel`
1692. Register in `src/config/panels.ts`
1703. Add to variant configs in `src/config/variants/`
1714. Wire data loading in `src/app/data-loader.ts`
172 
173### Circuit Breakers
174 
175- `src/utils/circuit-breaker.ts` for client-side
176- Used in data loaders to prevent cascade failures
177- Separate breaker per data domain
178 
179### Caching
180 
181- Redis (Upstash) via `server/_shared/redis.ts`
182- `cachedFetchJson()` coalesces concurrent cache misses
183- Cache tiers: fast (5m), medium (10m), slow (30m), static (2h), daily (24h)
184- Cache key MUST include request-varying params
185 
186## Testing
187 
188- **Unit/Integration**: `tests/*.test.{mjs,mts}` using `node:test` runner
189- **Sidecar tests**: `api/*.test.mjs`, `src-tauri/sidecar/*.test.mjs`
190- **E2E**: `e2e/*.spec.ts` using Playwright
191- **Visual regression**: Golden screenshot comparison per variant
192 
193## CI Checks (GitHub Actions)
194 
195| Workflow | Trigger | What it checks |
196|---|---|---|
197| `typecheck.yml` | PR + push to main | `tsc --noEmit` for src and API |
198| `lint.yml` | PR (markdown changes) | markdownlint-cli2 |
199| `proto-check.yml` | PR (proto changes) | Generated code freshness |
200| `build-desktop.yml` | `v*` tag, manual | Tauri desktop build |
201| `test-linux-app.yml` | Twice-weekly schedule, manual | Desktop Canary (Linux): release-processed AppImage smoke — crash, sidecar readiness/liveness, rendered content |
202| `test.yml` (`desktop-config`, `desktop-rust` jobs) | PR touching desktop-coupled paths | Desktop version consistency, AppImage post-processing syntax, Tauri config/capability parse, desktop build env parity (#5905, also in `unit`), `cargo test --locked` (#5902) |
203 
204## Pre-Push Hook
205 
206Runs automatically before `git push`. Two tiers:
207 
208**Always (state-dependent, fast — run even on a cache hit):** local Vercel env-dump guard, PR-state check (no pushes to merged/closed PR branches), branch-contamination guard (>20 commits ahead), `scripts/` lockfile sync.
209 
210**Tree-dependent (skipped entirely on a green-tree cache hit):** Unicode safety and version sync (always run for uncached trees), plus the diff-scoped checks: TypeScript (frontend tsc on `src/`-surface changes; `typecheck:api` on `api/|server/|scripts/|src/generated/`; Convex tsc on `convex/`), CJS syntax, boundary/safe-html/Sentry-coverage/rate-limit/premium-fetch lints (each also fires when its own guardrail script changes), edge esbuild check (`api/|server/|src/generated/|scripts/check-edge-function-bundles.mjs` — edge entries bundle-import server code, and the shared checker retriggers its own gate), markdown/MDX lint, proto + pro-test bundle freshness, change-scoped tests. `package.json`/`tsconfig` changes — or an unresolvable `origin/main` diff — force everything (an unresolvable diff also bypasses the green-tree cache: a blind run trusts nothing, including prior attestations).
211 
212**Green-tree cache:** a tree that passed the full gate is recorded (`$GIT_DIR/wm-prepush-green`); re-pushing the identical tree (remote failure, message-only amend) skips all tree-dependent checks — same tree, same result. Delete that file to force a full re-run.
213 
214Heavy checks (`test:data`, typechecks, edge-bundle) must run **sequentially** in worktrees — parallel runs OOM (exit 137).
215 
216## Shipping Velocity (Agent Workflow)
217 
218- **Before starting work on an issue:** check for parallel/duplicate work first — `gh pr list --search "<issue#>"` AND `git worktree list` (background codex/claude sessions ship PRs under the same account).
219- **PR delivery authority:** a user request to implement, fix, or ship a scoped change authorizes creating and updating the ready PRs needed to deliver it, including corrective follow-up PRs discovered by review or CI, plus monitoring and repairing those PRs without additional per-PR confirmation. This authority is limited to the requested change and its delivery branches; review-only or diagnostic requests remain read-only.
220- **Merge authority is explicit and non-delegable:** never merge a PR, enable auto-merge, queue a merge, or run any equivalent GitHub merge action unless the user has explicitly requested that specific action in the current conversation. A request to implement, ship, push, create a PR, or monitor CI does **not** authorize merging. Wait for clear approval and report the ready state instead.
221- **PR push readiness is mandatory:** before every push, re-fetch the live PR head and base, verify the remote head has not advanced, and check GitHub mergeability. Do not push a branch that is behind, `CONFLICTING`, or `DIRTY`; update from the latest PR/base state and resolve conflicts first. A successful `git push` is not delivery completion.
222- **After pushing a PR:** start `gh pr checks <n> --watch` (or an equivalent bounded monitor), wait for all required CI checks to reach green, then re-fetch the PR head and verify GitHub reports no conflict (`mergeable: MERGEABLE` / clean merge state). If checks are pending, failing, or the PR becomes conflicting, keep repairing and re-checking; do not report the PR as ready or complete until both CI and mergeability are green. Never use `--no-verify` to bypass this gate or turn on auto-merge without the explicit approval above.
223- **docs/plans/ is gitignored** — plan documents are local working state and do not travel between worktrees or ship in PRs.
224- **PR-review verification:** never assert a finding is fixed/stale from memory — re-fetch the PR head SHA and diff the cited lines first.
225 
226## Deployment
227 
228- **Web**: Vercel (auto-deploy on push to main)
229- **Relay/Seeds**: Railway (Docker, cron services)
230- **Desktop**: Tauri builds via GitHub Actions
231- **Docs**: Mintlify (proxied through Vercel at `/docs`)
232 
233## Critical Conventions
234 
235- `fetch.bind(globalThis)` is BANNED. Use `(...args) => globalThis.fetch(...args)` instead
236- Edge Functions cannot use `node:http`, `node:https`, `node:zlib`
237- Always include `User-Agent` header in server-side fetch calls
238- Yahoo Finance requests must be staggered (150ms delays)
239- New data sources MUST have bootstrap hydration wired in `api/bootstrap.js` — unless nothing in `src/` renders them. A dataset with no dashboard consumer registers in `api/health.js` `STANDALONE_KEYS` instead and stays out of the tiered payload every client downloads; `tests/bootstrap.test.mjs` enforces the converse, that no tier key lacks a `getHydratedData`/`ensureHydrated` consumer. Once a panel does read one, promote it into `BOOTSTRAP_CACHE_KEYS` with a tier — `ON_DEMAND_KEY_NAMES` for an opt-in panel, so the payload is fetched per-key on render rather than riding a tier every visitor downloads (`fxYoy` and `sharedFxRates` went this way for the FX panel, #6199)
240- Redis seed scripts MUST write `seed-meta:<key>` for health monitoring
241- Seed credentials load only via `loadEnvFile()` (inert under test runtimes, resolves `.env.local` at the checkout root, `only:` narrows the keys) — never hand-roll a `.env` reader or resolve one from `$HOME` or an absolute literal. Note `worktree:bootstrap` symlinks the source checkout's `.env.local`, so a bootstrapped worktree shares real credentials when a seeder is actually run
242 
243## External References
244 
245- [Architecture (system reference)](ARCHITECTURE.md)
246- [Design Philosophy (why decisions were made)](docs/architecture.mdx)
247- [Contributing guide](CONTRIBUTING.md)
248- [Data sources catalog](docs/data-sources.mdx)
249- [Health endpoints](docs/health-endpoints.mdx)
250- [Adding endpoints guide](docs/adding-endpoints.mdx)
251- [API reference (OpenAPI)](docs/api/)
252