AGENTS.md

1# pi-webfetch
2 
3URL retrieval as a Pi extension, vendored from the standalone
4`code-yeongyu/pi-webfetch` repository into this monorepo as
5`@oh-my-opencode/pi-webfetch` (private workspace package, Adapter layer:
6Pi-harness-coupled).
7 
8Registers one LLM-callable tool, `webfetch`, which fetches a URL and returns its
9content as markdown (HTML converted via turndown), plain text, or raw HTML, with
10bounded timeout and a 5 MB response-size cap. Toggle off with `PI_WEBFETCH=0`.
11 
12## Anatomy
13 
14| Path | Purpose |
15|------|---------|
16| `src/index.ts` | Extension entry: `PI_WEBFETCH` toggle + tool registration + UI clearing on session lifecycle |
17| `src/webfetch/tool.ts` | `webfetch` tool definition (typebox params, execute) |
18| `src/webfetch/fetcher.ts` | Bounded fetch: URL validation, timeout, size cap, Cloudflare-challenge retry |
19| `src/webfetch/content.ts` | HTML -> markdown (turndown) and HTML -> text conversion + entity decoding |
20| `src/webfetch/renderers.ts` | TUI call/result renderers |
21| `src/webfetch/errors.ts` | Typed fetch errors |
22| `test/` | Vendored suite (bun:test): real HTTP fixture server + tool-definition + toggle tests |
23| `scripts/qa/drive.mjs` | Live QA driver: real pi CLI (RPC mode) fetching a local HTTP fixture, `--self-test` |
24| `scripts/qa/mock-provider/` | Self-contained scripted provider extension (no network to the model, no keys) |
25 
26## Conventions
27 
28- Vendored source: keep diffs against upstream intentional and reviewable. Tests
29  were converted vitest -> bun:test; the HTTP-fixture teardown was hardened to
30  destroy tracked sockets directly because Bun's fetch pools a cancelled socket
31  (so `server.close()` would otherwise hang), and the two oversized tests plus the
32  Cloudflare-challenge test drop the undici-specific server-side socket-close
33  proxy in favor of asserting the production contract (reject oversized, retry
34  past the challenge).
35- `turndown` is a real runtime dependency. Peer deps (`@mariozechner/pi-*`,
36  `typebox`) resolve from the host Pi runtime; pinned devDependencies exist only
37  for typecheck + tests + live QA.
38- Not wired into any OpenCode/Codex/omo-senpi component. Ships as a standalone Pi
39  package surface (`pi.extensions` manifest field).
40 
41## QA
42 
43```sh
44bun test packages/pi-webfetch                      # unit + real-HTTP-fixture gate
45tsgo --noEmit -p packages/pi-webfetch/tsconfig.json
46node packages/pi-webfetch/scripts/qa/drive.mjs --self-test
47node packages/pi-webfetch/scripts/qa/drive.mjs     # live pi-harness proof (RPC mode, sandboxed)
48```
49 
50The live driver is the real-harness gate: it drives the real pi CLI to call
51`webfetch` against a local fixture and asserts the HTML-to-markdown result.
52Evidence goes to `.omo/evidence/<date>-<slug>/`.
53