CLAUDE.md

1# OpenWeb
2 
3Agent-native way to access any website. Bridging agent CLI and human GUI through API.
4 
5## Quick Start
6 
7```bash
8pnpm install && pnpm build
9pnpm test              # run all tests
10pnpm lint              # biome lint
11pnpm dev sites         # list available sites
12pnpm dev <site> <op> '{...}'   # execute an operation
13```
14 
15## Documentation
16 
17- **Shipped skill:** `skills/openweb/` (SKILL.md router, references, knowledge)
18- **Architecture & system docs:** `doc/main/README.md`
19- **Dev workflow & guides:** `doc/dev/development.md`
20- **Progress history:** `doc/PROGRESS.md`
21- **Project docs:** `projects/active/` (active work) and `projects/archive/` (historical docs), symlinked from the sibling openweb-projects repo, commit separately
22 
23## Documentation Sync
24 
25When running /update-doc, also check and sync `skills/openweb/` — this repo ships skill docs as a deliverable alongside code. See `doc/main/README.md` § Skill Documentation for the governing principles and structure. Priority areas: files that reference runtime behavior, CLI flags, x-openweb fields, or transport/auth semantics.
26 
27## Project Structure
28 
29```
30src/
31  compiler/
32    types.ts          # Core types (RecordedRequestSample, SampleResponse)
33    types-v2.ts       # Pipeline v2 contracts (5-phase type definitions)
34    recorder.ts       # HAR parsing + scripted recording
35    analyzer/         # Phase 2: label → normalize → cluster → schema → auth
36      analyze.ts      # Orchestrator: analyzeCapture() → AnalysisReport
37      labeler.ts      # Sample categorization (api/static/tracking/off_domain)
38      path-normalize.ts  # Path template normalization
39      graphql-cluster.ts # GraphQL sub-clustering
40      auth-candidates.ts # Ranked auth bundling with evidence + CSRF options
41      constant-headers.ts # Detect constant headers across cluster samples
42      schema-v2.ts    # Schema inference with enum/format controls
43      example-select.ts # Tiered example value selection with PII scrub
44      classify.ts     # Extraction signals (ssr_next_data, script_json, page_global)
45      differentiate.ts # Query vs path param differentiation
46      auth-detect.ts, csrf-detect.ts, signing-detect.ts  # Primitive detection
47    curation/         # Phase 3: apply-curation.ts, scrub.ts (PII)
48    generator/        # Phase 4: generate-v2.ts (OpenAPI + AsyncAPI emission)
49    ws-analyzer/      # WS capture → classify → cluster → schema
50  runtime/
51    executor.ts     # Main dispatch (HTTP + WS)
52    http-executor.ts, browser-fetch-executor.ts, node-ssr-executor.ts
53    browser-lifecycle.ts  # Auto browser management (ensureBrowser, auth cascade, watchdog)
54    warm-session.ts       # Anti-bot sensor warm-up (Akamai, DataDome cookie wait)
55    response-unwrap.ts    # Generic response unwrap (x-openweb.unwrap dot-path extraction)
56    session-executor.ts, adapter-executor.ts, extraction-executor.ts
57    ws-executor.ts, ws-connection.ts, ws-router.ts, ws-runtime.ts
58    cache-manager.ts, token-cache.ts
59    navigator.ts    # CLI navigation helper (render site/operation info)
60    page-polyfill.ts, page-candidates.ts  # Browser page utilities
61    primitives/     # Auth/CSRF/signing resolvers
62  lifecycle/        # Verify, registry, shape-diff (site drift detection)
63  capture/          # CDP browser recording (body-size-gate, no content filtering)
64  commands/         # CLI command handlers
65  types/            # Meta-spec type system
66  sites/            # Site packages (openapi.yaml, DOC.md, adapters/)
67  lib/
68    site-resolver.ts, spec-loader.ts, site-package.ts
69    param-validator.ts, permissions.ts, permission-derive.ts
70    logger.ts, config.ts, cookies.ts, adapter-params.ts
71    errors.ts, ssrf.ts, openapi.ts, asyncapi.ts
72    adapter-helpers.ts  # pageFetch, graphqlFetch — injected into adapters via runtime
73    manifest.ts, url-builder.ts, response-parser.ts, csrf-scope.ts
74    config/             # Config files: blocked-domains, blocked-paths, tracking-cookies, static-extensions
75skills/openweb/      # The shipped skill (references, knowledge)
76install-skill.sh    # Skill installer for Claude Code, Codex, OpenCode, OpenClaw
77```
78