AGENTS.md

1# ultimate-browsing/engine — Generic WAF-Profile Fetch Chain (Python)
2 
3**Generated:** 2026-08-10 / 38d268995
4 
5## OVERVIEW
6 
7A 17-module Python package embedded in the `ultimate-browsing` skill: a site-agnostic fetch chain that escalates from a cheap curl probe to a real browser, with declarative WAF and surrogate registries. Not "optional scripts" — it has its own CLI entry (`python3 -m engine URL`), two YAML config schemas, a 4-file test suite, and a standalone CI guard. Package exports (`__init__.py`): `fetch`, `FetchResult`, `Attempt`, `Verdict`, `ValidationResult`, `validate`, `CHALLENGE_MARKERS`, `detect`, `TRANSFORMS`, `apply_transform`.
8 
9## THE NO-SITE-NAME RULE (enforced in CI)
10 
11`engine/**` must contain **zero** hard-coded site names, brands, or target domains. Site specifics belong to runtime hints or observations, never to code. `bias_check.py` is a standalone scanner enforcing this: a brand denylist, a URL regex scan, an allowlist for genuine infrastructure hosts (archive.org, r.jina.ai, google.com, httpbin.org, relay.invalid), and a `# NOTE-BIAS-OK` comment convention for legitimate exemptions such as test fixtures.
12 
13```bash
14python3 engine/bias_check.py       # fails on any site-specific leak
15```
16 
17## FETCH CHAIN PHASES
18 
19```
20fetch(url, ...)                                   # fetch_chain.py
21  Phase 1  curl_probe.py    — curl_cffi TLS-impersonation probe
22  Phase 2  grid             — referer/transform/device attempt grid
23  Phase 2.5 surrogate.py    — third-party archive/reader/proxy routes
24  Phase 3  executor.py      — capability-matched Playwright fallback
25```
26 
27Ordering is **not** hardcoded: each `waf_profiles.yaml` profile carries a `fallback_when_challenge` list that drives the ladder. `surrogate_wayback` precedes browser executors in every profile, so archives are tried before paying for a browser spin-up.
28 
29## PROVENANCE / TRUST CONTRACT
30 
31`result_schema.py` puts two literals on every `FetchResult`:
32 
33- `Provenance = "live" | "snapshot" | "proxy"`
34- `Trust = "origin" | "archive" | "untrusted"`
35 
36A `snapshot` result carries `snapshot_timestamp` and **must** be cited with that timestamp — never presented as the live page. `surrogates.yaml` `kind` fixes these values: `archive` -> snapshot/archive, `reader` -> live, `proxy` -> proxy/untrusted.
37 
38## SURROGATE REGISTRY (`surrogates.yaml`)
39 
40Site-agnostic infrastructure only. Every entry carries `last_verified` (ISO date); entries older than 90 days are deprioritized and flagged, because surrogate routes rot (a 2026-08 probe found 4 of 6 known routes dead or stubbed). `proxy` routes are MITM by construction: they require the explicit `--allow-proxy` flag and never receive `Cookie` or `Authorization` headers. Every surrogate response is re-validated with `target_url` set, so an interstitial or a redirect stub is rejected instead of returned as content.
41 
42## VALIDATOR LAYERS (`validators.py`)
43 
44```
45L1    challenge markers (CHALLENGE_MARKERS)
46L1.5  surrogate dead ends — interstitial titles + AMP-style redirect stubs
47      (is_redirect_stub(), needs target_url)
48L2    size/shape fingerprints
49L3+   content checks
50```
51 
52## CLI
53 
54```bash
55python3 -m engine URL [--selector S] [--device auto|desktop|mobile]
56                      [--timeout 25] [--max-attempts 12]
57                      [--no-playwright] [--allow-proxy] [--json] [--trace]
58```
59 
60## TESTS
61 
62`tests/` — `test_surrogate.py` (staleness, proxy gating, short-circuit), `test_surrogate_validators.py`, `test_fetch_chain.py`, `test_playwright_templates.py`, plus HTML/JSON fixtures under `tests/fixtures/`.
63 
64## NOTES
65 
66- `summary.py` emits an **R7 API-first hint** after >=3 challenge verdicts against a known WAF profile: look for `/api/`, `/graphql`, or `.json` endpoints, which usually carry weaker WAF protection than the HTML surface.
67- `templates/` holds the Playwright JS templates (`playwright_real_chrome.js`, `playwright_mobile_chrome.js`) the executor drives.
68- `url_transforms.py` transforms stay domain-agnostic (`mobile_subdomain`, `am_prefix`, `drop_www`).
69- Parent: [`packages/shared-skills/AGENTS.md`](../../../AGENTS.md).
70