Copilot instructions

1---
2applyTo: 'build/azure-pipelines/{oss/**,common/downloadNotice.ts,{win32,linux,darwin}/steps/product-build-*-compile.yml}'
3---
4 
5# VS Code OSS Third-Party-Notices pipeline
6 
7This directory contains the thin VS Code-specific layer around Component Governance (CG) for producing OSS third-party notices. It replaces the legacy custom OSS tool with CG plus a gap-filling scanner, so release owners do not need to babysit `ThirdPartyNotices.txt` every release. The generated output should be at least as good as the legacy tool: CG provides the base NOTICE, the local scanner fills known CG gaps, and human-authored overrides cover the small set that automation cannot safely resolve.
8 
9The pipeline has two halves: **generation** (this `oss/` directory — CG + scanner + merge, producing the `notice_output` artifact) and **application** (the cutover that swaps the merged notice into the shipped product — see "Applying the NOTICE (cutover)" below).
10 
11## Architecture
12 
13The product-quality pipeline in `build/azure-pipelines/product-quality-checks.yml` is the main CI entry point.
14 
151. `notice@0` generates the CG base NOTICE at `ThirdPartyNotices.generated.txt`.
162. `scan-licenses.ts` scans local sources CG misses and writes `ThirdPartyNotices.extensions.txt`, plus sibling index files used by the merge step.
173. `merge-notices.ts` merges the CG output and scanner output, then applies `cglicenses.json` overrides through `apply-overrides.ts`.
184. `check-pr-dependencies.ts` is used by `pr-oss-check.yml` as a PR-time gate. It blocks dependency additions that have no license source.
195. If CG is down or `notice@0` emits an empty/non-trivial failure output, the pipeline substitutes the last good `ThirdPartyNotices.generated.txt` artifact from the same branch, then from `main`. Only the CG portion is cached; local scanning and overrides still run fresh against the current commit.
20 
21Final merge output is uploaded as `ThirdPartyNotices.new.txt` in the `notice_output` artifact.
22 
23## Pipeline flow in CI
24 
25In `product-quality-checks.yml`:
26 
271. Component Detection runs with `ComponentGovernanceComponentDetection@0`.
282. `notice@0` writes `$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt` and is `continueOnError: true`.
293. The cache fallback checks whether the CG file exists and is larger than 1 KB. If not, it downloads the latest `notice_output` artifact for the current branch, then `main`, and copies the cached `ThirdPartyNotices.generated.txt` into place.
304. TypeScript compiles the OSS scripts into `.oss-build-out`:
31   - `apply-overrides.ts`
32   - `scan-licenses.ts`
33   - `parse-notices.ts`
34   - `merge-notices.ts`
355. `scan-licenses.js` runs with `--repo`, `--cg`, and `--output`.
366. `merge-notices.js` runs with `--cg`, `--extensions`, `--cglicenses`, and `--output`.
377. The generated CG file, scanner file, final merged file, and optional cache metadata are uploaded under `notice_output`.
38 
39## Applying the NOTICE (cutover)
40 
41Generation (above) produces the merged `ThirdPartyNotices.new.txt` in the `notice_output` artifact. **Application** is the cutover that makes that file the one VS Code actually ships, replacing the legacy mixin notice.
42 
43Consumer script: `build/azure-pipelines/common/downloadNotice.ts` (read its header comment for the full design rationale).
44 
45How it works, per desktop platform compile template (`build/azure-pipelines/{win32,linux,darwin}/steps/product-build-*-compile.yml`):
46 
471. **Pull CG NOTICE (background)** — at compile start, `deemon --detach` launches `downloadNotice.ts` as a detached poller. It polls the parallel Quality stage's `notice_output` artifact while compilation runs, so the wait overlaps work already happening.
482. **Apply CG NOTICE** — right before gulp packages the app, `deemon --attach` blocks on that poller. It downloads, extracts, validates the merged notice, and **overwrites the repo-root `ThirdPartyNotices.txt`**. `build/gulpfile.vscode.ts` then packages whatever file is at that path → the CG notice ships.
49 
50Only the 7 desktop targets that bundle a notice run these steps (win32 x64/arm64, darwin universal, linux x64/arm64/armhf). REH/server/web/alpine are unaffected.
51 
52### Fallback chain (never fail the build)
53 
54`downloadNotice.ts` is **non-fatal — it always exits 0.** A notice problem must never break packaging. The outcomes, in order:
55 
561. **CG fresh** — the `notice_output` artifact is present and valid → overwrite with it.
572. **Cached CG** — if CG generation failed upstream, the Quality stage substitutes the last good `ThirdPartyNotices.generated.txt` (same branch, then `main`) before the artifact is published — so the consumer still gets a CG notice.
583. **Legacy** — if no usable artifact is available, the legacy mixin notice that `mixin-quality.ts` already laid down is left in place. `mixin-quality.ts` is deliberately left untouched (it's a shared chokepoint used by 8+ pipelines), which is what guarantees we never ship with *no* notice.
59 
60The accept-gate validates the *extracted content* (file present AND non-trivial) inside the poll loop — it does not trust the artifact listing alone. A mid-upload miss re-polls rather than falling back, which prevents a "mixed notice" race where one platform ships legacy while its siblings ship CG.
61 
62### Rollback lever
63 
64A queue-time checkbox **"Use legacy OSS Notice"** (parameter `VSCODE_USE_LEGACY_OSS_NOTICE`, default `false`) is the instant rollback. When checked, it derives `VSCODE_OVERWRITE_TPN=false`, and `downloadNotice.ts` skips the overwrite so the legacy notice ships — no code change or redeploy needed.
65 
66> ⚠️ `downloadNotice.ts` normalizes the flag with `.trim().toLowerCase()` before comparing, so YAML casing (`true`/`false` vs `True`/`False`) can't break the rollback lever. The pipeline still derives `VSCODE_OVERWRITE_TPN` as a lowercase string literal (`'true'`/`'false'`) for clarity. See the comment in `product-build-variables.yml`.
67 
68### Diagnosing a build
69 
70`downloadNotice.ts` logs three greppable markers so a build log answers "did the cutover work?":
71 
72- `[notice-cutover] RESULT=fresh` — overwrote from `notice_output` (the happy path).
73- `[notice-cutover] RESULT=fallback` — artifact missing → kept the legacy notice.
74- `[notice-cutover] RESULT=disabled` — feature flag off → kept the legacy notice.
75 
76To confirm a shipped artifact carries the CG notice, the root `resources/app/ThirdPartyNotices.txt` should be the large CG-merged file (multi-MB) rather than the ~3 MB legacy notice. Validate *intra-build* parity — all platforms in one build should produce a byte-identical notice (same SHA-256) — rather than checking against a fixed byte count, since CG's exact output size varies run-to-run.
77 
78## Script reference
79 
80### `scan-licenses.ts`
81 
82Purpose: produces the supplemental NOTICE entries that CG misses.
83 
84Invocation in CI:
85 
86```sh
87node .oss-build-out/scan-licenses.js \
88  --repo "$(Build.SourcesDirectory)" \
89  --cg "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" \
90  --output "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt"
91```
92 
93Key inputs:
94 
95- `--repo`: VS Code repo root.
96- `--cg`: optional CG NOTICE. Used to build `cgCovered` and `cgBodies`, so scanner network work is bounded and Cargo stub bodies can be detected.
97- `node_modules` trees under root, built-in extensions, `remote`, and `build`.
98- `cgmanifest.json` files.
99- `Cargo.lock` files.
100 
101Key outputs:
102 
103- Supplemental NOTICE text at `--output`.
104- Presence index at `<output>.presence.json` unless `--presence` is supplied.
105- Cargo stub override index at `<output>.stuboverride.json` unless `--stuboverride` is supplied.
106 
107Scanner sections:
108 
109- Section 1 scans built-in extension dependencies. CG skips npm packages with `engines.vscode`, but VS Code ships built-in extensions, so their bundled dependencies are scanned from extension `node_modules` folders.
110- Section 2 scans root `node_modules` for packages whose LICENSE files exist on disk but whose ClearlyDefined coverage did not produce CG NOTICE text.
111- Section 3 reads `licenseDetail` from `cgmanifest.json`, and for uncovered git components without inline text it tries `fetchLicenseFromGitRepo()` at the pinned `commitHash`.
112- Section 4 harvests Rust crate licenses from `Cargo.lock`; details below.
113- Section 5 enumerates platform-specific npm binary packages; details below.
114 
115### `merge-notices.ts`
116 
117Purpose: combines CG NOTICE entries, scanner entries, Cargo stub replacements, and `cglicenses.json` overrides into one NOTICE file.
118 
119Invocation in CI:
120 
121```sh
122node .oss-build-out/merge-notices.js \
123  --cg &quot;$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt&quot; \
124  --extensions &quot;$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt&quot; \
125  --cglicenses &quot;$(Build.SourcesDirectory)/cglicenses.json&quot; \
126  --output &quot;$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.new.txt&quot;
127```
128 
129Key behavior:
130 
131- Parses both input NOTICE files.
132- Deduplicates by lowercased `<name>@<version>`, preserving multiple shipped versions of the same package.
133- CG normally wins collisions.
134- Exception: keys listed in `<extensions>.stuboverride.json` let scanner Cargo entries replace CG entries whose body is only an SPDX-expression stub.
135- Reads `<extensions>.presence.json` so override injection can distinguish present-but-unlicensed packages from stale overrides.
136- Applies `cglicenses.json` with `applyOverrides()`.
137- Writes sorted NOTICE output with provenance markers for scanner, Cargo stub override, and manual override entries.
138 
139### `apply-overrides.ts`
140 
141Purpose: reads and applies human-authored `cglicenses.json` entries.
142 
143Key inputs:
144 
145- `cglicenses.json` parsed by `readCglicenses()` after `stripJsonComments()` removes JSONC line comments.
146- The merged `Map<string, MergedEntry>` from `merge-notices.ts`.
147- Optional `presentNames` from the scanner presence index.
148- Optional URI fetching for `fullLicenseTextUri` entries.
149 
150Supported override shapes:
151 
152- `{ name, prependLicenseText }`
153- `{ name, fullLicenseText }`
154- `{ name, fullLicenseTextUri, prependLicenseText? }`
155- Optional `version` narrows matching to one package version.
156- Legacy `licenseDetail` is treated as an alias for `fullLicenseText`.
157 
158Key behavior:
159 
160- Matching is case-insensitive on `name` and exact on `version`.
161- Matching overrides edit existing entries.
162- Overrides with usable text and a present package inject a new `cglicenses-override` entry.
163- Overrides for names absent from the presence index are stale: warn and skip.
164- Overrides with no target and no usable text are unmatched; `--strict` in `merge-notices.ts` fails on unmatched entries or URI errors, but stale entries are warn-only.
165 
166### `parse-notices.ts`
167 
168Purpose: parses NOTICE text files for diagnostics and for scanner logic.
169 
170CLI modes:
171 
172```sh
173node parse-notices.js --file &lt;path&gt;
174node parse-notices.js --diff &lt;pathA&gt; &lt;pathB&gt;
175```
176 
177Key behavior:
178 
179- `parseNoticeFile()` returns package name, version, license, line number, license text length, and optional `licenseText`.
180- The parser recognizes separator-delimited NOTICE entries and uses `isPackageHeader()` heuristics to avoid treating license prose as package headers.
181- Section 4 in `scan-licenses.ts` imports this parser to detect CG entries whose body is just an SPDX expression.
182 
183### `check-pr-dependencies.ts`
184 
185Purpose: PR-time license coverage gate for dependency changes.
186 
187Invocation in `pr-oss-check.yml`:
188 
189```sh
190node .oss-build-out/check-pr-dependencies.js \
191  --repo &quot;$(Build.SourcesDirectory)&quot; \
192  --base &quot;origin/main&quot; \
193  --head &quot;HEAD&quot; \
194  --cglicenses &quot;$(Build.SourcesDirectory)/cglicenses.json&quot;
195```
196 
197Key behavior:
198 
199- Diffs changed `package.json` files between `--base` and `--head`.
200- For added dependencies, passes only if one of these is true:
201  - ClearlyDefined has usable license metadata and a non-zero score.
202  - A LICENSE file exists in local or hoisted `node_modules`.
203  - `cglicenses.json` has an override for the package.
204- For removed dependencies, fails if `cglicenses.json` still has an override for that package.
205- `--no-clearlydefined` disables the HTTP lookup for offline runs.
206 
207### `cglicenses.json`
208 
209Purpose: last-resort manual override file at repo root.
210 
211Use this only when CG and direct source scanning genuinely cannot provide usable text. Entries are read by `readCglicenses()` and applied by `applyOverrides()` during merge.
212 
213Important: the contents must be human-authored. Do not machine-manufacture license or copyright text. If CG's raw output has wrong text, file an upstream bug and ship CG's output as-is rather than hand-authoring a replacement. Add an override only when CG has no usable text or when a package lacks text in a way CELA has approved for manual override.
214 
215## Notable scanner details
216 
217### Cargo Section 4
218 
219Section 4 closes two Rust gaps:
220 
2211. Crates in `Cargo.lock` that are absent from CG.
2222. Crates where CG emitted the SPDX expression as the NOTICE body, such as `Zlib OR Apache-2.0 OR MIT`, instead of real license text.
223 
224Important functions and constants:
225 
226- `parseCargoLock()` extracts `name`, `version`, and `source` from `[[package]]` blocks.
227- `fetchCratesIoJson()` calls `https://crates.io/api/v1/crates/<name>` with `CRATES_IO_USER_AGENT`.
228- `getCrateRepository()` applies legacy repository URL overrides for crates such as `isatty`, `redox_syscall`, `redox_termios`, and `termion`.
229- `crateLicenseRefs()` tries `v<version>`, `<version>`, `<name>-v<version>`, `<name>-<version>`, then `main` and `master`.
230- `isSpdxStub()` detects CG bodies that are only SPDX expressions.
231- `fetchCargoLicense()` uses SPDX IDs to fetch real per-license files such as `LICENSE-MIT` or `LICENSES/Apache-2.0.txt`, with `fetchLicenseFromGitRepo()` as a generic fallback.
232- `hasSpdxAnd()` warns when the expression contains `AND`.
233 
234Maintenance rules:
235 
236- OR-license selection is first-in-expression wins: `spdxLicenseIds()` preserves order and `fetchCargoLicense()` returns the first SPDX ID that yields a real body. For `OR`, that is the licensee choice.
237- Conjunctive `AND` expressions are not reduced to one safe body. The scanner warns with `AND-LICENSE INCOMPLETE`; add a human-authored `cglicenses.json` override with the combined required text if this appears.
238- Cargo license fetch failures should warn and continue, not crash the build.
239 
240Tests:
241 
242- `cargo-section4.test.ts` covers `isSpdxStub()`, `parseCargoLock()`, `getCrateRepository()`, and `crateLicenseRefs()` without network.
243- `cargo-section4.live.ts` is a bounded live check that hits crates.io and GitHub for representative crates.
244 
245### Platform-binary Section 5
246 
247Section 5 closes a parity gap with the legacy OSS tool for arch-specific npm packages. Packages such as `@img/sharp-win32-x64` or `@esbuild/linux-x64` are optional dependencies of an arch-independent parent. On a single-platform build agent, only the host arch may be installed, but VS Code ships multiple platform and arch combinations.
248 
249Important functions and constants:
250 
251- `ARCH_SUFFIX_RE` and `isArchPackageName()` identify arch-bearing npm package names.
252- `VSCODE_SHIPPED_PLATFORMS` is currently `darwin`, `linux`, `linuxmusl`, and `win32`.
253- `VSCODE_SHIPPED_ARCHS` is currently `x64`, `arm64`, and `arm`.
254- `isShippedArch()` strips ABI suffixes like `-gnu`, `-musl`, `-msvc`, `-glibc`, `-gnueabihf`, `-eabihf`, and `-androideabi`, then checks the static shipped sets.
255- `npmLicenseId()` reads npm license shapes: string, `{ type }`, `licenses: [{ type }]`, and `licenses: [string]`.
256- `fetchNpmRegistryJson()` fetches a package packument when an arch child is not installed on disk.
257- `familyText()` caches license text by repository URL and reuses it across arch siblings.
258- `parentTextIfCompatible()` allows parent text fallback only when parent and child license IDs match exactly, case-insensitively.
259 
260Maintenance rules:
261 
262- The shipped-arch filter intentionally uses the static constants `VSCODE_SHIPPED_PLATFORMS` and `VSCODE_SHIPPED_ARCHS` in `scan-licenses.ts`.
263- Do not import these constants from `build/agent-sdk/common.ts`. Agent SDK excludes `armhf` and uses `alpine` where npm package names use `linuxmusl`, so it is not a direct fit for this scanner.
264- If VS Code adds or removes a shipped platform or arch, update those constants. The source of truth is noted in the code comment: `build/azure-pipelines/product-build.yml` plus `build/agent-sdk/common.ts`. Expect this to change about once a year.
265- Each arch child's own license ID is authoritative. Do not blindly reuse parent license text across a different child license ID; that would repeat the legacy sharp/libvips defect.
266 
267Tests:
268 
269- `platform-binary.test.ts` covers arch suffix recognition, shipped-arch filtering, npm license shape parsing, family text reuse, and parent fallback compatibility.
270 
271## Running locally
272 
273A helper exists at `build/azure-pipelines/oss/run-local.ps1`.
274 
275From the oss directory:
276 
277```powershell
278.\run-local.ps1
279```
280 
281Useful options:
282 
283```powershell
284.\run-local.ps1 -CgNotice C:\path\to\ThirdPartyNotices.generated.txt
285.\run-local.ps1 -SkipScan
286.\run-local.ps1 -RepoRoot C:\src\vscode.worktrees\oss-cg-validation -OutDir C:\path\to\oss-local
287```
288 
289What it does:
290 
2911. Runs `scan-licenses.ts` through `npx --yes tsx`.
2922. Writes `ext-notices.txt` and `ext-notices.txt.presence.json` under `-OutDir`.
2933. Runs `merge-notices.ts` with `cglicenses.json`.
2944. Writes `ThirdPartyNotices.new.txt` and a focused summary.
295 
296Notes:
297 
298- Pass `-CgNotice` with a cached CG NOTICE for realistic local validation. Without it, the merge still exercises override injection and stale detection, but it does not represent the full CG base.
299- If root `node_modules` is missing, run `npm i` first or the presence index will be incomplete.
300- Full validation is parity against the legacy OSS tool's `ThirdPartyNotices.txt`: the new output should be at least as good as legacy. Use `parse-notices.ts --diff` to compare NOTICE files and inspect package/version differences.
301 
302## Unit checks
303 
304Run from `build/azure-pipelines/oss`:
305 
306```powershell
307npx tsx cargo-section4.test.ts
308npx tsx platform-binary.test.ts
309```
310 
311Optional live Cargo check:
312 
313```powershell
314npx tsx cargo-section4.live.ts
315```
316 
317The live check performs bounded network calls to crates.io and GitHub. It is useful for validating the real Section 4 fetch path without running a full scanner pass over every crate.
318 
319## Maintenance rules
320 
321- Keep `cglicenses.json` human-authored. Never synthesize license or copyright text by script, model, or other automation.
322- Add `cglicenses.json` entries only for genuine no-usable-text gaps. If CG returns wrong raw text, file an upstream bug and ship CG's result as-is rather than replacing it with a hand-authored body.
323- Preserve the `name@version` merge key behavior. Multiple shipped versions of a package must remain separate NOTICE entries.
324- Treat stale overrides as warn-only in the release pipeline. The PR-time gate is the enforcement point for dependency additions/removals.
325- When updating shipped platform or arch support, update `VSCODE_SHIPPED_PLATFORMS` and `VSCODE_SHIPPED_ARCHS` in `scan-licenses.ts`, then update `platform-binary.test.ts` expectations.
326- When changing Cargo license logic, update `cargo-section4.test.ts`, and run `cargo-section4.live.ts` if the network fetch path changed.
327- Scanner network failures should log and continue. A service outage should not crash the build unless a later explicit validation step chooses to fail.
328 
329## Pending
330 
331The scanner does not yet consume the bundled `ThirdPartyNotices` files from externally-built, pre-built built-in extensions downloaded into the product. The planned addition under investigation is to read those notices for `js-debug`, `js-debug-companion`, and `js-profile-table`, so their own bundled dependencies are covered even though those extensions are built outside this repo.
332 
333### `audit-notices.ts`
334 
335Purpose: pre-ship sanity check that validates a generated NOTICE file against the repo's declared dependencies.
336 
337Invocation:
338 
339```sh
340npx tsx build/azure-pipelines/oss/audit-notices.ts \
341  --notice &lt;path-to-ThirdPartyNotices.new.txt&gt; \
342  --repo .
343```
344 
345What it checks:
346 
3471. **NOTICE stats** — total entries, duplicate detection (same name@version = bug), multi-version packages (expected), license type breakdown.
3482. **Repo manifest cross-reference** — walks all `package.json` (direct deps), `package-lock.json` (full transitive tree, dev filtered out), `Cargo.lock`, and `cgmanifest.json`. Reports overlap, NOTICE-only entries (CG transitive deps), and manifest-only gaps (packages missing from the NOTICE).
3493. **Summary** — coverage percentage and actionable gap list.
3504. **Package source breakdown** — per-lockfile package counts sorted by size. Shows which extensions/directories contribute the most dependencies (e.g., `extensions/copilot: 329 packages`).
351 
352Key behavior:
353 
354- Lockfile parsing filters out `dev: true` and `devOptional: true` entries — only non-dev (shipping) packages are counted.
355- Cross-reference uses lowercase package names for matching.
356- The "manifest-only" gap list is the primary ship-readiness signal: each gap should be explainable (binary component, known TODO, platform-specific absence).
357 
358When to use:
359 
360- Before opening a PR: download `notice_output` from the latest CI build and run locally.
361- After a build: verify the .new.txt artifact covers all declared dependencies.
362- The script does NOT require `node_modules` on disk — it reads lockfiles as the ground truth for the full transitive dependency tree.
363