Cursor rule

1---
2description: Buildkite CI pipeline structure, API access, and known patterns
3globs: .buildkite/**
4---
5 
6# ml-cpp Buildkite CI
7 
8## Pipeline Structure
9 
10| Pipeline Slug | Trigger | Generator Script |
11|---|---|---|
12| `ml-cpp-pr-builds` | PR commits/comments | `.buildkite/pipeline.json.py` |
13| `ml-cpp-snapshot-builds` | Nightly on main | `.buildkite/branch.json.py` |
14| `ml-cpp-debug-build` | Nightly on main | `.buildkite/job-build-test-all-debug.json.py` |
15 
16## Build Step Keys
17 
18| Step Key | Platform |
19|---|---|
20| `build_test_linux-x86_64-RelWithDebInfo` | Linux x86_64 (Docker, k8s) |
21| `build_test_linux-aarch64-RelWithDebInfo` | Linux aarch64 (Docker, AWS) |
22| `build_test_macos-aarch64-RelWithDebInfo` | macOS aarch64 (Orka, 4 cores) |
23| `build_test_Windows-x86_64-RelWithDebInfo` | Windows x86_64 (GCP, 16 vCPU) |
24 
25## CI Hardware
26 
27| Platform | Agent | CPUs | RAM |
28|---|---|---|---|
29| Linux x86_64 | k8s (Docker) | 6 vCPU | 32GB |
30| Linux aarch64 | AWS (Docker) | 8 vCPU | 32GB |
31| macOS aarch64 | Orka VM | 4 cores | 8GB |
32| Windows x86_64 | GCP c2-standard-16 | 16 vCPU | 64GB |
33 
34## API Access
35 
36- Token env var: `BUILDKITE_TOKEN` (local) or `BUILDKITE_API_READ_TOKEN` (CI)
37- Base URL: `https://api.buildkite.com/v2/organizations/elastic/`
38- Common queries:
39  - Builds: `pipelines/{slug}/builds?per_page=N`
40  - Single build: `pipelines/{slug}/builds/{number}`
41  - Job log: use `raw_log_url` from job object (not `url` which may be empty)
42 
43## Vault Secrets (post-checkout hook)
44 
45| Secret Path | Used By |
46|---|---|
47| `secret/ci/elastic-ml-cpp/buildkite/api_read_token` | ingest/check/analyze steps |
48| `secret/ci/elastic-ml-cpp/elasticsearch/ci_analytics` | ingest/check steps |
49| `secret/ci/elastic-ml-cpp/anthropic/claude` | analyze_build_failure step |
50| `secret/ci/elastic-ml-cpp/slack/build_failure_webhook` | analyze_build_failure step |
51| `secret/ci/elastic-ml-cpp/sccache/gcs_service_account` | all build_test steps |
52 
53## Conditional / Diagnostic Steps
54 
55- `if: "build.state == 'failed'"` — only run on failure
56- `allow_dependency_failure: true` — run even if dependencies failed
57- `soft_fail: true` — don't affect overall build status
58- These three combined are the pattern for diagnostic steps
59 
60## Known Failure Patterns
61 
62| Pattern | Classification | Fix |
63|---|---|---|
64| `Unable to download toolchain` / Adoptium JDK | infrastructure/transient | Retry |
65| `Exceeded maximum artifact size limit of 10 GiB` | configuration | Narrow artifact_paths glob |
66| `sccache: error: couldn't connect to server` | infrastructure | Check setup_sccache.sh |
67| `CKMostCorrelatedTest/testScale` timeout | test failure | PR #2910 fixes with CPU time |
68| `CMultiFileDataAdderTest` collision | test failure | Use process ID for temp dirs |
69| `redefinition of` in unity builds | code bug | SKIP_UNITY_BUILD_INCLUSION |
70