Cursor rule

1---
2description: CI build timing analytics, Elasticsearch integration, and AI failure analysis
3globs: dev-tools/ingest_build_timings.py, dev-tools/check_build_regression.py, dev-tools/analyze_build_failure.py
4---
5 
6# ml-cpp CI Analytics & Observability
7 
8## Elasticsearch Serverless Cluster
9 
10- Project: Elastic Cloud Serverless (Sydney region)
11- Index: `buildkite-build-timings`
12- Credentials in Vault: `secret/ci/elastic-ml-cpp/elasticsearch/ci_analytics` (fields: `endpoint`, `api_key`)
13- Local credential files: `~/.elastic/serverless-endpoint`, `~/.elastic/serverless-api-key`
14 
15## Build Timing Ingestion
16 
17`dev-tools/ingest_build_timings.py`:
18- Fetches step-level timings from Buildkite API
19- Computes `duration_seconds` and `agent_wait_seconds` per step
20- Infers `platform` and `step_type` from step keys
21- Indexes into `buildkite-build-timings` via bulk API
22- Runs as a post-build step in nightly pipelines
23 
24## PR Regression Check
25 
26`dev-tools/check_build_regression.py`:
27- Queries ES for 30-day rolling mean + stddev of step durations
28- Compares current PR build against `mean + 2*stddev` threshold
29- Posts Buildkite annotation with `--annotate` flag
30- Runs as soft-fail step in PR pipelines
31 
32## AI Failure Analysis
33 
34`dev-tools/analyze_build_failure.py`:
35- Fetches failed step logs from Buildkite API
36- Sends to Claude (Sonnet) with repo context and known failure patterns
37- Posts structured annotation: root cause, classification, suggested fix, confidence
38- Optionally posts to Slack via `SLACK_WEBHOOK_URL`
39- Only runs when build is failing (`if: "build.state == 'failed'"`)
40- Claude API key in Vault: `secret/ci/elastic-ml-cpp/anthropic/claude`
41 
42## ML Anomaly Detection
43 
44- Job: `build-timing-regressions` on the serverless cluster
45- Detector: `high_mean(duration_seconds)` partitioned by `step_key`
46- Bucket span: 12h
47- Processes nightly build data to detect step-level duration anomalies
48 
49## Kibana Dashboard
50 
51- Data view: `Buildkite Build Timings`
52- Dashboard: `ML-CPP Build Timing Overview` (6 panels)
53- Accessible via the Elastic Cloud Serverless Kibana UI
54 
55## Credential Flow
56 
57```
58Vault → post-checkout hook → env vars → Python scripts → ES/Buildkite/Claude APIs
59```
60