Copilot instructions

1---
2applyTo: "**/slo/**/*.md,**/runbooks/**/*.md,**/*.slo.yml,**/monitoring/**"
3---
4 
5# SRE Practice Standards
6 
7## SLI/SLO Definition
8 
9- Define SLIs that directly measure user experience — not internal implementation metrics
10- Every production service must have at minimum: availability SLO and latency SLO
11- Express SLOs as percentages over a rolling 30-day window
12- Measure SLIs from the user's perspective (ALB access logs, synthetic probes) — not internal health checks
13 
14## Error Budgets
15 
16- Calculate error budget as `(1 - SLO_target) × window_duration`
17- Define an error budget policy with three states: healthy (>50%), at-risk (25–50%), exhausted (<25%)
18- Implement fast-burn and slow-burn alerts:
19  - Fast burn: consuming 5% of monthly budget in 1 hour → P2 alert
20  - Slow burn: consuming 10% of monthly budget in 6 hours → P3 alert
21- Error budget exhaustion triggers feature freeze — reliability work takes priority
22 
23## Monitoring Requirements
24 
25- Every service must emit the four golden signals: latency, traffic, errors, saturation
26- P99 latency is the minimum latency SLI — P50 alone is insufficient
27- All CloudWatch alarms must have a linked runbook in the alarm description
28- Set `TreatMissingData: BREACHING` on availability alarms — missing data is not healthy
29 
30## Toil Management
31 
32- Define toil as: manual, repetitive, reactive, scales with service growth, provides no enduring value
33- Track toil time per week per engineer — target <50% of working time on toil
34- Automate toil with Lambda, Step Functions, or shell automation within one quarter of identification
35- Measure toil elimination: record time saved per week after automation
36 
37## Reliability Patterns
38 
39- Implement retry with exponential backoff and jitter for all external calls
40- Use circuit breakers for dependencies with >1% error rate sustained over 5 minutes
41- Implement timeouts on all outbound calls — no blocking call without a deadline
42- Design for graceful degradation: define what the service does when each dependency fails
43 
44## On-Call
45 
46- Every P1/P2 alert must be actionable — if an alert fires and there is no action to take, remove or tune it
47- On-call runbooks must be executable in under 15 minutes by any on-call engineer
48- Conduct game day exercises quarterly to test incident response procedures
49- Measure and improve MTTR and MTTD as team-level SRE KPIs
50