Copilot instructions

1---
2applyTo: "**/runbooks/**, **/oncall/**, **/incidents/**, **/ops/**, **/sre/**"
3description: "Operational standards: incident severity classification, runbook format, CloudWatch alarm patterns, on-call responsibilities, and post-incident review process."
4---
5 
6## Context
7 
8This instruction file applies to operational documentation: runbooks, incident reports, on-call playbooks, and observability configuration. Operational documents are production artefacts — they must be tested, reviewed, and maintained with the same rigor as code.
9 
10---
11 
12## Incident Severity Matrix
13 
14| Priority | Definition | Response SLA | Escalation SLA |
15|----------|-----------|-------------|---------------|
16| **P1** | Complete service outage, data loss risk, or security breach | 15 min initial response | Immediate: Engineering VP, CTO |
17| **P2** | Significant service degradation, > 10% users impacted, or revenue impact | 30 min initial response | 1 hour if no progress |
18| **P3** | Partial degradation, < 10% users, workaround available | 4 hours | Business hours only |
19| **P4** | Minor issue, single user, cosmetic | Next business day | — |
20 
21---
22 
23## Runbook Format (Required)
24 
25```markdown
26# Runbook: {Descriptive Title}
27 
28## Overview
29**System:** {Service name}
30**Severity:** P1 / P2 / P3
31**Last Reviewed:** {date}
32**Owner:** {Team name}
33 
34## Symptoms
35- {Observable symptom 1 — what the user or monitoring sees}
36- {Observable symptom 2}
37 
38## Diagnosis Steps
39 
401. **Check CloudWatch dashboard**
41   - Navigate to: [{Dashboard Name}]({CloudWatch URL})
42   - Look for: {specific metric anomaly}
43 
442. **Check application logs**
45```
46   filter @message like /ERROR/
47   | stats count(*) by bin(5m)
48```
49 
503. **Check database health**
51   - RDS Console → {instance} → Monitoring
52   - Alert if: CPU > 85%, free storage < 20%, connections > 80% of max
53 
54## Resolution Steps
55 
56### Scenario A: {Most common cause}
571. {Specific action}
582. {Verify resolution: what metric or log confirms it's fixed}
59 
60### Scenario B: {Second most common cause}
611. {Specific action}
62 
63## Rollback Procedure
641. {Step to revert the last deployment}
652. {Verification step}
66 
67## Escalation Path
68If not resolved within {N} minutes:
69- {Name/Role} — {contact method}
70- {Name/Role} — {contact method}
71 
72## Related Resources
73- [Architecture Diagram]({link})
74- [Previous Incident Reports]({link})
75```
76 
77---
78 
79## CloudWatch Alarm Conventions
80 
81### Alarm Naming
82 
83`{Project}-{Environment}-{Service}-{Metric}-{Severity}`
84 
85Example: `smart-retail-prod-order-api-5xx-rate-p1`
86 
87### Standard Alarm Thresholds
88 
89| Metric | Namespace | Warning | Critical | Period |
90|--------|----------|---------|----------|--------|
91| ALB 5xx error rate | AWS/ApplicationELB | > 1% | > 5% | 5 min |
92| ECS CPU utilisation | AWS/ECS | > 70% | > 85% | 5 min |
93| ECS Memory utilisation | AWS/ECS | > 75% | > 90% | 5 min |
94| RDS CPU | AWS/RDS | > 70% | > 85% | 5 min |
95| Lambda error rate | AWS/Lambda | > 1% | > 5% | 5 min |
96| SQS DLQ messages | AWS/SQS | > 0 | > 10 | 1 min |
97| API Gateway 4xx rate | AWS/ApiGateway | > 5% | > 20% | 5 min |
98 
99### CloudWatch Logs Insights — Standard Queries
100 
101```sql
102-- Error rate over time
103filter @message like /ERROR/
104| stats count(*) as errorCount by bin(5m)
105| sort @timestamp desc
106 
107-- Slow requests (> 2s)
108filter @message like /took/
109| parse @message "took *ms" as duration
110| filter duration > 2000
111| stats count(*), avg(duration) as avgDuration by bin(5m)
112 
113-- Exception summary
114filter @message like /Exception/
115| parse @message "* Exception" as exceptionType
116| stats count(*) as frequency by exceptionType
117| sort frequency desc
118```
119 
120---
121 
122## Post-Incident Review Process
123 
124| Step | Owner | Timeline |
125|------|-------|----------|
126| Draft RCA document | On-call engineer | Within 24h of resolution |
127| Circulate draft | Incident Commander | Within 48h |
128| Review meeting | All responders | Within 72h |
129| Corrective actions assigned | Tech Lead / Team Lead | At review meeting |
130| Corrective actions tracked | Project Tracker | Sprint backlog |
131| RCA published | Engineering Manager | Within 1 week |
132 
133Post-incident reviews are blameless. The goal is system improvement, not accountability assignment.
134 
135---
136 
137## On-Call Responsibilities
138 
139- Acknowledge PagerDuty alerts within SLA (P1: 15 min, P2: 30 min)
140- Post incident declaration in Slack `#incidents` channel within 5 minutes of declaring P1/P2
141- Update incident status every 30 minutes for P1, every 60 minutes for P2
142- Hand off cleanly at shift end: summarise active incidents, investigations, and outstanding actions
143- Complete runbook after resolving an incident if the runbook was missing or incorrect
144