Copilot instructions

1---
2applyTo: "**/*.ipynb, **/sagemaker/**, **/bedrock/**, **/glue/**, **/etl/**, **/ml/**, **/ai/**, **/data-science/**, **/notebooks/**"
3description: "AWS data, ML, and AI platform standards: SageMaker, Bedrock, Glue ETL, Athena, feature store, RAG pipelines, and responsible AI controls."
4---
5 
6## Context
7 
8This instruction file applies to data science notebooks, ML pipeline code, Glue ETL jobs, and AI/LLM application code on AWS. The AWS data platform follows a Bronze/Silver/Gold lakehouse pattern. All ML models must be version-controlled in SageMaker Model Registry. All LLM integrations must implement output guardrails.
9 
10---
11 
12## Data Lake Architecture (Bronze / Silver / Gold)
13 
14| Layer | S3 Prefix | Format | Purpose | Retention |
15|-------|-----------|--------|---------|----------|
16| **Bronze** | `s3://{bucket}/bronze/{source}/{date}/` | Raw (JSON, CSV, Parquet) | Immutable raw ingestion | 7 years |
17| **Silver** | `s3://{bucket}/silver/{domain}/{date}/` | Parquet, partitioned | Cleaned, validated, normalised | 3 years |
18| **Gold** | `s3://{bucket}/gold/{product}/{date}/` | Parquet, aggregated | Business-ready analytics | 1 year |
19 
20- **Never overwrite Bronze** — Bronze is immutable; always append
21- **Partition by date** (`year=YYYY/month=MM/day=DD`) for time-series data
22- **Register all tables** in Glue Data Catalog with schemas
23- **Apply Lake Formation permissions** — no direct S3 bucket policies for data access
24 
25---
26 
27## Glue ETL Standards
28 
29```python
30import sys
31from awsglue.context import GlueContext
32from awsglue.utils import getResolvedOptions
33from pyspark.context import SparkContext
34 
35args = getResolvedOptions(sys.argv, ['JOB_NAME', 'source_database', 'target_bucket', 'target_prefix'])
36sc = SparkContext()
37glueContext = GlueContext(sc)
38logger = glueContext.get_logger()
39 
40# CORRECT: Parameterised, no hardcoded paths
41source_database = args['source_database']
42target_path = f"s3://{args['target_bucket']}/{args['target_prefix']}"
43```
44 
45- **Parameterise all jobs** — no hardcoded bucket names, table names, or dates
46- **Log record counts** before and after transformations for data quality auditing
47- **Use DynamicFrame → DataFrame → DynamicFrame** pattern for complex transformations
48- **Handle bad records** explicitly — use `ResolveChoice` and rejection sinks
49 
50---
51 
52## SageMaker Standards
53 
54### Notebooks (SageMaker Studio)
55 
56- **Structure notebooks** as: Imports → Data Loading → EDA → Feature Engineering → Model/Export
57- **Parameterise cell inputs** using Papermill parameters for automated execution
58- **Register features** in SageMaker Feature Store for reuse across models
59- **Store all artefacts in S3** — never rely on notebook instance ephemeral storage
60 
61### Training Jobs
62 
63```python
64from sagemaker.estimator import Estimator
65 
66estimator = Estimator(
67    image_uri=training_image_uri,
68    role=sagemaker_role,
69    instance_type='ml.m5.xlarge',
70    instance_count=1,
71    use_spot_instances=True,   # Up to 90% cost saving
72    max_wait=7200,
73    output_path=f's3://{bucket}/models/',
74    hyperparameters={
75        'epochs': 10,
76        'learning-rate': 0.001,
77    },
78)
79```
80 
81- **Use Spot Instances** for training jobs (non-time-critical) — up to 90% cost reduction
82- **Enable SageMaker Experiments** tracking on all training jobs
83- **Version all models** in Model Registry with: training data URI, metrics, approval status
84 
85### Model Deployment
86 
87- **`PendingManualApproval`** for model registry entries — no automatic prod promotion
88- **Enable Model Monitor** on all production endpoints for data drift and model quality
89- **Set auto-scaling** — never fixed-instance production endpoints
90- **A/B test** new model versions with production variant weights before full cutover
91 
92---
93 
94## Amazon Bedrock / LLM Standards
95 
96### Model Invocation
97 
98```python
99import boto3, json
100 
101bedrock = boto3.client('bedrock-runtime', region_name='eu-west-1')
102 
103def invoke_claude(messages: list, max_tokens: int = 1000, system: str = "") -> str:
104    body = {
105        "anthropic_version": "bedrock-2023-05-31",
106        "max_tokens": max_tokens,
107        "system": system,
108        "messages": messages,
109    }
110    response = bedrock.invoke_model(
111        modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
112        body=json.dumps(body),
113    )
114    return json.loads(response['body'].read())['content'][0]['text']
115```
116 
117### Safety Non-Negotiables
118 
119- **Never log full prompts** containing user PII — log token count, latency, model ID only
120- **Always implement Bedrock Guardrails** for customer-facing features
121- **Always set `max_tokens`** — unbounded generation is a cost and safety risk
122- **Always validate structured outputs** against JSON schema before downstream use
123- **Never pass user input directly into system prompts** — sanitise to prevent prompt injection
124 
125### RAG Pipeline Checklist
126 
127- [ ] Chunking strategy documented (chunk size, overlap, splitting method)
128- [ ] Embedding model version pinned (re-index if model changes)
129- [ ] Retrieval evaluated with RAGAS: faithfulness ≥ 0.7, answer relevance ≥ 0.7
130- [ ] Guardrails configured for topic blocking and PII filtering
131- [ ] All LLM calls logged to CloudWatch with token count and latency
132 
133---
134 
135## Responsible AI Controls
136 
137| Control | Implementation |
138|---------|---------------|
139| Bias monitoring | SageMaker Clarify on training data and model predictions |
140| Explainability | SHAP values via SageMaker Clarify |
141| Hallucination detection | Groundedness check via Bedrock Guardrails |
142| PII handling | Amazon Comprehend PII detection before LLM processing |
143| Audit trail | All model invocations logged to CloudWatch + S3 |
144