AGENTS.md

1# CLAUDE.md
2 
3This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4 
5## Project Overview
6 
7**AI Research Skills Library** - A comprehensive open-source library of 90 AI research skills enabling AI agents to autonomously conduct AI research — from idea to paper. Each skill provides expert-level guidance (200-500 lines) with real code examples, troubleshooting guides, and production-ready workflows.
8 
9**Mission**: Enable AI agents to autonomously conduct AI research from hypothesis to experimental verification, covering the full lifecycle: literature survey, ideation, dataset preparation, training pipelines, model deployment, evaluation, and paper writing.
10 
11## Repository Architecture
12 
13### Directory Structure (90 Skills Across 23 Categories)
14 
15Skills are organized into numbered categories representing the AI research lifecycle:
16 
17- `0-autoresearch-skill/` - **Autonomous research orchestration** (1 skill: Autoresearch — central layer that manages the full lifecycle and routes to all other skills)
18- `01-model-architecture/` - Model architectures (5 skills: Megatron-Core, LitGPT, Mamba, RWKV, NanoGPT)
19- `02-tokenization/` - Tokenizers (2 skills: HuggingFace Tokenizers, SentencePiece)
20- `03-fine-tuning/` - Fine-tuning frameworks (4 skills: Axolotl, LLaMA-Factory, Unsloth, PEFT)
21- `04-mechanistic-interpretability/` - Interpretability tools (4 skills: TransformerLens, SAELens, NNsight, Pyvene)
22- `05-data-processing/` - Data curation (2 skills: Ray Data, NeMo Curator)
23- `06-post-training/` - RLHF/DPO/GRPO (8 skills: TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge)
24- `07-safety-alignment/` - Safety and guardrails (4 skills: Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard)
25- `08-distributed-training/` - Distributed systems (6 skills: Megatron-Core, DeepSpeed, FSDP, Accelerate, PyTorch Lightning, Ray Train)
26- `09-infrastructure/` - Cloud compute (3 skills: Modal, SkyPilot, Lambda Labs)
27- `10-optimization/` - Optimization techniques (6 skills: Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF)
28- `11-evaluation/` - Benchmarking (3 skills: lm-evaluation-harness, BigCode, NeMo Evaluator)
29- `12-inference-serving/` - Inference engines (4 skills: vLLM, TensorRT-LLM, llama.cpp, SGLang)
30- `13-mlops/` - Experiment tracking (3 skills: Weights & Biases, MLflow, TensorBoard)
31- `14-agents/` - Agent frameworks (4 skills: LangChain, LlamaIndex, CrewAI, AutoGPT)
32- `15-rag/` - Retrieval-augmented generation (5 skills: Chroma, FAISS, Sentence Transformers, Pinecone, Qdrant)
33- `16-prompt-engineering/` - Structured output (4 skills: DSPy, Instructor, Guidance, Outlines)
34- `17-observability/` - LLM observability (2 skills: LangSmith, Phoenix)
35- `18-multimodal/` - Vision and speech (7 skills: CLIP, Whisper, LLaVA, Stable Diffusion, SAM, BLIP-2, AudioCraft)
36- `19-emerging-techniques/` - Advanced methods (6 skills: MoE Training, Model Merging, Long Context, Speculative Decoding, Knowledge Distillation, Model Pruning)
37- `20-ml-paper-writing/` - Paper writing (1 skill: ML Paper Writing with LaTeX templates for NeurIPS, ICML, ICLR, ACL, AAAI, COLM)
38- `21-research-ideation/` - Ideation (2 skills: Research Brainstorming, Creative Thinking)
39- `22-agent-native-research-artifact/` - Agent-Native Research Artifact tooling (3 skills: ARA Compiler, ARA Research Manager, ARA Rigor Reviewer — ingestion, post-task provenance recording, and Seal Level 2 epistemic review)
40 
41### Skill File Structure
42 
43Each skill follows a standardized format:
44```
45skill-name/
46├── SKILL.md                    # Main guidance (200-600 lines with YAML frontmatter)
47├── references/                 # Deep documentation (300KB+ target)
48│   ├── README.md              # From official docs
49│   ├── api.md                 # API reference
50│   ├── tutorials.md           # Step-by-step guides
51│   ├── issues.md              # Real GitHub issues & solutions
52│   └── releases.md            # Version history
53├── scripts/                    # Helper scripts (optional)
54├── templates/                  # Code templates (optional)
55└── examples/                   # Example implementations (optional)
56```
57 
58## Skill Quality Standards
59 
60### YAML Frontmatter Requirements (CRITICAL)
61 
62All `SKILL.md` files MUST include YAML frontmatter with these exact fields:
63 
64```yaml
65---
66name: skill-name-here              # kebab-case, no quotes, gerund form preferred
67description: Third-person description of what AND when to use this skill  # No quotes, max 1024 chars
68version: 1.0.0                     # Semantic versioning
69author: Orchestra Research         # Standard author
70license: MIT                       # Standard license
71tags: [Tag One, Tag Two]          # Title Case (except UPPERCASE acronyms like GRPO, TRL, RLHF)
72dependencies: [pkg>=1.0.0]         # Optional, with version constraints
73---
74```
75 
76**Critical Rules**:
77- `name`: Use gerund form (e.g., `serving-llms`, `processing-data`, `grpo-rl-training`)
78- `description`: Third person ("Provides guidance for..."), include WHAT it does AND WHEN to use it
79- `tags`: Title Case for regular words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO)
80- No quotes around any field values (except in arrays)
81- Dependencies should include version constraints: `transformers>=4.47.0`
82 
83### Content Quality Standards
84 
85**Core Requirements** (based on Anthropic official best practices):
86- ✅ SKILL.md body: **200-500 lines** (under 500 lines is critical for performance)
87- ✅ Progressive disclosure: SKILL.md as overview, details in separate reference files
88- ✅ Workflows with copy-paste checklists for complex tasks
89- ✅ "When to use vs alternatives" guidance section
90- ✅ Common issues section with solutions
91- ✅ Concise content: assume Claude is smart, no over-explaining basics
92- ✅ Code examples with language detection (```python, ```bash, etc.)
93- ✅ References ONE level deep from SKILL.md (no nested references)
94 
95**Gold Standard** (aim for this - see `06-post-training/grpo-rl-training/`):
96- ✅ 2-3 complete workflows with step-by-step checklists
97- ✅ Reference files for advanced topics (one level deep)
98- ✅ Feedback loops (validate → fix → repeat) for quality-critical operations
99- ✅ Consistent terminology throughout
100- ✅ Concrete input/output examples
101- ✅ Real GitHub issues with solutions (when available)
102 
103**NOT Acceptable**:
104- ❌ SKILL.md over 500 lines (split into reference files instead)
105- ❌ Over-explaining basics that Claude already knows
106- ❌ First-person descriptions ("I can help you...")
107- ❌ Vague skill names ("helper", "utils", "tools")
108- ❌ Nested references (SKILL.md → ref1.md → ref2.md)
109- ❌ Missing workflows with checklists for complex tasks
110 
111## Development Workflow
112 
113### Adding a New Skill
114 
1151. **Choose skill from roadmap** (see CONTRIBUTING.md or README.md)
1162. **Create directory structure** in appropriate category (01-19)
1173. **Write SKILL.md** with YAML frontmatter following standards above
1184. **Add reference documentation** (target 300KB+ from official sources)
1195. **Validate quality**:
120   - Check SKILL.md has YAML frontmatter
121   - Verify SKILL.md is 200-500 lines
122   - Ensure code blocks have language tags
123   - Confirm references are one level deep from SKILL.md
124   - Check documentation size: `du -sh skill-name/references/`
1256. **Test the skill** with real use cases before submitting
126 
127### Improving Existing Skills
128 
129When updating skills:
1301. **Maintain YAML frontmatter** format and fields
1312. **Keep SKILL.md under 500 lines** - split into reference files if needed
1323. **Add workflows** with checklists for complex operations
1334. **Update version number** in YAML frontmatter
1345. **Test changes** with representative tasks
135 
136### Quality Validation Commands
137 
138```bash
139# Check YAML frontmatter exists
140head -20 skill-name/SKILL.md
141 
142# Verify SKILL.md line count (target 200-500 lines)
143wc -l skill-name/SKILL.md
144 
145# Check documentation size (target 300KB+)
146du -sh skill-name/references/
147 
148# Verify code blocks have language tags
149grep -A 1 '```' skill-name/SKILL.md | head -20
150 
151# Validate YAML frontmatter syntax
152python -c "import yaml; yaml.safe_load(open('skill-name/SKILL.md').read().split('---')[1])"
153```
154 
155## Key Files
156 
157- **README.md** - Project overview, all 90 skills listed with descriptions and stats
158- **CONTRIBUTING.md** - Complete contribution guidelines and quality standards
159- **SKILL_TEMPLATE.md** - Copy-paste scaffold for new skills
160- **ROADMAP.md** - Development roadmap (90 skills achieved)
161- **anthropic_official_docs/** - Anthropic's official best practices for skills
162 
163## Git Workflow
164 
165Standard Git workflow:
166```bash
167# Create feature branch
168git checkout -b add-skill-name
169 
170# Add and commit changes
171git add category/skill-name/
172git commit -m "Add [Skill Name] skill
173 
174- X lines of documentation
175- Y GitHub issues with solutions
176- API reference and examples included"
177 
178# Push to fork and create PR
179git push origin add-skill-name
180```
181 
182## Automation: Orchestra Skill Marketplace Sync
183 
184### How Auto-Sync Works
185 
186When skills are committed to the `main` branch, GitHub Actions automatically syncs them to the Orchestra skill marketplace:
187 
1881. **GitHub Actions detects** changed skill folders on push to `main`
1892. **For each changed skill**:
190   - Extracts metadata from SKILL.md frontmatter (`name`, `author`, etc.)
191   - Creates ZIP file containing entire skill directory (SKILL.md, references/, scripts/, etc.)
192   - Uploads to Orchestra API endpoint
1933. **Orchestra stores** ZIP in Supabase Storage and creates database record
1944. **Skill appears** in marketplace at `https://orchestra.com/research-skills`
195 
196### Workflow File Location
197 
198- **File**: `.github/workflows/sync-skills.yml`
199- **Triggers**: Push to `main` branch, manual workflow dispatch
200- **What syncs**: Only skill directories that changed in the commit
201 
202### Author Detection (Orchestra vs Community)
203 
204The workflow reads the `author:` field from SKILL.md frontmatter to determine badge:
205 
206**Official Orchestra Skill**:
207```yaml
208---
209author: Orchestra Research  # Contains "Orchestra"
210---
211```
212- Result: Source = `orchestra` (Official badge)
213- Storage: `research-skills/orchestra/skill-name.zip`
214 
215**Community Skill**:
216```yaml
217---
218author: Jane Doe  # Does NOT contain "Orchestra"
219---
220```
221- Result: Source = `community` (Community badge)
222- Storage: `research-skills/community/skill-name.zip`
223 
224### What Gets Synced
225 
226The workflow zips **ALL contents** of skill directory:
227- ✅ SKILL.md
228- ✅ references/ (all subdirectories)
229- ✅ scripts/ (if exists)
230- ✅ assets/ (if exists)
231- ✅ examples/ (if exists)
232- ✅ templates/ (if exists)
233- ❌ Hidden files (`.gitkeep`, `.DS_Store`)
234 
235### Testing the Sync
236 
237**Manual trigger**:
2381. Go to GitHub Actions tab
2392. Select "Sync Skills to Orchestra" workflow
2403. Click "Run workflow"
241 
242**Test with commit**:
243```bash
244# Make a small change to any skill
245echo "\n<!-- Updated $(date) -->" >> 01-model-architecture/litgpt/SKILL.md
246 
247# Commit and push to main
248git add .
249git commit -m "test: trigger auto-sync"
250git push origin main
251```
252 
253**Verify sync worked**:
2541. Check GitHub Actions tab for workflow run status
2552. Check Orchestra marketplace for updated skill
2563. Check Supabase Storage for ZIP file
257 
258### Important Notes
259 
260- **GitHub Secrets required**: `ORCHESTRA_API_URL`, `ORCHESTRA_SYNC_API_KEY` (already configured)
261- **Only syncs changed skills**: Workflow detects which skill directories changed in commit
262- **SKILL.md required**: Skills without SKILL.md are skipped with warning
263- **See detailed setup**: `dev_data/GITHUB_SKILLS_SYNC_SETUP.md`
264 
265## npm Package Publishing
266 
267### How It Works
268 
269The `publish-npm.yml` workflow auto-publishes to npm when the version in `packages/ai-research-skills/package.json` changes on `main`.
270 
271- **Auth**: Uses OIDC trusted publishing (no npm tokens). Configured on npmjs.com under the package's Trusted Publishers settings.
272- **Provenance**: `--provenance` flag signs packages with Sigstore for supply chain security.
273- **Workflow**: `.github/workflows/publish-npm.yml`
274 
275### Bumping Versions
276 
277**Always use `npm version`** (not manual edits) to keep `package-lock.json` in sync:
278 
279```bash
280cd packages/ai-research-skills
281npm version patch   # 1.3.6 → 1.3.7
282npm version minor   # 1.3.7 → 1.4.0
283npm version major   # 1.4.0 → 2.0.0
284```
285 
286Use `--no-git-tag-version` if you want to commit manually.
287 
288### Common Issues
289 
290- **`npm ci` fails in CI**: `package-lock.json` is out of sync. Run `npm install` locally and commit the lockfile.
291- **OIDC auth fails**: The trusted publisher config on npmjs.com must match the repo exactly (case-sensitive: `Orchestra-Research/AI-Research-SKILLs`, workflow: `publish-npm.yml`).
292- **`NODE_AUTH_TOKEN` blocks OIDC**: `actions/setup-node` with `registry-url` auto-sets this token. The workflow unsets it before publish so OIDC takes over.
293- **Version unchanged skip**: The workflow compares `HEAD` vs `HEAD~1`. If only the lockfile changed (not `package.json` version), publish is skipped. Bump the version to trigger.
294 
295## Important Conventions
296 
297### Naming Conventions
298 
299- **Skill names**: Use gerund form (verb + -ing) in kebab-case: `processing-pdfs`, `serving-llms`, `grpo-rl-training`
300- **Tags**: Title Case for words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO, FSDP, MoE)
301- **Descriptions**: Third person, include what AND when to use
302 
303### Code Examples
304 
305Always use language detection in code blocks:
306```python
307# Good - has language tag
308from transformers import AutoModel
309```
310 
311NOT:
312```
313# Bad - no language tag
314from transformers import AutoModel
315```
316 
317### Progressive Disclosure Pattern
318 
319SKILL.md should link directly to reference files (one level deep):
320 
321```markdown
322## Advanced Features
323 
324**API Reference**: See [references/api.md](references/api.md)
325**Troubleshooting**: See [references/issues.md](references/issues.md)
326```
327 
328## Philosophy
329 
330**Quality over Quantity**: This library maintains high standards by:
331- Requiring 200-500 line SKILL.md files (focused, actionable guidance)
332- Including 300KB+ documentation from official sources
333- Providing real GitHub issues with solutions
334- Following Anthropic's official best practices for skills
335- Testing skills with real use cases before inclusion
336 
337Each skill represents expert-level knowledge distilled into a format optimized for AI agent consumption.
338