AGENTS.md

1# AGENTS.md
2 
3This file provides guidance for AI agents working with the Linguist repository.
4 
5## Project Overview
6 
7Linguist is a Ruby library used on GitHub.com to:
8- Detect blob languages
9- Ignore binary and vendored files
10- Suppress generated files in diffs
11- Generate language breakdown graphs
12 
13It's packaged as the `github-linguist` gem and includes CLI tools: `github-linguist` and `git-linguist`.
14 
15## Key Files and Directories
16 
17| Path | Purpose |
18|------|---------|
19| `lib/linguist/languages.yml` | Master list of all languages, extensions, filenames, and their properties |
20| `lib/linguist/heuristics.yml` | Rules for disambiguating files with shared extensions |
21| `lib/linguist/heuristics.rb` | Ruby code implementing heuristic logic |
22| `lib/linguist/classifier.rb` | Centroid-based classifier for language detection |
23| `samples/` | Sample code files for each language (used by classifier) |
24| `grammars.yml` | Lists all TextMate grammars used for syntax highlighting |
25| `vendor/` | Contains grammar submodules and cached license files for the grammars |
26| `script/` | Utility scripts for maintenance tasks |
27 
28## Development Environment
29 
30Use GitHub Codespaces or the dev container for the easiest setup. The environment includes Ruby, Node.js, Docker, and all required dependencies.
31 
32To bootstrap locally:
33```bash
34script/bootstrap
35```
36 
37To run Linguist from the repo:
38```bash
39bundle exec bin/github-linguist --breakdown
40```
41 
42## Common Tasks
43 
44### Adding a Language Extension
451. Add extension to the language entry in `lib/linguist/languages.yml` (alphabetical order, case-sensitive; primary extension first)
462. Add sample files to `samples/<Language>/`
473. If extension is shared with another language, ensure there are at least two samples and add a heuristic in `lib/linguist/heuristics.yml`
48 
49### Adding a New Language
501. Add entry to `lib/linguist/languages.yml` (omit `language_id` initially)
512. Add grammar: `script/add-grammar <grammar-url>`
523. Add samples to `samples/<Language>/`
534. Generate ID: `script/update-ids`
545. If an extension is shared with another language, ensure there are at least two samples and add a heuristic in `lib/linguist/heuristics.yml`
55 
56### Replacing a Grammar
57```bash
58script/add-grammar --replace &lt;GrammarName&gt; &lt;new-grammar-url&gt;
59```
60 
61### Important Scripts
62- `script/bootstrap` - Install dependencies
63- `script/add-grammar` - Add or replace TextMate grammars
64- `script/update-ids` - Generate unique language IDs
65- `script/cross-validation` - Test the classifier
66- `script/cibuild` - CI build script
67 
68## Testing
69 
70Run tests with:
71```bash
72bundle exec rake test
73```
74 
75Test the classifier:
76```bash
77bundle exec script/cross-validation --test
78```
79 
80Always ensure tests pass before submitting changes. GitHub Actions runs tests on all PRs.
81 
82## Coding Conventions
83 
84- **Ruby style**: Follow existing code patterns in the repository
85- **languages.yml**: Keep alphabetically sorted (case-sensitive, uppercase before lowercase) with the comment at the top. Use the comment at the top to determine the fields to add for a language.
86- **Extensions in languages.yml**: Keep alphabetically sorted (case-sensitive, uppercase before lowercase); primary extension first
87- **Samples**: Use real-world code examples, not "Hello World" and other common examples used in tutorials
88- **Heuristics**: Write patterns to minimize false positives. Patterns must be linear, safe from ReDoS attacks, and RE2 compatible. All heuristics must have tests.
89- **Don't refactor unrelated code.** Keep changes scoped to the task at hand.
90 
91## Generated and script maintained files — do not edit by hand
92 
93- `.gitmodules`
94- `grammars.yml`
95- all files in `vendor/`
96 
97These are all maintained by the `script/add-grammars` script.
98 
99## PR Requirements
100 
101All PRs must:
1021. Use the provided PR template. Irrelevant sections may be removed.
1032. Link to GitHub search results showing in-the-wild usage, excluding forks, for each extension being added (minimum 2000 files for common extensions, 200 for once-per-repo files or extensions).
1043. Link to the original source of any samples added. This must not be a link to the fork of Linguist used for the contribution.
1054. State the license of any sample code added individually.
1065. Pass all CI tests.
107 
108Do not open a PR if any of the above conditions are not met and notify the user why the PR has not been created.
109 
110## Usage Requirements for New Languages/Extensions
111 
112- At least 2000 files indexed on GitHub (or 200 for once-per-repo files like Makefile), excluding forks
113- Reasonable distribution across unique `user/repo` combinations
114- High-proportion users may be filtered out during assessment
115 
116## Grammar Requirements
117 
118Only add grammars with approved licenses (see `vendor/licenses/config.yml` for the list). TextMate-compatible grammars are used for syntax highlighting.
119 
120## Key Dependencies
121 
122- `charlock_holmes` - Character encoding detection (requires ICU)
123- `rugged` - libgit2 bindings for Ruby
124- Docker - Required for grammar operations
125