Two files, one repository
huggingface/transformers ships 2 formats across 2 indexed files. The question worth asking is whether the second one says anything the first does not.
| Dimension | Shared | Only in A | Only in B | Overlap |
|---|---|---|---|---|
| Sections | 0 | 7 | 5 | 0% |
| Commands | 0 | 9 | 5 | 0% |
| Section tags | 3 | 2 | 2 | 43% |
What each file covers
Sections
0 shared · 7 only in A · 5 only in B- − Useful commands
- − Local agent setup
- − Coordination before coding
- − Mandatory duplicate-work checks
- − No low-value busywork PRs
- − Accountability for AI-assisted patches
- − Copies and Modular Models
- + copilot-instructions.md Guide for Hugging Face Transformers
- + Core Project Structure
- + Coding Conventions for Hugging Face Transformers
- + Copying and inheritance
- + Testing
Commands
0 shared · 9 only in A · 5 only in B- − gh issue view <issue_number> --repo huggingface/transformers --comments
- − gh pr list --repo huggingface/transformers --state open --search "<issue_number> in:body"
- − gh pr list --repo huggingface/transformers --state open --search "<short area keywords>"
- − make style
- − make typing
- − make fix-repo
- − make check-repo
- − make codex
- − make claude
- + pip install -e .[quality]
- + make fixup
- + pytest tests/models/[name]/test_modeling_[name].py
- + pip install -e .[testing]
- + pip install torch accelerate
Section tags
3 shared · 2 only in A · 2 only in B- − lint-format
- − git-pr
- + code-style
- + architecture
- setup
- test
- agent-behaviour
Line diff
huggingface/transformers · .ai/AGENTS.md
@@ −1 @@
1## Useful commands
2- `make style`: runs formatters and linters (ruff), necessary to pass code style checks
3- `make typing`: runs the ty type checker and model structure rules
4- `make fix-repo`: auto-fixes copies, modular conversions, doc TOCs, docstrings in addition to the `make style` fixes
5- `make check-repo` — runs `make typing` and consistency checks.
6- Many tests are marked as 'slow' and skipped by default in the CI. To run them, use: `RUN_SLOW=1 pytest ...`
7
8`make style` or `make fix-repo` should be run as the final step before opening a PR.
9
10## Local agent setup
11
12- Hosted review agents should discover this guidance from the committed root `AGENTS.md` / `CLAUDE.md` files.
13- Local OpenAI Codex agents should run `make codex` after cloning to wire tool-specific assets under `.agents/`.
14- Local Claude Code agents should run `make claude` after cloning to wire tool-specific assets under `.claude/`.
15
16
17### Coordination before coding
18- If work maps to an existing issue, coordinate on that issue before opening a PR.
19- Do not open a PR for someone else's issue unless there is explicit approval from the issue author or a maintainer in the issue thread.
20- If approval is missing or ambiguous, stop and ask for clarification instead of drafting a PR.
21- Do not start duplicate work on issues.
22
23### Mandatory duplicate-work checks
24Before proposing a PR, check for overlapping open PRs and issue ownership:
25
26```bash
27gh issue view <issue_number> --repo huggingface/transformers --comments
28gh pr list --repo huggingface/transformers --state open --search "<issue_number> in:body"
29gh pr list --repo huggingface/transformers --state open --search "<short area keywords>"
30```
31
32- If an open PR already addresses the same fix, do not open another.
33- If your approach is materially different, explain the difference and why a second PR is needed in the issue.
34
35### No low-value busywork PRs
36- Do not open one-off PRs for tiny edits (single typo, isolated lint cleanup, one mutable default argument, etc.).
37- Mechanical cleanups are acceptable but not as first contributions.
38
39### Accountability for AI-assisted patches
40- First-time contributors should not use code agents to submit PRs or issues. Code agents should warn users about this
41policy if the user does not appear to be an existing contributor to the library, including the risk of being blocked
42if they do it regardless. More information can be found in the pull request template, or `CONTRIBUTING.md`.
43
44## Copies and Modular Models
45
46We try to avoid direct inheritance between model-specific files in `src/transformers/models/`. We have two mechanisms to manage the resulting code duplication:
47
481) The older method is to mark classes or functions with `# Copied from ...`. Copies are kept in sync by `make fix-repo`. Do not edit a `# Copied from` block, as it will be reverted by `make fix-repo`. Ideally you should edit the code it's copying from and propagate the change, but you can break the `# Copied from` link if needed.
492) The newer method is to add a file named `modular_<name>.py` in the model directory. `modular` files **can** inherit from other models. `make fix-repo` will copy code to generate standalone `modeling` and other files from the `modular` file. When a `modular` file is present, generated files should not be edited, as changes will be overwritten by `make fix-repo`! Instead, edit the `modular` file. See [docs/source/en/modular_transformers.md](../docs/source/en/modular_transformers.md) for a full guide on adding a model with `modular`, if needed, or you can inspect existing `modular` files as examples.
50
huggingface/transformers · .github/copilot-instructions.md
@@ +1 @@
1# copilot-instructions.md Guide for Hugging Face Transformers
2
3This copilot-instructions.md file provides guidance for code agents working with this codebase.
4
5## Core Project Structure
6
7- `/src/transformers`: This contains the core source code for the library
8 - `/models`: Code for individual models. Models inherit from base classes in the root `/src/transformers` directory.
9- `/tests`: This contains the core test classes for the library. These are usually inherited rather than directly run.
10 - `/models`: Tests for individual models. Model tests inherit from common tests in the root `/tests` directory.
11- `/docs`: This contains the documentation for the library, including guides, tutorials, and API references.
12
13## Coding Conventions for Hugging Face Transformers
14
15- PRs should be as brief as possible. Bugfix PRs in particular can often be only one or two lines long, and do not need large comments, docstrings or new functions in this case. Aim to minimize the size of the diff.
16- When writing tests, they should be added to an existing file. The only exception is for PRs to add a new model, when a new test directory should be created for that model.
17- Code style is enforced in the CI. You can install the style tools with `pip install -e .[quality]`. You can then run `make fixup` to apply style and consistency fixes to your code.
18
19## Copying and inheritance
20
21Many models in the codebase have similar code, but it is not shared by inheritance because we want each model file to be self-contained.
22We use two mechanisms to keep this code in sync:
23
24- "Copied from" syntax. Functions or entire classes can have a comment at the top like this: `# Copied from transformers.models.llama.modeling_llama.rotate_half` or `# Copied from transformers.models.t5.modeling_t5.T5LayerNorm with T5->MT5`
25 These comments are actively checked by the style tools, and copies will automatically be updated when the base code is updated. If you need to update a copied function, you should
26 either update the base function and use `make fixup` to propagate the change to all copies, or simply remove the `# Copied from` comment if that is inappropriate.
27- "Modular" files. These files briefly define models by composing them using inheritance from other models. They are not meant to be used directly. Instead, the style tools
28 automatically generate a complete modeling file, like `modeling_bert.py`, from the modular file like `modular_bert.py`. If a model has a modular file, the modeling file
29 should never be edited directly! Instead, changes should be made in the modular file, and then you should run `make fixup` to update the modeling file automatically.
30
31When adding new models, you should prefer `modular` style and inherit as many classes as possible from existing models.
32
33## Testing
34
35After making changes, you should usually run `make fixup` to ensure any copies and modular files are updated, and then test all affected models. This includes both
36the model you made the changes in and any other models that were updated by `make fixup`. Tests can be run with `pytest tests/models/[name]/test_modeling_[name].py`
37If your changes affect code in other classes like tokenizers or processors, you should run those tests instead, like `test_processing_[name].py` or `test_tokenization_[name].py`.
38
39In order to run tests, you may need to install dependencies. You can do this with `pip install -e .[testing]`. You will probably also need to `pip install torch accelerate` if your environment does not already have them.
40
@@ −1 +1 @@
1−## Useful commands
2−- `make style`: runs formatters and linters (ruff), necessary to pass code style checks
3−- `make typing`: runs the ty type checker and model structure rules
4−- `make fix-repo`: auto-fixes copies, modular conversions, doc TOCs, docstrings in addition to the `make style` fixes
5−- `make check-repo` — runs `make typing` and consistency checks.
6−- Many tests are marked as 'slow' and skipped by default in the CI. To run them, use: `RUN_SLOW=1 pytest ...`
1+# copilot-instructions.md Guide for Hugging Face Transformers
72
8−`make style` or `make fix-repo` should be run as the final step before opening a PR.
3+This copilot-instructions.md file provides guidance for code agents working with this codebase.
94
10−## Local agent setup
5+## Core Project Structure
116
12−- Hosted review agents should discover this guidance from the committed root `AGENTS.md` / `CLAUDE.md` files.
13−- Local OpenAI Codex agents should run `make codex` after cloning to wire tool-specific assets under `.agents/`.
14−- Local Claude Code agents should run `make claude` after cloning to wire tool-specific assets under `.claude/`.
7+- `/src/transformers`: This contains the core source code for the library
8+ - `/models`: Code for individual models. Models inherit from base classes in the root `/src/transformers` directory.
9+- `/tests`: This contains the core test classes for the library. These are usually inherited rather than directly run.
10+ - `/models`: Tests for individual models. Model tests inherit from common tests in the root `/tests` directory.
11+- `/docs`: This contains the documentation for the library, including guides, tutorials, and API references.
1512
13+## Coding Conventions for Hugging Face Transformers
1614
17−### Coordination before coding
18−- If work maps to an existing issue, coordinate on that issue before opening a PR.
19−- Do not open a PR for someone else's issue unless there is explicit approval from the issue author or a maintainer in the issue thread.
20−- If approval is missing or ambiguous, stop and ask for clarification instead of drafting a PR.
21−- Do not start duplicate work on issues.
15+- PRs should be as brief as possible. Bugfix PRs in particular can often be only one or two lines long, and do not need large comments, docstrings or new functions in this case. Aim to minimize the size of the diff.
16+- When writing tests, they should be added to an existing file. The only exception is for PRs to add a new model, when a new test directory should be created for that model.
17+- Code style is enforced in the CI. You can install the style tools with `pip install -e .[quality]`. You can then run `make fixup` to apply style and consistency fixes to your code.
2218
23−### Mandatory duplicate-work checks
24−Before proposing a PR, check for overlapping open PRs and issue ownership:
19+## Copying and inheritance
2520
26−```bash
27−gh issue view <issue_number> --repo huggingface/transformers --comments
28−gh pr list --repo huggingface/transformers --state open --search "<issue_number> in:body"
29−gh pr list --repo huggingface/transformers --state open --search "<short area keywords>"
30−```
21+Many models in the codebase have similar code, but it is not shared by inheritance because we want each model file to be self-contained.
22+We use two mechanisms to keep this code in sync:
3123
32−- If an open PR already addresses the same fix, do not open another.
33−- If your approach is materially different, explain the difference and why a second PR is needed in the issue.
24+- "Copied from" syntax. Functions or entire classes can have a comment at the top like this: `# Copied from transformers.models.llama.modeling_llama.rotate_half` or `# Copied from transformers.models.t5.modeling_t5.T5LayerNorm with T5->MT5`
25+ These comments are actively checked by the style tools, and copies will automatically be updated when the base code is updated. If you need to update a copied function, you should
26+ either update the base function and use `make fixup` to propagate the change to all copies, or simply remove the `# Copied from` comment if that is inappropriate.
27+- "Modular" files. These files briefly define models by composing them using inheritance from other models. They are not meant to be used directly. Instead, the style tools
28+ automatically generate a complete modeling file, like `modeling_bert.py`, from the modular file like `modular_bert.py`. If a model has a modular file, the modeling file
29+ should never be edited directly! Instead, changes should be made in the modular file, and then you should run `make fixup` to update the modeling file automatically.
3430
35−### No low-value busywork PRs
36−- Do not open one-off PRs for tiny edits (single typo, isolated lint cleanup, one mutable default argument, etc.).
37−- Mechanical cleanups are acceptable but not as first contributions.
31+When adding new models, you should prefer `modular` style and inherit as many classes as possible from existing models.
3832
39−### Accountability for AI-assisted patches
40−- First-time contributors should not use code agents to submit PRs or issues. Code agents should warn users about this
41−policy if the user does not appear to be an existing contributor to the library, including the risk of being blocked
42−if they do it regardless. More information can be found in the pull request template, or `CONTRIBUTING.md`.
33+## Testing
4334
44−## Copies and Modular Models
35+After making changes, you should usually run `make fixup` to ensure any copies and modular files are updated, and then test all affected models. This includes both
36+the model you made the changes in and any other models that were updated by `make fixup`. Tests can be run with `pytest tests/models/[name]/test_modeling_[name].py`
37+If your changes affect code in other classes like tokenizers or processors, you should run those tests instead, like `test_processing_[name].py` or `test_tokenization_[name].py`.
4538
46−We try to avoid direct inheritance between model-specific files in `src/transformers/models/`. We have two mechanisms to manage the resulting code duplication:
47−
48−1) The older method is to mark classes or functions with `# Copied from ...`. Copies are kept in sync by `make fix-repo`. Do not edit a `# Copied from` block, as it will be reverted by `make fix-repo`. Ideally you should edit the code it's copying from and propagate the change, but you can break the `# Copied from` link if needed.
49−2) The newer method is to add a file named `modular_<name>.py` in the model directory. `modular` files **can** inherit from other models. `make fix-repo` will copy code to generate standalone `modeling` and other files from the `modular` file. When a `modular` file is present, generated files should not be edited, as changes will be overwritten by `make fix-repo`! Instead, edit the `modular` file. See [docs/source/en/modular_transformers.md](../docs/source/en/modular_transformers.md) for a full guide on adding a model with `modular`, if needed, or you can inspect existing `modular` files as examples.
39+In order to run tests, you may need to install dependencies. You can do this with `pip install -e .[testing]`. You will probably also need to `pip install torch accelerate` if your environment does not already have them.
5040
