AGENTS.md

1# AGENTS.md — AutogenerateAgentsMD.md
2 
3## Project Overview
4 
5`GenerateAgents.md` is a Python command-line tool that automates the creation of a comprehensive `AGENTS.md` file for any public GitHub or local code repository. It acts as an automated codebase analyst and technical writer, using the `dspy` framework to programmatically interface with LLMs. The tool clones and analyzes a target codebase to produce a standardized blueprint, enabling AI coding agents to rapidly understand a project's architecture, conventions, and data flow. The primary language is Python (>=3.12).
6 
7## Tech Stack
8 
9*   **Primary Language:** Python (>=3.12)
10*   **Core AI Framework:** `dspy`
11*   **LLM Abstraction Layer:** `litellm`
12*   **Dependency Management:** `uv`
13*   **CLI Framework:** `argparse` (standard library)
14*   **Configuration:** `python-dotenv`
15*   **Version Control Interaction:** `git` (via `subprocess`)
16*   **Testing:** `pytest`
17 
18## Architecture
19 
20The application follows a modular, stateless pipeline pattern orchestrated by the main CLI entry point.
21 
22*   `src/autogenerateagentsmd/cli.py`: The command-line interface entry point. It parses arguments and orchestrates the entire analysis and generation pipeline via the `run_agents_md_pipeline` function.
23*   `src/autogenerateagentsmd/modules.py`: Contains the core `dspy.Module` classes (`CodebaseConventionExtractor`, `AgentsMdCreator`, `AntiPatternExtractor`). These modules encapsulate the primary LLM-driven logic for analyzing code and synthesizing the final document.
24*   `src/autogenerateagentsmd/signatures.py`: Defines the contracts for LLM interactions using `dspy.Signature`. These signatures specify the expected inputs (e.g., source code) and outputs (e.g., extracted conventions) for each LLM-powered step.
25*   `src/autogenerateagentsmd/model_config.py`: Centralizes the configuration for supported LLMs, making it easy to switch between models like Gemini, Claude, and OpenAI.
26*   `src/autogenerateagentsmd/utils.py`: Contains helper functions for non-LLM tasks, such as cloning Git repositories, loading files into memory, and other file system operations.
27*   `tests/`: The test suite, containing end-to-end and unit tests.
28*   `pyproject.toml`: Defines project metadata, dependencies, and the `autogenerateagentsmd` console script entry point.
29 
30## Code Style
31 
32The project enforces a strict and consistent Python coding style.
33 
34*   **Type Hinting**: Type hints are strictly mandatory for all function parameters and return values.
35 
36```python
37    # Good
38    def load_source_tree(repo_path: str) -> dict[str, str]:
39        # ...
40 
41    # Bad
42    def load_source_tree(repo_path):
43        # ...
44```
45 
46*   **Naming Conventions**:
47    *   `snake_case` for functions, methods, and variables (e.g., `run_agents_md_pipeline`).
48    *   `PascalCase` for all classes (e.g., `CodebaseConventionExtractor`).
49    *   `ALL_CAPS_WITH_UNDERSCORES` for module-level constants.
50 
51*   **Import Ordering**: Imports must be grouped at the top of each file in the following order: 1) Standard library, 2) Third-party libraries, 3) Local application imports.
52 
53```python
54    # Good
55    import argparse
56    import logging
57    from pathlib import Path
58 
59    import dspy
60    from dotenv import load_dotenv
61 
62    from .modules import AgentsMdCreator, CodebaseConventionExtractor
63    from .utils import clone_repo, load_source_tree
64```
65 
66*   **Formatting**: Use 4-space indentation and maintain a line length between 80-100 characters.
67 
68## Anti-Patterns & Restrictions
69 
70*   **NEVER use on private codebases**: The tool sends the full source code to third-party LLM APIs. Using it on private, proprietary, or sensitive codebases is a significant security risk. It is designed **exclusively** for public, open-source repositories.
71*   **NEVER commit secrets**: Do not commit the `.env` file or any other file containing API keys or secrets to version control.
72*   **DO NOT introduce new LLM frameworks**: The project architecture is tightly coupled to `dspy`. Avoid introducing other orchestration frameworks like LangChain or LlamaIndex.
73*   **AVOID new end-to-end tests**: E2E tests are slow and costly due to live API calls. Prioritize mocked unit tests for new functionality unless there is a strong justification for an E2E test.
74 
75## Database & State Management
76 
77The application is entirely **stateless**. It does not use a database or any form of persistent storage between runs. All configuration is loaded at runtime from command-line arguments and the `.env` file. The application's state exists only for the duration of a single execution, primarily as an in-memory dictionary (`source_tree`) holding the target repository's code. The only output is the generated `AGENTS.md` file.
78 
79## Error Handling & Logging
80 
81*   **Error Handling**: Application logic within modules and utilities should raise specific exceptions (e.g., `FileNotFoundError`, `subprocess.CalledProcessError`). Generic `except Exception` blocks should be avoided. A single global `try...except Exception` block exists in `src/autogenerateagentsmd/cli.py` to catch any unhandled exceptions at the top level and provide a clean exit with a user-friendly error message.
82*   **Logging**: The standard `logging` module is used for progress reporting. It is configured in `cli.py` to print `INFO`-level messages to the console, informing the user about the current stage of the pipeline (e.g., "Cloning repository...", "Extracting conventions...").
83 
84## Testing Commands
85 
86*   **Install dependencies for testing:**
87```bash
88    uv sync --extra dev
89```
90*   **Run all tests (excluding slow E2E tests):**
91```bash
92    pytest
93```
94*   **Run only the end-to-end (E2E) tests:**
95```bash
96    pytest -m e2e
97```
98*   **Run tests for a specific file:**
99```bash
100    pytest tests/test_utils.py
101```
102 
103## Testing Guidelines
104 
105The project uses `pytest` for testing. The testing strategy is two-pronged:
106 
107*   **End-to-End (E2E) Tests**:
108    *   Located in `tests/test_e2e_pipeline.py`.
109    *   These tests validate the entire pipeline by cloning real public repositories and making live LLM API calls.
110    *   They are marked with `@pytest.mark.e2e` and are run sparingly due to their cost and long execution time.
111    *   They serve as the ultimate validation that the integrated system works as expected.
112 
113*   **Unit Tests**:
114    *   This is the preferred method for testing new contributions.
115    *   Focus on testing individual functions in `src/autogenerateagentsmd/utils.py`.
116    *   For `dspy` modules in `src/autogenerateagentsmd/modules.py`, tests should use mocking to avoid actual LLM API calls. This ensures tests are fast, deterministic, and free of cost.
117 
118## Security & Compliance
119 
120*   **API Key Management**: All API keys and other secrets **must** be stored in a `.env` file at the project root. This file is explicitly listed in `.gitignore` and must never be committed to version control.
121*   **Data Handling and Privacy**: The tool's core function involves sending the entire source code of a target repository to external, third-party LLM APIs. This is a critical security consideration. **NEVER** run this tool on any repository containing proprietary code, sensitive data, secrets, or personally identifiable information (PII). It is intended for use only on publicly available, open-source software.
122 
123## Dependencies & Environment
124 
125*   **Dependency Management**: Dependencies are managed with `uv` and are defined in `pyproject.toml`.
126    *   Production dependencies are under `[project.dependencies]`.
127    *   Development dependencies (like `pytest`) are under `[project.optional-dependencies]dev`.
128*   **Installation**: To install all required dependencies for development and testing, run the following command from the project root:
129```bash
130    uv sync --extra dev
131```
132*   **Environment Variables**: The application requires API keys for the desired LLM providers. Create a `.env` file in the project root and add the necessary keys.
133```bash
134    # Example .env file
135    OPENAI_API_KEY="sk-..."
136    ANTHROPIC_API_KEY="..."
137    GEMINI_API_KEY="..."
138```
139*   **Runtime Version**: The project requires Python version 3.12 or newer.
140 
141## PR & Git Rules
142 
143The project uses Git for version control. The repository includes a standard Python `.gitignore` file to exclude common artifacts like `__pycache__`, virtual environments (`.venv`), build directories, and the `.env` file containing secrets. No specific branch naming conventions or commit message formats are formally documented.
144 
145## Documentation Standards
146 
147*   **User Documentation**: The `README.md` file serves as the primary user-facing guide. It contains the project's purpose, installation instructions, and command-line usage examples.
148*   **Agent/Developer Documentation**: The `AGENTS.md` file (which this tool generates for itself) is the definitive technical guide for developers and AI agents. It provides a deep, structured overview of the architecture, conventions, and patterns.
149*   **In-Code Documentation**:
150    *   **Type Hints**: Mandatory for all function signatures.
151    *   **Docstrings**: Public modules and complex functions should have descriptive docstrings explaining their purpose, arguments, and return values.
152 
153## Common Patterns
154 
155*   **Stateless Pipeline Pattern**: The entire application is orchestrated as a linear, stateless pipeline in `src/autogenerateagentsmd/cli.py`. Data flows from one stage to the next (e.g., `load_source_tree` -> `CodebaseConventionExtractor` -> `AgentsMdCreator`) without persisting state between executions.
156*   **DSPy Modules for LLM Logic**: All direct interactions with large language models are encapsulated within `dspy.Module` classes (e.g., `AgentsMdCreator`). This separates the prompt engineering and LLM logic from the main application orchestration code.
157*   **Strict Type Hinting**: ALWAYS add type hints to all function parameters and return values. This is a non-negotiable standard for code clarity and static analysis.
158```python
159    # ALWAYS do this
160    def clone_repo(github_url: str, target_dir: Path) -> None:
161        ...
162```
163*   **Specific Exception Handling**: NEVER use a generic `except Exception:` in application modules. ALWAYS catch specific, anticipated exceptions to handle errors gracefully and avoid masking unknown bugs.
164```python
165    # Good
166    try:
167        # some file operation
168    except FileNotFoundError:
169        logger.error("Could not find the specified file.")
170 
171    # Bad
172    try:
173        # some file operation
174    except Exception as e:
175        logger.error(f"An unknown error occurred: {e}")
176```
177 
178## Agent Workflow / SOP
179 
180When tasked with modifying or extending this codebase, follow this standard operating procedure:
181 
1821.  **Understand the Goal**: Clarify the specific change required. Is it adding a new analysis capability, supporting a new LLM, or fixing a bug in the file processing?
1832.  **Locate Relevant Code**:
184    *   For CLI changes (new arguments): `src/autogenerateagentsmd/cli.py`.
185    *   For new LLM-driven analysis: Define a new `dspy.Signature` in `signatures.py` and a new `dspy.Module` in `modules.py`.
186    *   For general helper functions (e.g., file handling): `src/autogenerateagentsmd/utils.py`.
187    *   For model configuration: `src/autogenerateagentsmd/model_config.py`.
1883.  **Implement the Change**: Adhere strictly to the coding conventions:
189    *   Add mandatory type hints for all new functions.
190    *   Use `snake_case` for functions/variables and `PascalCase` for classes.
191    *   Isolate LLM logic within a `dspy.Module`.
1924.  **Write Tests**:
193    *   For changes in `utils.py`, add a new unit test to the appropriate file in the `tests/` directory.
194    *   For a new `dspy.Module`, write a unit test that mocks the LLM call to verify the module's behavior without making a real API request.
1955.  **Verify**: Run the local test suite using `pytest` to ensure your changes have not introduced any regressions.
1966.  **Document**: If you've added a new user-facing feature (like a new CLI flag), update the `README.md`. The `AGENTS.md` is auto-generated and does not need manual updates.
197 
198## Few-Shot Examples
199 
200### 1. Type Hinting
201 
202*   **Good**: Mandatory type hints for parameters and return values.
203```python
204    from pathlib import Path
205 
206    def save_markdown_file(content: str, output_path: Path) -> None:
207        """Saves the given content to a file."""
208        output_path.parent.mkdir(parents=True, exist_ok=True)
209        output_path.write_text(content, encoding="utf-8")
210```
211 
212*   **Bad**: Missing type hints.
213```python
214    def save_markdown_file(content, output_path):
215        """Saves the given content to a file."""
216        output_path.parent.mkdir(parents=True, exist_ok=True)
217        output_path.write_text(content, encoding="utf-8")
218```
219 
220### 2. Import Ordering
221 
222*   **Good**: Imports are correctly grouped (standard library, third-party, local).
223```python
224    import logging
225    import subprocess
226    from pathlib import Path
227 
228    from git.repo import Repo # Hypothetical third-party library
229 
230    from .exceptions import GitCloneError
231```
232 
233*   **Bad**: Imports are mixed together without logical grouping.
234```python
235    from pathlib import Path
236    from .exceptions import GitCloneError
237    import logging
238    from git.repo import Repo
239    import subprocess
240```
241 
242### 3. Error Handling
243 
244*   **Good**: Catching a specific, expected exception.
245```python
246    import subprocess
247 
248    def run_git_command(command: list[str]) -> str:
249        try:
250            result = subprocess.run(
251                command,
252                check=True,
253                capture_output=True,
254                text=True
255            )
256            return result.stdout
257        except subprocess.CalledProcessError as e:
258            logging.error(f"Git command failed: {e.stderr}")
259            raise
260```
261 
262*   **Bad**: Using a generic `except Exception` which can hide other bugs.
263```python
264    import subprocess
265 
266    def run_git_command(command: list[str]) -> str:
267        try:
268            result = subprocess.run(
269                command,
270                check=True,
271                capture_output=True,
272                text=True
273            )
274            return result.stdout
275        except Exception as e: # This is too broad
276            logging.error(f"An unexpected error occurred: {e}")
277            raise
278    
279```