AGENTS.md

1# AGENTS.md
2 
3This file provides guidance to coding agents when working with code in this repository.
4 
5## Overview
6 
7DuckDB is a high-performance analytical database system designed to be fast, reliable, portable, and easy to use. It is an analytical database management system with a rich SQL dialect, vectorized execution engine, and columnar storage format.
8 
9## Build Commands
10 
11### Basic Build
12```bash
13make debug                   # Builds debug version with sanitizers and assertions
14make reldebug                # Builds optimized release version with debug symbols
15FORCE_DEBUG=1 make relassert # Builds optimized version with sanitizers and assertions
16```
17 
18## Testing
19 
20### Running Tests
21```bash
22build/reldebug/test/unittest # Fast unit tests
23```
24 
25### Running Specific Tests
26```bash
27# Run specific test file
28build/reldebug/test/unittest test/sql/order/test_limit.test
29 
30# Run all tests including slow tests
31build/reldebug/test/unittest "*"
32```
33 
34It is recommended to use `make reldebug` and `build/reldebug/test/unittest` unless a good reason exists to use the debug build - the debug build is much slower than the reldebug build.
35 
36### Test File Format
37Tests use the sqllogictest format (`.test` files). Example structure:
38```sql
39# name: test/sql/order/test_limit.test
40# description: Test LIMIT keyword
41# group: [order]
42 
43statement ok
44CREATE TABLE test (a INTEGER, b INTEGER);
45 
46query I
47SELECT a FROM test LIMIT 1
48----
4911
50 
51statement error
52SELECT a FROM test LIMIT a
53----
54<REGEX>:Binder Error:.*not found.*
55```
56 
57Test directives:
58- `statement ok` - Statement should succeed
59- `statement error` - Statement should fail
60- `query I` - Query returning INTEGER column
61- `query II` - Query returning two columns
62- `----` - Separates query from expected results
63- `<REGEX>:` - Expected error message pattern
64- `require-env VAR` - Test requires environment variable
65 
66Slow tests should use `.test_slow` extension instead of `.test`.
67 
68## Code Formatting
69 
70```bash
71make format-fix        # Format all code (clang-format + black)
72make generate-files    # Generate files + format all code
73```
74 
75Ensure you run formatting before committing.
76 
77## Extensive Testing / Making CI Work
78 
79Below is a set of tests that should be run in order to make sure a changeset passes extensive tests in CI. If the user is asking you to fix CI make sure that the below commands succeed.
80 
81```bash
82make allunit
83FORCE_DEBUG=1 FORCE_ASSERT=1 make reldebug && build/reldebug/test/unittest
84make test_configs
85make test_vector
86```
87 
88 
89## Architecture
90 
91### Query Execution Pipeline
92 
93```
94SQL String
95    ↓
96[PARSER] - Uses a PEG parser to parse SQL into AST
97    ↓
98SQLStatement tree (ParsedExpression, TableRef objects)
99    ↓
100[PLANNER/BINDER] - Binds symbols to catalog, creates logical plan
101    ↓
102Logical Plan (LogicalOperator tree with bound Expressions)
103    ↓
104[OPTIMIZER] - Applies rule-based and cost-based optimizations
105    ↓
106Optimized Logical Plan
107    ↓
108[PHYSICAL PLAN GENERATOR] - Converts to physical operators
109    ↓
110Physical Plan (PhysicalOperator tree)
111    ↓
112[EXECUTOR] - Executes with vectorized, parallel pipelines
113    ↓
114Results
115```
116 
117### Core Components
118 
119**Parser** (`src/parser/`)
120- Converts SQL strings to Abstract Syntax Tree (AST)
121- Uses a PEG-based parser
122- The grammar is located in `*.gram` files and generated using `scripts/build_grammar.sh`
123- Outputs: `SQLStatement`, `ParsedExpression`, `TableRef` objects
124- Key subdirectories: `expression/`, `statement/`, `tableref/`, `peg/`
125 
126For more details on adding new grammar, see the README located at `src/parser/peg/README.md`. 
127Each new grammar rule must have a corresponding transformer rule, located at `peg/transformer`.
128 
129**Planner** (`src/planner/`)
130- Binds symbols to catalog entries and resolves types
131- Creates logical query execution plan
132- Key classes: `Binder`, `LogicalOperator`, bound `Expression` types
133- Subdirectories: `binder/`, `expression/`, `subquery/`
134 
135**Optimizer** (`src/optimizer/`)
136- Transforms logical plans without changing semantics
137- Applies predicate pushdown, join ordering, expression rewriting, etc.
138- Subdirectories: `join_order/`, `statistics/`, `rule/`, `pushdown/`
139 
140**Execution Engine** (`src/execution/`)
141- Converts logical plan to physical plan and executes
142- Push-based vectorized execution model
143- Processes data in batches (typically 2048 rows)
144- Key subdirectories: `operator/` (scan, join, filter, aggregate, etc.), `expression_executor/`
145 
146**Storage** (`src/storage/`)
147- Manages persistent data storage and buffer management
148- Block-based storage with compression
149- Includes WAL (Write-Ahead Log) for durability
150- Subdirectories: `buffer/`, `compression/`, `checkpoint/`, `table/`
151 
152**Catalog** (`src/catalog/`)
153- Metadata management for tables, schemas, functions, types, etc.
154- Single source of truth for database metadata
155- Key classes: `Catalog`, `CatalogEntry`, `SchemaCatalogEntry`
156 
157**Transaction Manager** (`src/transaction/`)
158- ACID transaction management with MVCC
159- Coordinates concurrent access to data
160- Key files: `transaction_manager.cpp`, `undo_buffer.cpp`, `wal_write_state.cpp`
161 
162**Parallel Execution** (`src/parallel/`)
163- Multi-threaded execution with task scheduling
164- Pipeline-based parallelism
165- Key files: `executor.cpp`, `pipeline_executor.cpp`, `task_scheduler.cpp`
166 
167**Functions** (`src/function/`)
168- Built-in function implementations
169- Types: `scalar/`, `aggregate/`, `table/`, `window/`, `pragma/`
170 
171### Directory Structure
172 
173```
174/duckdb
175├── src/                   # Core C++ source code
176│   ├── include/duckdb/   # Public headers
177│   ├── parser/           # SQL parsing
178│   ├── planner/          # Logical planning
179│   ├── optimizer/        # Query optimization
180│   ├── execution/        # Physical execution
181│   ├── storage/          # Data storage
182│   ├── catalog/          # Metadata management
183│   ├── transaction/      # Transaction management
184│   ├── parallel/         # Parallelization
185│   ├── function/         # Built-in functions
186│   ├── common/           # Shared utilities and types
187│   └── main/             # Database/connection management
188├── extension/            # In-tree extensions (parquet, json, icu, etc.)
189├── test/                 # Test framework and test cases
190│   ├── sql/             # SQL regression tests (.test files)
191│   └── api/             # C/C++ API tests
192├── tools/                # Language bindings (pythonpkg, shell, etc.)
193├── benchmark/            # Benchmark suites (TPC-H, TPC-DS, etc.)
194├── scripts/              # Build and utility scripts
195└── third_party/          # Third-party dependencies
196```
197 
198## Extensions
199 
200DuckDB supports two types of extensions:
201 
202**In-Tree Extensions** (in `extension/` directory):
203- Extensions are located in-tree
204- Full list in `.github/config/in_tree_extensions.cmake`
205- Code can be edited directly and checked into the repository.
206 
207**Out-of-Tree Extensions**:
208- Extensions are located in a separate git repository
209- Full list in `.github/config/out_of_tree_extensions.cmake`
210- When changes have to be made, they have to be made in patch files stored in `.github/patches`
211 
212Building with extensions:
213```bash
214# build all extensions
215BUILD_ALL_EXT=1 make
216# build specific extensions
217DUCKDB_EXTENSIONS='json;icu' make
218```
219 
220## Key Development Patterns
221 
222### Data Flow
223- **Vectorized Processing**: Data processed in columnar batches (not row-by-row), typically 2048 rows per batch
224- **Vector class**: Represents a columnar batch of data
225- **ColumnBinding**: Unique identifier `(table_index, column_index)` for columns throughout planning/execution
226 
227### Expression Types
228- `ParsedExpression` - From parser, unbound
229- `Expression` - Bound with type information
230- `ExpressionExecutor` - Vectorized execution of expressions
231 
232### Memory Management
233- Prefer `unique_ptr<T>` for exclusive ownership
234- Use `shared_ptr<T>` only when necessary
235- `optional_ptr<T>` for nullable references, `reference<T>` for non-nullable references
236- Never use raw pointers
237 
238### Type System
239- `LogicalType` - Abstract data type representation
240- Type promotion rules in `src/function/cast_rules.cpp`
241- Custom types supported via extension system
242 
243### Common Patterns
244- **Visitor Pattern**: For tree traversal (e.g., `LogicalOperatorVisitor`, `ExpressionIterator`)
245- **Factory Pattern**: `Deserialize()` methods for object creation
246- **Class Hierarchy**: Base classes like `*Operator`, `*Entry`, `*Expression` with typed subclasses
247 
248## Coding Guidelines (Key Points)
249 
250### C++ Style
251- Use tabs for indentation, spaces for alignment
252- Lines should not exceed 120 columns (run formatter)
253- Use `[u]int(8|16|32|64)_t` instead of `int`, `long`, etc.
254- Use `idx_t` instead of `size_t` for offsets/indices/counts
255- Use `const` references for non-trivial objects
256- Use C++11 range-based for loops when possible
257- Always use braces for if statements and loops
258- Never use `const_cast`
259 
260### Comment Conventions
261 
262Try to keep comments short. In general, comments should be one short line. Only in exceptional situations should comments be more than one short line. Code should be mostly self-descriptive and too many large comments make code harder to read and understand.
263 
264Avoid adding comments specific to how a change was made to the code that relates to a specific issue. For example, a comment like "add +1 to fix an off-by-one error" is not relevant to understanding the code. Such comments related to specific issues that were addressed belong in a PR description or commit message, not in the code itself.
265 
266### Naming Conventions
267- **Files**: `snake_case` (e.g., `abstract_operator.cpp`)
268- **Types**: `PascalCase` (e.g., `LogicalOperator`)
269- **Variables**: `snake_case` (e.g., `chunk_size`)
270- **Functions**: `PascalCase` (e.g., `GetChunk`)
271 
272### Class Layout
273```cpp
274class MyClass {
275public:
276    MyClass();
277    int my_public_variable;
278 
279public:
280    void MyFunction();
281 
282private:
283    void MyPrivateFunction();
284 
285private:
286    int my_private_variable;
287};
288```
289 
290### Error Handling
291- Use exceptions for query-terminating errors (parser error, table not found, out-of-memory, etc.)
292- Use return values for errors that are recoverable during a query
293- Use `D_ASSERT` for programmer errors (never triggered by user input)
294- Assert liberally with clear comments
295 
296### Testing Requirements
297- Prefer sqllogictest framework (`.test` files) over C++ tests
298- Test with different types (numerics, strings, nested types)
299- Test unexpected/incorrect usage, not just happy path
300- Slow tests should use `.test_slow` extension
301- All tests must pass before submitting PR (`make allunit`)
302- Aim for high code coverage
303 
304## Navigation Tips
305 
306### Finding Components
307- Entry point: `src/main/database.cpp` (DatabaseInstance)
308- Query execution coordinator: `src/main/client_context.cpp`
309- SQL parsing: `src/parser/parser.cpp`
310- Logical planning: `src/planner/binder/query_planner.cpp`
311- Optimization orchestration: `src/optimizer/optimizer.cpp`
312- Physical plan generation: `src/execution/physical_plan/physical_plan_generator.cpp`
313- Execution orchestration: `src/parallel/executor.cpp`
314 
315### Searching the Codebase
316- Use `grep` or `ripgrep` for code search
317- Function definitions typically in `.cpp` files
318- Class declarations in `src/include/duckdb/` headers
319- Test cases in `test/sql/` by functionality
320 
321### Understanding a Feature
3221. Find test cases in `test/sql/` to see usage examples
3232. Trace from parser → planner → optimizer → execution
3243. Look for corresponding `*Statement`, `*Operator`, `*Expression` classes
3254. Check function registration in catalog
326 
327### Modifying Generated Files
328Some files are auto-generated. After modifying their sources, run:
329```bash
330make generate-files
331```
332This regenerates:
333- C API bindings
334- Function registration
335- Settings
336- Serialization code
337- Storage info
338- Metric enums
339- Enum utilities
340 
341## Documentation
342 
343- Main docs: https://duckdb.org/docs/
344- Development docs: https://duckdb.org/dev/
345- Build guide: https://duckdb.org/docs/dev/building/overview
346- Testing docs: https://duckdb.org/dev/testing
347 
348## Important Files
349 
350- `Makefile` - Main build configuration
351- `CMakeLists.txt` - CMake configuration
352- `CONTRIBUTING.md` - Contribution guidelines
353- `test/README.md` - Testing documentation
354- `extension/extension_config.cmake` - Extension configuration
355- `scripts/format.py` - Code formatter
356- `scripts/generate_*.py` - Code generation scripts
357