Loading
Also from Kynth Studios
Also from Kynth Studios
Also from Kynth Studios
1# Dragonfly Development Guide23> **Essential reference for working with the Dragonfly codebase**4> Architecture, build system, testing infrastructure, and development workflows.56---78## Table of Contents9101. [Critical Workflow Rules](#critical-workflow-rules)112. [Quick Command Reference](#quick-command-reference)123. [Project Overview](#project-overview)134. [Repository Structure](#repository-structure)145. [Build Instructions](#build-instructions)156. [Testing](#testing)167. [CI/CD Pipeline](#cicd-pipeline)178. [Code Style & Pre-commit Hooks](#code-style--pre-commit-hooks)189. [Third-Party Dependencies](#third-party-dependencies)1910. [Platform Support](#platform-support)2011. [CMake Build Options](#cmake-build-options)2112. [Key Files Reference](#key-files-reference)2213. [Common Pitfalls](#common-pitfalls)2314. [Debugging Tips](#debugging-tips)2415. [Validation Checklist](#validation-checklist)2526---2728## Critical Workflow Rules2930**MANDATORY - Always Follow This Order:**31321. ✅ **Read Before Edit** - Always read files before modifying332. ✅ **Use Correct Build Commands** - See [Quick Command Reference](#quick-command-reference) below343. ✅ **Test After Changes** - Build and run a relevant unit test -35 `ninja <unit_test> && ./unit_test`364. ✅ **Format Code** - `pre-commit run --files <files>`375. ✅ **Follow Architecture** - See [Architecture Patterns](#architecture-patterns) below386. ✅ **Never Push to Main** - Always create a feature branch and open a PR. Never run `git push origin main`.3940### Pull Request Guidelines4142**Conciseness is Key**: PR descriptions should be short, focused, and easy to scan.43- **Title**: Imperative, descriptive (e.g., "Fix fiber stack overflow in test_reply_guard_oom")44- **Summary**: 1-2 sentences explaining *what* changed and *why*45- **Changes**: Bullet points for key changes46- **Fixes**: Link issues (e.g., "Fixes #123")47- **Commit messages**: Keep every line (subject and body) <= 100 characters; wrap long descriptions4849---5051## Quick Command Reference5253**CRITICAL: Read the full sections below for context. These are shortcuts only.**5455### Building (see [Build Instructions](#build-instructions) for details)5657```bash58# Debug build (for development)59./helio/blaze.sh -DWITH_AWS=OFF -DWITH_GCP=OFF60cd build-dbg && ninja dragonfly # Build main binary61cd build-dbg && ninja generic_family_test # Build specific test6263# Release build for local benchmarking64./helio/blaze.sh -release -DWITH_AWS=OFF -DWITH_GCP=OFF65cd build-opt && ninja dragonfly66```6768### Testing (see [Testing](#testing) for details)6970```bash71# C++ Unit Tests72cd build-dbg73ctest -V -L DFLY # Run all tests74./generic_family_test # Run specific test binary75./generic_family_test --gtest_filter="Set.*" # Run specific test case76```7778### Code Formatting7980```bash81# Setup (once)82pipx install pre-commit clang-format black83pre-commit install8485# Format code86pre-commit run --files <files> # Format specific files87pre-commit run --all-files # Format all files88```8990### Common Operations9192```bash93# Check git status94git status9596# Check current branch97git branch9899# View recent commits100git log --oneline -10101```102103---104105## Architecture Patterns106107**Code Style**: [.clang-format](.clang-format) - snake_case vars, PascalCase functions, kPascalCase constants108109**DO ✅**:110- Fiber-aware: `util::fb2::Mutex`, `util::fb2::Fiber` → [helio/util/fibers/](helio/util/fibers/)111- Per-shard ops (no global state) → [docs/df-share-nothing.md](docs/df-share-nothing.md)112- Command pattern → [src/server/set_family.cc](src/server/set_family.cc)113- Error handling: `OpStatus` → [src/server/common.h](src/server/common.h)114- Test patterns → [tests/dragonfly/conftest.py](tests/dragonfly/conftest.py)115116**DON'T ❌**:117- `std::thread`, `std::mutex` (deadlocks!)118- Global mutable state119- Edit without reading120- Skip tests121- Use `std::regex` in fiber/server paths (recursive implementation can overflow small fiber stacks)122- Use `./tools/docker/build.sh` for local development (use `ninja` instead)123- Use `make` for incremental builds (use `ninja` instead)124125---126127## Project Overview128129**Dragonfly** is a high-performance, Redis and Memcached compatible in-memory data store written in C++20. It delivers significantly higher throughput than traditional single-threaded Redis implementations through innovative architectural choices.130131### Key Characteristics132133- **Language**: C++20 (Google C++ Style Guide 2020 version)134- **Architecture**: Shared-nothing multi-threaded design (via `helio` library)135- **Performance**: Uses io_uring (Linux 5.11+) for high-performance async I/O, with epoll fallback136- **Threading Model**: Fiber-based cooperative multitasking with lock-free data structures137- **Build System**: CMake + Ninja via `helio/blaze.sh` wrapper script138- **Target Platform**: Linux (kernel 5.11+ recommended), FreeBSD support available139- **Protocols**: Redis RESP2/RESP3, Memcached binary protocol140- **Compatibility**: Drop-in replacement for Redis API coverage141142### Architectural Highlights143144**For detailed architecture documentation, see [docs/df-share-nothing.md](docs/df-share-nothing.md)**1451461. **Shared-Nothing Design**: Each thread operates independently with its own data structures, minimizing lock contention1472. **Helio Framework**: Custom I/O and threading library built on io_uring/epoll with fiber support1483. **DashTable**: Novel hash table implementation optimized for multi-core systems - see [docs/dashtable.md](docs/dashtable.md)1494. **Transaction Model**: Non-blocking optimistic transactions - see [docs/transaction.md](docs/transaction.md)1505. **Tiering Support**: Optional disk-backed storage for large datasets1516. **Search Module**: Full-text search capabilities (when enabled with WITH_SEARCH)152153---154155## Repository Structure156157```158dragonfly/159├── src/ # Main C++ source code160│ ├── server/ # Core server implementation161│ │ ├── dfly_main.cc # Main entry point162│ │ ├── main_service.cc # Service lifecycle & command routing163│ │ ├── db_slice.cc # Per-thread database shard164│ │ ├── engine_shard_set.cc # Shard management165│ │ ├── cluster/ # Cluster mode implementation166│ │ ├── journal/ # Replication journal167│ │ ├── tiering/ # Tiered storage168│ │ ├── search/ # Search module169│ │ └── acl/ # Access control lists170│ ├── core/ # Core data structures171│ │ ├── dash.h # DashTable hash table172│ │ ├── dense_set.h # Compact set implementation173│ │ ├── string_map.h # Optimized string-keyed maps174│ │ ├── search/ # Search core algorithms175│ │ └── json/ # JSON support176│ ├── facade/ # Network & command handling177│ │ ├── dragonfly_connection.cc # Connection management178│ │ ├── redis_parser.cc # RESP protocol parser179│ │ └── memcache_parser.cc # Memcached protocol180│ └── redis/ # Redis-specific implementations181│ └── lua/ # Lua scripting support182│183├── helio/ # Git submodule: I/O and threading library184│ │ # ** DO NOT EDIT unless contributing to helio **185│ ├── util/ # Utilities: fibers, I/O, synchronization186│ ├── io/ # io_uring & epoll abstraction187│ └── blaze.sh # Build configuration wrapper188│189├── tests/ # Test suite190│ ├── dragonfly/ # Python pytest integration/regression tests191│ │ ├── conftest.py # Pytest fixtures & configuration192│ │ ├── requirements.txt # Python test dependencies193│ │ └── *.py # Test files194│ └── pytest.ini # Pytest configuration & markers195│196├── docs/ # Documentation197│ ├── build-from-source.md # Build instructions198│ ├── dashtable.md # DashTable internals199│ ├── transaction.md # Transaction model200│ ├── df-share-nothing.md # Shared-nothing architecture201│ └── differences.md # Differences from Redis202│203├── contrib/ # Utilities204│ ├── docker/ # Docker configurations205│ └── charts/dragonfly/ # Helm chart for Kubernetes206│207├── tools/ # Benchmarking & utility tools208│ └── packaging/ # Packaging scripts209│210├── CMakeLists.txt # Root CMake configuration211├── .clang-format # C++ formatting rules (clang-format v14.0.6)212├── .pre-commit-config.yaml # Pre-commit hooks configuration213├── pyproject.toml # Python formatting (Black, 100 chars)214└── CONTRIBUTING.md # Contribution guidelines215```216217### Critical Paths to Remember218219- **Main entry**: `src/server/dfly_main.cc`220- **Command dispatch**: `src/server/main_service.cc`221- **Data storage**: `src/server/db_slice.cc`222- **Networking**: `src/facade/dragonfly_connection.cc`223- **Helio library**: `helio/` (I/O and threading library)224225---226227## Build Instructions228229**For complete build instructions, see [docs/build-from-source.md](docs/build-from-source.md)**230231### Quick Start232233**Debug build** (for development):234```bash235./helio/blaze.sh236cd build-dbg && ninja dragonfly237./dragonfly --alsologtostderr238```239240**Release build** (for production/benchmarking):241```bash242./helio/blaze.sh -release243cd build-opt && ninja dragonfly244```245246**Production release build** (static linking, optimized):247```bash248make release # Configure + build249make package # Create release packages with debug symbols250```251252The [Makefile](Makefile) builds production releases with:253- Static linking: libstdc++, libgcc, Boost, OpenSSL254- Architecture optimizations (x86_64: `-march=core2 -msse4.1 -mtune=skylake`)255- Debug symbols (compressed)256- Output: `build-release/dragonfly-{arch}.tar.gz`257258**Common build options**:259- See [docs/build-from-source.md](docs/build-from-source.md) for all options260261---262263## Testing264265**For complete testing documentation, see [tests/README.md](tests/README.md)**266267### Quick Reference268269**C++ Unit Tests**:270```bash271cd build-dbg272ctest -V -L DFLY # Run all tests273./generic_family_test # Run specific test binary274./generic_family_test --gtest_filter="Set.*" # Run specific test case275```276277**Python Integration Tests (pytest)**:278```bash279# Run from the repo root. The binary path defaults to build-dbg/dragonfly.280# Override with the DRAGONFLY_PATH env var:281DRAGONFLY_PATH=build-dbg/dragonfly python3 -m pytest tests/dragonfly/pymemcached_test.py -xvs282283# Run a single test:284python3 -m pytest tests/dragonfly/pymemcached_test.py::TestMemcached::test_basic -xvs285```286287- `DRAGONFLY_PATH` — sets the path to the Dragonfly binary the test harness starts. Defaults to `build-dbg/dragonfly` relative to the `tests/dragonfly/` directory.288- `--df` — passes **extra flags to the Dragonfly process** (not the binary path). For example: `--df logtostdout --df "vmodule=*=1"`.289290---291292## CI/CD Pipeline293294**For complete CI configuration, see [.github/workflows/ci.yml](.github/workflows/ci.yml)**295296The CI workflow runs on all PRs and includes:297- **Pre-commit checks**: clang-format, black formatters298- **Build matrix**: Multiple OS/compiler/sanitizer combinations (Ubuntu 20/24, Alpine, GCC/Clang, ASAN/UBSAN)299- **Test execution**: C++ unit tests, Python integration tests, cluster mode tests300- **Additional validations**: Helm charts, Docker image builds301302---303304## Code Style & Pre-commit Hooks305306**For complete contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md)**307308**Code style configuration files**:309- **C++**: [.clang-format](.clang-format) - Google C++ Style Guide (2020), clang-format v14.0.6, 100 char limit310- **Python**: [pyproject.toml](pyproject.toml) - Black formatter, 100 char limit, PEP 8 compliant311- **Pre-commit hooks**: [.pre-commit-config.yaml](.pre-commit-config.yaml) - Automated formatting checks312313**Quick setup**:314```bash315pipx install pre-commit clang-format black316pre-commit install317pre-commit run --all-files # Run all formatters318```319320---321322## Third-Party Dependencies323324**Key Libraries**: Abseil (strings/flags), Boost 1.71+ (context/intrusive), mimalloc (allocator), jsoncons (JSON), OpenSSL (TLS), libunwind (traces)325326**Build artifacts**: `build-dbg/third_party/` - DO NOT edit327328**For complete dependency info, see [docs/build-from-source.md](docs/build-from-source.md)**329330---331332## Platform Support333334**Linux**: Primary platform. Kernel 5.11+ (io_uring), 5.1+ (basic), < 5.1 (epoll fallback)335- Check: `uname -r`336- Force epoll: `--proactor_type=epoll`337- Docker: `--security-opt seccomp=unconfined`338339**FreeBSD**: Supported (kqueue backend)340341**macOS**: Not supported for production (use Docker/Linux)342343**For complete platform info, see [docs/build-from-source.md](docs/build-from-source.md)**344345---346347## CMake Build Options348349**For complete list of build options, see [docs/build-from-source.md](docs/build-from-source.md)**350351### Common Options352353Pass options to `helio/blaze.sh` with `-D` prefix:354355```bash356./helio/blaze.sh -DWITH_SEARCH=OFF -DWITH_AWS=ON357```358359**Most useful options**:360- `WITH_ASAN=ON` / `WITH_USAN=ON` - Enable sanitizers for debugging361- `WITH_SEARCH=OFF` - Disable search module for faster builds362- `WITH_AWS=OFF` / `WITH_GCP=OFF` - Disable cloud libraries363- `WITH_TIERING=OFF` - Disable disk storage364- `USE_MOLD=ON` - Faster linking with LTO (production builds)365366**Quick configurations**:367```bash368# Minimal build (fast compilation)369./helio/blaze.sh -DWITH_GPERF=OFF -DWITH_AWS=OFF -DWITH_GCP=OFF -DWITH_TIERING=OFF -DWITH_SEARCH=OFF370371# Full-featured (all options ON by default)372./helio/blaze.sh373374# Production optimized375./helio/blaze.sh -release -DUSE_MOLD=ON376```377378---379380## Key Files Reference381382Quick reference to the most important files in the codebase.383384| Purpose | File Path |385|---------|-----------|386| **Entry Points & Core** | |387| Main entry point | `src/server/dfly_main.cc` |388| Server lifecycle & command routing | `src/server/main_service.cc` |389| Per-thread database shard | `src/server/db_slice.cc` |390| Shard management | `src/server/engine_shard_set.cc` |391| **Data Structures** | |392| DashTable hash table | `src/core/dash.h` |393| Dense set implementation | `src/core/dense_set.h` |394| String map | `src/core/string_map.h` |395| **Networking** | |396| Connection handling | `src/facade/dragonfly_connection.cc` |397| Redis protocol parser | `src/facade/redis_parser.cc` |398| Memcached protocol parser | `src/facade/memcache_parser.cc` |399| **Build System** | |400| Root CMake config | `CMakeLists.txt` |401| Build script wrapper | `helio/blaze.sh` |402| Server CMake config | `src/server/CMakeLists.txt` |403| **CI/CD** | |404| Main CI workflow | `.github/workflows/ci.yml` |405| Pre-commit config | `.pre-commit-config.yaml` |406| **Code Style** | |407| C++ formatting | `.clang-format` |408| Python formatting | `pyproject.toml` |409| **Testing** | |410| Pytest configuration | `tests/pytest.ini` |411| Pytest fixtures | `tests/dragonfly/conftest.py` |412| Test requirements | `tests/dragonfly/requirements.txt` |413| **Documentation** | |414| Build instructions | `docs/build-from-source.md` |415| Architecture overview | `docs/df-share-nothing.md` |416| DashTable internals | `docs/dashtable.md` |417| Transaction model | `docs/transaction.md` |418| **Configuration** | |419| Contributing guide | `CONTRIBUTING.md` |420| CLA agreement | `CLA.txt` |421422---423424## Common Pitfalls4254261. **Pre-commit not installed**: `pipx install pre-commit clang-format black && pre-commit install`4272. **Wrong binary**: Debug: `build-dbg/dragonfly`, Release: `build-opt/dragonfly`4283. **Wrong build command**: Use `cd build-dbg && ninja <target>`, NOT `./tools/docker/build.sh`4294. **Test timeouts**: `timeout 20m ctest -V -L DFLY`4305. **ASAN leaks**: Check CI, suppress in `helio/util/asan_suppressions.txt`4316. **Helio modifications**: DON'T edit `helio/` (it's a git submodule - changes go upstream)4327. **CodeQL checks**: DON'T run codeql_checker when testing changes - it's slow and unnecessary for development433434---435436## Debugging Tips437438**Logging**: `--alsologtostderr --v=1 --vmodule=module=2`439440**ASAN**: `ASAN_OPTIONS=detect_leaks=1:symbolize=1`, suppressions: `helio/util/asan_suppressions.txt`441442**CI reproduction**: See [.github/workflows/ci.yml](.github/workflows/ci.yml)443444**Troubleshooting**: Check fiber deadlocks (use `util::fb2` not `std::mutex`), timeout issues (`--test_timeout`), ASAN reports445446---447448## Validation Checklist449450Before claiming a task is complete, verify:451452### Code Quality453454- [ ] Code compiles without errors: `cd build-dbg && ninja dragonfly`455- [ ] Code compiles without warnings (CI uses `-Werror`)456- [ ] Code follows Google C++ Style Guide (run `clang-format`)457- [ ] No new ASAN/UBSAN violations458459### Testing460461- [ ] All existing C++ unit tests pass: `ctest -V -L DFLY`462- [ ] New feature has corresponding test coverage463- [ ] Tests pass in both Debug and Release builds464- [ ] Tests pass with ASAN/UBSAN enabled (if applicable)465- [ ] **DO NOT run codeql_checker** - it's slow and unnecessary for development testing466467### Pre-commit & Style468469- [ ] Pre-commit hooks installed: `pre-commit install`470- [ ] Code formatted with clang-format (C++) and black (Python)471472### Documentation473474- [ ] Public APIs have comments explaining purpose475- [ ] Complex algorithms have explanatory comments476- [ ] README or docs updated if behavior changes477- [ ] No commented-out code left in final commit478479### Performance480481- [ ] No obvious performance regressions (run benchmarks if needed)482- [ ] No unnecessary allocations in hot paths483- [ ] Lock-free data structures used where appropriate484
One repository carrying more than one format is the comparison this product exists for: does anyone actually write different content in each file, or is one a copy of the other?
| Repository | Format | Stack | Covers | Score | Changed |
|---|---|---|---|---|---|
| dragonflydb/dragonfly.github/copilot-instructions.md · 31k | Copilot instructions | stylegitdo-notagent-behaviour | 50/100 | 13 days ago | |
| dragonflydb/dragonfly.github/instructions/code-review.instructions.md · 31k | Copilot instructions | styletesting-strategygitdo-not+2 | 81/100 | 13 days ago |
Same format, overlapping stack, ranked by quality.
| Repository | Format | Stack | Covers | Score | Changed |
|---|---|---|---|---|---|
| unoplat/unoplat-code-confluenceunoplat-code-confluence-frontend/AGENTS.md · 95 | AGENTS.md | setupbuildtestlint-format+6 | 100/100 | 13 days ago | |
| deepseek-ai/deepseek-harnessnative/landlock-run/AGENTS.md · 104k | AGENTS.md | setupteststylearch+3 | 100/100 | today | |
| rails/railsAGENTS.md · 59k | AGENTS.md | teststylearchgit+4 | 100/100 | 14 days ago | |
| vllm-project/vllmAGENTS.md · 89k | AGENTS.md | setuptestlint-formatstyle+5 | 100/100 | 14 days ago | |
| n8n-io/n8npackages/@n8n/agents/AGENTS.md · 201k | AGENTS.md | buildteststylearch+3 | 100/100 | 14 days ago | |
| SkeneTechnologies/skene-cookbookAGENTS.md · 52 | AGENTS.md | setupbuildtestlint-format+7 | 100/100 | 13 days ago | |
| netdata/netdatasrc/go/plugin/ibm.d/AGENTS.md · 80k | AGENTS.md | buildtestlint-formatarch+3 | 99/100 | today | |
| react/react-nativepackages/react-native-compatibility-check/AGENTS.md · 126k | AGENTS.md | testlint-formatstylearch+4 | 99/100 | 14 days ago |
A badge carrying the measured quality of the strongest agent config file in this repository, out of 100. It reads from this index every time somebody loads your page, so it changes when the measurement changes and there is nothing to keep up to date. Free, no account, and the value is not something you or we can set by hand.
[](https://rulestack.kynth.studio/configs/dragonflydb-dragonfly-agents)Would rather not hotlink us? Every badge is also served in shields.io’s endpoint schema, so shields renders the image and your readers never talk to our domain:
Published by Toolproof, the masthead over this index and eight others. The method behind the number is at toolproof.kynth.studio/methodology, and the whole thing is readable as JSON with no key at /api.