AGENTS.md

1# Elasticsearch Machine Learning C++
2 
3## Toolchain
4- **Language**: C++20 (`CMAKE_CXX_STANDARD 20`).
5- **Build system**: CMake (primary) or Gradle wrapper (`./gradlew`).
6- **Compilers**: GCC 13.3.0 on Linux (built from source, installed to `/usr/local/gcc133/`), Xcode Clang on macOS (Xcode 15.2+ for Ventura/Sonoma), Visual Studio 2022 Professional (MSVC) on Windows.
7- **Key dependencies**: Boost 1.86.0 (dynamic linking, includes Boost.Json for JSON handling), PyTorch 2.7.1 (libtorch), libxml2.
8- **Header-only libraries**: Eigen and valijson are header-only and managed by the `3rd_party/` CMake system (pulled automatically during configuration).
9- **Platforms**: Linux x86_64/aarch64, macOS aarch64, Windows x86_64.
10- **Toolchain files**: Auto-selected from `cmake/<os>-<arch>.cmake` or set via `CMAKE_TOOLCHAIN_FILE`.
11 
12## Build & Run Commands
13 
14Configure and build (default `RelWithDebInfo`):
15```
16cmake -B cmake-build-relwithdebinfo
17cmake --build cmake-build-relwithdebinfo -j$(nproc)
18```
19 
20Or via Gradle:
21```
22./gradlew :compile
23```
24 
25Set `ML_DEBUG=1` to switch to a Debug build. Compiler caching (sccache/ccache) is auto-detected.
26 
27Refer to `CONTRIBUTING.md` and the `build-setup/` directory for full platform-specific setup instructions.
28 
29## Project Structure
30 
31```
32bin/                    # Application executables
33  autodetect/           #   Anomaly detection
34  categorize/           #   Log categorization
35  controller/           #   Process lifecycle controller
36  data_frame_analyzer/  #   Data frame analytics (classification, regression)
37  normalize/            #   Anomaly score normalization
38  pytorch_inference/    #   PyTorch model inference
39lib/                    # Shared libraries
40  api/                  #   JSON/REST API layer
41  core/                 #   Platform abstractions, I/O, logging, compression
42  maths/                #   Mathematical and statistical algorithms
43    analytics/          #     Boosted tree, data frame analytics
44    common/             #     Bayesian optimisation, distributions, time series
45    time_series/        #     Time series decomposition, forecasting
46  model/                #   Anomaly detection models
47  seccomp/              #   Seccomp/sandbox filters
48  test/                 #   Shared test utilities (CBoostTestXmlOutput, etc.)
49  ver/                  #   Version information
50include/                # Public headers (mirrors lib/ structure)
513rd_party/              # Header-only third-party libraries (Eigen, valijson), licenses
52cmake/                  # CMake toolchain files, helper functions, test runners
53build-setup/            # Platform-specific build environment instructions
54.buildkite/             # CI pipeline definitions (Buildkite)
55.ci/                    # Packer scripts for building Orka macOS CI VMs
56.github/workflows/      # GitHub Actions (automatic backport)
57dev-tools/              # Developer scripts (clang-format, benchmarks)
58```
59 
60Libraries must not have circular dependencies. The dependency order is roughly:
61`core` -> `maths` -> `model` -> `api` -> `bin/*`.
62 
63## CMake Helper Functions
64 
65The build uses custom CMake functions defined in `cmake/functions.cmake`. Use these instead of raw `add_library`/`add_executable` — they handle platform-specific sources, linking, installation, and Windows resource generation automatically.
66 
67### Adding a Shared Library
68 
69Set `ML_LINK_LIBRARIES` then call `ml_add_library`:
70 
71```cmake
72project("ML MyLib")
73 
74set(ML_LINK_LIBRARIES
75  ${Boost_LIBRARIES}
76  MlCore
77  )
78 
79ml_add_library(MlMyLib SHARED
80  CMyClass.cc
81  CMyOtherClass.cc
82  )
83```
84 
85Libraries are named with the `Ml` prefix (e.g. `MlCore`, `MlModel`). The function handles shared library versioning, RPATH, and installation. Use `SHARED` for distributed libraries or `STATIC` for internal-only ones.
86 
87For libraries that should not be installed/distributed (e.g. internal helpers), use `ml_add_non_distributed_library` instead.
88 
89### Adding an Executable
90 
91Set `ML_LINK_LIBRARIES` then call `ml_add_executable`. A `Main.cc` file is included automatically — do not list it in the sources:
92 
93```cmake
94project("ML MyApp")
95 
96set(ML_LINK_LIBRARIES
97  ${Boost_LIBRARIES}
98  MlCore
99  MlApi
100  MlVer
101  )
102 
103ml_add_executable(myapp
104  CCmdLineParser.cc
105  )
106```
107 
108The function creates a companion OBJECT library (`MlMyApp`) from the listed sources, which test executables can link against. The executable itself always builds from `Main.cc` plus those objects.
109 
110For executables not intended for distribution (dev tools, benchmarks), use `ml_add_non_distributed_executable`.
111 
112### Adding a Test Executable
113 
114Test executables live in `unittest/` subdirectories. Set `ML_LINK_LIBRARIES` (including `${Boost_LIBRARIES_WITH_UNIT_TEST}` and `MlTest`), then call `ml_add_test_executable`:
115 
116```cmake
117project("ML MyLib unit tests")
118 
119set(SRCS
120  CMyClassTest.cc
121  CMyOtherClassTest.cc
122  Main.cc
123  )
124 
125set(ML_LINK_LIBRARIES
126  ${Boost_LIBRARIES_WITH_UNIT_TEST}
127  MlCore
128  MlMyLib
129  MlTest
130  )
131 
132ml_add_test_executable(mylib ${SRCS})
133```
134 
135The `_target` argument (e.g. `mylib`) is used to derive the test executable name (`ml_test_mylib`) and the CMake targets `test_mylib` and `test_mylib_individually`.
136 
137### Registering Tests with the Build
138 
139After creating the test executable, register it in `test/CMakeLists.txt` by adding an `ml_add_test` call alongside the existing entries:
140 
141```cmake
142ml_add_test(lib/core/unittest core)
143ml_add_test(lib/maths/common/unittest maths_common)
144ml_add_test(lib/maths/time_series/unittest maths_time_series)
145ml_add_test(lib/maths/analytics/unittest maths_analytics)
146ml_add_test(lib/model/unittest model)
147ml_add_test(lib/api/unittest api)
148ml_add_test(lib/ver/unittest ver)
149ml_add_test(lib/seccomp/unittest seccomp)
150ml_add_test(bin/controller/unittest controller)
151ml_add_test(bin/pytorch_inference/unittest pytorch_inference)
152ml_add_test(lib/mylib/unittest mylib)          # <-- new entry
153```
154 
155The first argument is the relative path to the unittest directory; the second is the target name matching `ml_add_test_executable`. Note how nested libraries use underscores in the target name (e.g. `lib/maths/common/unittest` -> `maths_common`).
156 
157### Platform-Specific Sources
158 
159If a source file has a platform-specific variant (e.g. `CMyClass_Linux.cc`, `CMyClass_Darwin.cc`), the `ml_generate_platform_sources` function (called internally by all `ml_add_*` functions) will automatically substitute the platform-specific file at build time. Just list the base filename (`CMyClass.cc`) in your sources.
160 
161## Testing
162 
163Tests use the **Boost.Test** framework. Each library and application has a `unittest/` subdirectory containing test files and a `Main.cc` entry point.
164 
165### Running Tests
166 
167Run all tests:
168```
169cmake --build cmake-build-relwithdebinfo -t test
170```
171 
172Run tests for a specific library:
173```
174cmake --build cmake-build-relwithdebinfo -t test_core
175cmake --build cmake-build-relwithdebinfo -t test_model
176```
177 
178Run specific test cases (wildcards supported):
179```
180TESTS="*/testPersist" cmake --build cmake-build-relwithdebinfo -t test_model
181```
182 
183Run tests individually in separate processes (better isolation, per-suite parallelism):
184```
185cmake --build cmake-build-relwithdebinfo -j8 -t test_individually
186cmake --build cmake-build-relwithdebinfo -j8 -t test_api_individually
187```
188 
189Run all test cases from all suites in a single CTest invocation (optimal cross-suite parallelism):
190```
191cmake --build cmake-build-relwithdebinfo -t test_all_parallel
192```
193 
194Pass extra flags to the Boost.Test runner:
195```
196TEST_FLAGS="--random" cmake --build cmake-build-relwithdebinfo -t test
197```
198 
199### Precommit (format + test)
200 
201```
202cmake --build cmake-build-relwithdebinfo -j8 -t precommit
203```
204Or: `./gradlew precommit`
205 
206### Writing Tests
207 
208- Test files are named `CClassNameTest.cc` and placed in `lib/<module>/unittest/` or `bin/<module>/unittest/`.
209- Each test file uses `BOOST_AUTO_TEST_SUITE(CClassNameTest)` / `BOOST_AUTO_TEST_CASE(testMethodName)`.
210- Use real classes over mocks wherever possible. Tests should reflect real-world usage.
211- Every class should have a corresponding test suite; every public method should have a test.
212- Test cases must be completely independent from one another — they may be run in parallel across separate processes, so they must not depend on execution order or share mutable state.
213 
214## Formatting & Style
215 
216Code is formatted with `clang-format` (LLVM-based style, 4-space indent). Run before committing:
217```
218cmake --build cmake-build-relwithdebinfo -t format
219```
220Or: `./gradlew format`
221 
222The CI pipeline enforces formatting via the `check-style` step; PRs that fail formatting will not pass CI.
223 
224The full coding standard is in `STYLEGUIDE.md`. Key points:
225 
226### Naming Conventions
227- Classes: `CClassName`, Structs: `SStructName`, Enums: `EEnumName`
228- Member variables: `m_ClassMember`, `s_StructMember`
229- Static members: `ms_ClassStatic`
230- Methods: `methodName` (camelCase)
231- Type aliases: `TTypeName` (e.g. `using TDoubleVec = std::vector<double>`)
232- Constants: `CONSTANT_NAME`
233- Non-boolean accessors: `clientId` (not `getClientId`)
234- Boolean accessors: `isComplete` (not `complete`)
235- Files: `CClassName.cc` / `CClassName.h`
236 
237### Code Conventions
238- Use `nullptr`, never `0` or `NULL`.
239- No exceptions — use return codes for error handling. Catch third-party exceptions at the smallest scope.
240- No `assert()`. No C-style casts. No macros unless unavoidable.
241- Prefer smart pointers over raw pointers; prefer references over pointers.
242- Scope member function calls with `this->`.
243- Use `auto` when the type is obvious; avoid it when the type is unclear.
244- Prefer `emplace_back` over `push_back`, range-based for loops, and uniform initializers.
245- `override` must be used consistently; `virtual` must not appear alongside `override`.
246 
247### File Layout
248- Implementation files (`.cc`): own header first, then other ML headers, third-party headers, standard library headers.
249- Group includes by library with blank lines between groups (clang-format will sort within groups).
250- Use unnamed namespaces in `.cc` files for file-local helpers, not private class members.
251- Forward-declare classes in headers rather than including their headers.
252 
253### Documentation
254- Doxygen comments (exclamation mark style: `//!`) are required for all header files and public/protected methods.
255- Implementation files use regular C++ comments, not Doxygen.
256- Focus comments on the "why", not the "what".
257 
258## License Headers
259 
260All source files must include the Elastic License 2.0 header. Copy from `copyright_code_header.txt` or any existing source file.
261 
262## CI
263 
264CI runs on **Buildkite** (`ml-cpp-pr-builds`). The pipeline builds and tests on all platforms (Linux x86_64, Linux aarch64, macOS aarch64, Windows x86_64) in both `RelWithDebInfo` and Debug configurations. It also runs:
265- `clang-format` style validation
266- Snyk security/license scanning
267- Java integration tests against Elasticsearch
268 
269Automatic backporting is handled by a GitHub Action (`.github/workflows/backport.yml`) — add version labels (e.g. `v9.3.0`) to a PR and a backport PR is created automatically when it merges.
270 
271## Pull Requests
272 
273- Title must be prefixed with `[ML]` (e.g. `[ML] Fix anomaly scoring edge case`).
274- Label with `:ml` (mandatory), a type label (`>bug`, `>enhancement`, `>feature`, `>refactoring`, `>test`, `>docs`), and version labels for applicable releases.
275- Squash-and-merge is the standard merge strategy; keep commits clean for review but don't squash manually.
276- Backports start after merging to `main`. Add version labels to trigger automatic backport PRs.
277 
278## Best Practices for Automation Agents
279 
280- Always read existing code before editing to understand patterns and conventions.
281- Never edit unrelated files; keep diffs tightly scoped.
282- Run `clang-format` before presenting any code changes.
283- Match the naming conventions exactly — the prefixes (`C`, `m_`, `ms_`, `T`, `E`) are strictly followed throughout the codebase.
284- When adding new classes, follow the existing directory and namespace structure. Production code in `lib/foo/` uses namespace `ml::foo`.
285- When adding tests, place them in the corresponding `unittest/` directory and register them in the `CMakeLists.txt`.
286- Do not introduce new third-party dependencies without discussion.
287- Do not add AI attribution trailers (e.g. `Co-Authored-By`) to commit messages.
288- Commit messages should follow the `[ML] Summary of change` format.
289- If unsure about a convention, check a nearby file for the established pattern — consistency with surrounding code is the highest priority.
290 
291Stay aligned with `CONTRIBUTING.md`, `STYLEGUIDE.md`, and the `build-setup/` guides; this AGENTS file summarizes but does not replace those authoritative documents.
292