AGENTS.md

1# ML contracts library (xpack-core/ml)
2 
3Guidance for coding agents working in the `org.elasticsearch.xpack.core.ml` package inside xpack-core. It holds the *contracts* (serializable config/state POJOs, transport action definitions, registries, cluster-state metadata, versioning), **not** runtime logic. The execution side (transport handlers, REST handlers, persistent-task executors, native processes) lives in `x-pack/plugin/ml` (see its own `AGENTS.md`). The repository-root `AGENTS.md` is authoritative for toolchain, formatting, logging, and transport-version rules — this file does not repeat them.
4 
5## Build & Test
6 
7This code is **not its own Gradle project** — it compiles and tests under `:x-pack:plugin:core` (esplugin `x-pack-core`). Run from the repository root.
8 
9```bash
10# Unit tests for the whole core project (ML tests live under .../core/ml/)
11./gradlew :x-pack:plugin:core:test
12 
13# Single test class / method
14./gradlew :x-pack:plugin:core:test --tests org.elasticsearch.xpack.core.ml.job.config.JobTests
15./gradlew :x-pack:plugin:core:test --tests org.elasticsearch.xpack.core.ml.job.config.JobTests.testMethodName
16 
17# Format / forbidden-API / style gate (run before claiming Java work done)
18./gradlew :x-pack:plugin:core:spotlessApply
19./gradlew :x-pack:plugin:core:precommit
20```
21 
22`x-pack-core` applies `internal-test-artifact`, so the ML plugin and other consumers reuse these test base classes and helpers via `testArtifact(xpackModule('core'))`. Changing a shared test base class here affects downstream test compilation.
23 
24## What lives here vs. the ML plugin
25 
26| Here (`core/ml`) | In `x-pack/plugin/ml` |
27|---|---|
28| `ActionType` definitions + nested `Request`/`Response` | `Transport*Action` handlers, `Rest*Action` handlers |
29| Config/state POJOs (`Job`, `DatafeedConfig`, `DataFrameAnalyticsConfig`, `TrainedModelConfig`, state enums) | Managers, indexers, persistent-task executors, native process I/O |
30| Registries, `MlMetadata`, `MlTasks`, system-index defs | Business logic that reads/writes them |
31 
32Adding an API typically means: define the `ActionType` + `Request`/`Response` (and any config POJO) **here**, then implement the `Transport*Action` and `Rest*Action` in the plugin.
33 
34## Serialization & registry mechanics
35 
36Every contract type implements both `Writeable` (wire) and `ToXContent` (JSON), and ships a matching round-trip test (see Testing).
37 
38**Action class pattern** (e.g. `action/PutJobAction.java`, `action/GetJobsStatsAction.java`):
39 
40```java
41public class FooAction extends ActionType<FooAction.Response> {
42    public static final FooAction INSTANCE = new FooAction();
43    public static final String NAME = "cluster:.../foo";   // scope-encoded name
44    private FooAction() { super(NAME); }
45    public static class Request  extends ... { /* StreamInput ctor, writeTo, toXContent, validate, equals/hashCode */ }
46    public static class Response extends ... { /* same contract */ }
47}
48```
49 
50**Polymorphic types are registered in NamedXContent providers**, not ad hoc. When you add a new inference config, tokenization, data-frame analysis, evaluation metric, or inference result subtype, register it (NamedWriteable + NamedXContent) in the matching provider, or it will not deserialize:
51 
52- `inference/MlInferenceNamedXContentProvider` — inference configs, preprocessors, trained-model/tokenization types, results
53- `dataframe/analyses/MlDataFrameAnalysisNamedXContentProvider` — Classification / Regression / OutlierDetection
54- `dataframe/evaluation/MlEvaluationNamedXContentProvider` — evaluation metrics
55- `ltr/MlLTRNamedXContentProvider` — learning-to-rank configs
56 
57## Parser duality (strict vs. lenient)
58 
59Most configs expose two `ObjectParser`s, and picking the wrong one is a recurring bug source:
60 
61- **`LENIENT_PARSER`** — used when reading **persisted documents or cluster state**. Ignores unknown fields (forward-compat) and parses internal/generated fields (`create_time`, version).
62- **`STRICT_PARSER` / `REST_REQUEST_PARSER`** — used for **REST API input**. Rejects unknown fields (catches client typos) and excludes internal fields (clients cannot set them).
63 
64Configs are immutable with a nested `Builder`; validation runs at `build()` time and throws `ActionRequestValidationException` / `ElasticsearchException`.
65 
66## Versioning & backwards compatibility
67 
68`MlConfigVersion` is a **config-format version, distinct from `TransportVersion`**: it is human-readable and persisted inside documents and cluster state, so it tracks the schema of stored ML configs rather than the wire protocol. Post-8.10 it uses detached incrementing integer ids (the latest is `MlConfigVersion.CURRENT`), and each constant carries a unique id string to avoid duplicate-id git merge collisions; `MlConfigVersionComponent` surfaces it in node info.
69 
70For wire-format changes across mixed-version clusters, follow the root `AGENTS.md` "Backwards compatibility" section — do not restate that workflow here. Any change to a `Writeable`'s `writeTo`/`StreamInput` constructor needs a new named `TransportVersion` gate and a BWC round-trip test. Referable transport-version ids are global (`server/src/main/resources/transport/definitions/referable/*.csv`); after merging `main`, re-check the highest allocated id before keeping a pre-merge id on your branch (`./gradlew :server:generateClusterFeaturesMetadata` fails fast on a duplicate id at class-init).
71 
72## Cluster-state, tasks, system indices
73 
74- `MlMetadata` — `Metadata.ProjectCustom` cluster-state custom; holds `upgradeMode` and `resetMode` flags (parsed leniently for forward-compat).
75- `MlTasks` — persistent-task name constants (`xpack/ml/job`, `xpack/ml/datafeed`, `xpack/ml/data_frame/analytics`, …), task-id prefix helpers, and task matchers/assignment constants.
76- System indices: `MlConfigIndex` (`.ml-config`), `MlMetaIndex` (`.ml-meta`), `MlStatsIndex` (`.ml-stats-*` with the `.ml-stats-write` rollover alias). Bumping mappings means bumping the index mappings version.
77 
78## Domain-model map (`core/ml/<subpackage>/`)
79 
80- `job/` — anomaly-detection `Job`, `AnalysisConfig`, `Detector`, `DataDescription`, model-snapshot & state types.
81- `datafeed/` — `DatafeedConfig`/`DatafeedUpdate`, chunking, delayed-data, cross-project routing, state.
82- `dataframe/` — `DataFrameAnalyticsConfig`, source/dest; `analyses/` (Classification/Regression/OutlierDetection), `evaluation/` (metrics), `stats/`.
83- `inference/` — `TrainedModelConfig`, model definitions; `trainedmodel/` (per-task InferenceConfig + tokenizations + ensemble/tree), `results/` (inference result types), `preprocessing/`.
84- `calendars/` — `Calendar` + `ScheduledEvent` (maintenance windows that suppress anomalies).
85- `annotations/` — `Annotation` + its system index definition.
86- `autoscaling/` — ML node autoscaling policy/decision contracts.
87- `ltr/` — learning-to-rank config contracts.
88- `process/` — native-process result contracts (`DataCounts`, `ModelSizeStats`, `TimingStats`).
89- `stats/` — aggregated usage stats (e.g. `ForecastStats`).
90- `notifications/` — audit-message contracts.
91- `packageloader/` — packaged-model (e.g. ELSER) loader contracts.
92- `vectors/` — dense-vector/embedding metadata.
93- `utils/` — shared validation/serialization helpers (`ExceptionsHelper`, `ToXContentParams`, `NamedXContentObjectHelper`, `RuntimeMappingsValidator`).
94 
95Top-level: `MachineLearningField` (ML settings/constants, `machine-learning` feature family, PLATINUM license), `MachineLearningFeatureSetUsage`.
96 
97## Gotchas
98 
99- **Pick the right parser.** Use `STRICT_PARSER`/`REST_REQUEST_PARSER` for API input and `LENIENT_PARSER` for persisted docs / cluster state. The wrong one either rejects valid stored data or lets clients set internal fields.
100- **Register new polymorphic subtypes.** A new inference config, tokenization, data-frame analysis, evaluation metric, or result type must be added to the matching `Ml*NamedXContentProvider` (NamedWriteable + NamedXContent) or it will not deserialize.
101- **Gate wire changes.** Any change to a `Writeable`'s `writeTo`/`StreamInput` constructor needs a new named `TransportVersion` gate **and** a BWC round-trip test (`AbstractBWC*SerializationTestCase`). Follow the root `AGENTS.md` "Backwards compatibility" workflow.
102- **`MlConfigVersion` ≠ `TransportVersion`.** They version different things (stored config schema vs. wire protocol); don't substitute one for the other, and don't reuse a version constant's unique id string.
103 
104## Testing conventions
105 
106ML contracts are tested almost entirely with serialization round-trip bases — pick by what you're verifying:
107 
108- `AbstractXContentSerializingTestCase` — JSON/XContent round-trip (most common).
109- `AbstractWireSerializingTestCase` — wire (`Writeable`) round-trip.
110- `AbstractBWCSerializationTestCase` / `AbstractBWCWireSerializationTestCase` — backwards-compat across versions; use these when a type's wire/XContent format changed under a version gate.
111- ML-local helpers: `org.elasticsearch.xpack.core.ml.AbstractBWCWireSerializationTestCase`, `AbstractChunkedBWCSerializationTestCase`.
112 
113Every new serialized contract type (or new field on one) needs a round-trip test; format changes need the BWC variant. Tests live under `x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ml/`, mirroring the main-source subpackages.
114