| Dimension | Shared | Only in A | Only in B | Overlap |
|---|---|---|---|---|
| Sections | 0 | 16 | 28 | 0% |
| Commands | 0 | 7 | 9 | 0% |
| Section tags | 6 | 1 | 4 | 55% |
What each file covers
Sections
0 shared · 16 only in A · 28 only in B- − Inference API plugin (x-pack-inference)
- − This plugin vs. the ML plugin — don't confuse them
- − Where the SPI lives (important)
- − Build & Test Commands
- − Unit tests
- − Internal cluster tests (*IT under ...inference.integration)
- − YAML REST tests (specs in src/yamlRestTest/resources/rest-api-spec/test/inference/)
- − Format / forbidden-API / style gate
- − QA suites (`qa/`)
- − Request lifecycle
- − Adding a new external service integration
- − External HTTP layer (`external/`) — traced flow
- − Registry & system indices
- − semantic_text & the search path
- − Gotchas
- − Testing conventions specific to inference
- + Elasticsearch
- + Toolchain Snapshot
- + Build & Run Commands
- + Verification & Lint Tasks
- + Project Structure
- + Stateless Elasticsearch
- + Plugin `deploymentTarget`
- + Plugin locations
- + Key subsystems
- + Testing Cheatsheet
- + Test Types
- + Distribution selection for external-module tests
- + Dependency Hygiene
- + Entitlement Policy
- + Formatting & Imports
- + Types, Generics, and Suppressions
- + Naming Conventions
- + Logging & Error Handling
- + Javadoc & Comments
- + License Headers
- + Generated Files
- + Debugging Missing Tests
- + `No tests found for given includes: [**/*$*.class]`
- + Best Practices for Automation Agents
- + Methods with Required Javadoc Reading
- + ES|QL tests
- + Backwards compatibility
- + Documentation
Commands
0 shared · 7 only in A · 9 only in B- − ./gradlew :x-pack:plugin:inference:test
- − ./gradlew :x-pack:plugin:inference:test --tests "org.elasticsearch.xpack.inference.ModelConfigurationsTests"
- − ./gradlew :x-pack:plugin:inference:test --tests "org.elasticsearch.xpack.inference.ModelConfigurationsTests.testSerialization"
- − ./gradlew :x-pack:plugin:inference:internalClusterTest
- − ./gradlew :x-pack:plugin:inference:yamlRestTest
- − ./gradlew :x-pack:plugin:inference:spotlessApply
- − ./gradlew :x-pack:plugin:inference:precommit
- + ./gradlew
- + ./gradlew spotlessJavaCheck
- + ./gradlew test
- + ./gradlew :server:test
- + ./gradlew :server:test --tests org.elasticsearch.package.ClassName
- + ./gradlew :server:test --tests 'org.elasticsearch.package.*'
- + ./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N
- + ./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.CsvIT.*<csv-file>*"
- + ./gradlew generateTransportVersion
Section tags
6 shared · 1 only in A · 4 only in B- − api
- + architecture
- + types
- + git-pr
- + docs
- build
- test
- lint-format
- code-style
- testing-strategy
- do-not
Line diff
elastic/elasticsearch · x-pack/plugin/inference/AGENTS.md
@@ −1 @@
1# Inference API plugin (x-pack-inference)
2
3Guidance for coding agents working in `x-pack/plugin/inference/` (Gradle `:x-pack:plugin:inference`, esplugin artifact `x-pack-inference`, class `InferencePlugin`). It implements the `_inference` API, integrates many external/internal inference services (see the provider subdirs under `services/`), and provides `semantic_text` and inference-based reranking. The repository-root `AGENTS.md` is authoritative for toolchain, formatting, logging, transport-version, and general testing conventions — this file does not repeat them. Read it first.
4
5## This plugin vs. the ML plugin — don't confuse them
6
7These two plugins both say "inference" but do different things:
8
9| | `x-pack/plugin/inference` (this plugin) | `x-pack/plugin/ml` |
10|---|---|---|
11| Scope | The `_inference` endpoint: register an *inference endpoint*, call out to a service, get embeddings/completions/reranks | Anomaly detection, datafeeds, data-frame analytics, **local trained-model deployment** (PyTorch via `ml-cpp`) |
12| Compute | Mostly **calls external HTTP APIs** (OpenAI, Cohere, Bedrock, …); the `elasticsearch` service runs models locally | Runs native processes on ML nodes |
13| Key feature | `semantic_text` field, semantic query, reranking | jobs, datafeeds, DFA, trained-model assignment |
14
15**The seam:** the internal `elasticsearch` service (`services/elasticsearch/`, e.g. ELSER and locally-hosted models) delegates to **ML-deployed trained models** — so a request through this plugin can end up running on the ML plugin's deployment infrastructure. The shared SPI lives in **server** (`org.elasticsearch.inference`); trained-model POJOs live in `xpack.core.ml.inference`.
16
17## Where the SPI lives (important)
18
19The core inference contracts are **not** in this plugin and **not** in xpack-core — they are in **server** at `server/src/main/java/org/elasticsearch/inference/`: `InferenceService`, `InferenceServiceRegistry`, `Model`, `ModelConfigurations`, `ServiceSettings`/`TaskSettings`/`SecretSettings`, `TaskType`, `InputType`, `InferenceServiceResults`, `ChunkingSettings`, `InferenceServiceExtension`. This plugin provides the *implementations*; other plugins integrate via `InferenceServiceExtension` (loaded through `ExtensiblePlugin`).
20
21## Build & Test Commands
22
23Gradle project `:x-pack:plugin:inference`. Run from repo root. `extendedPlugins = ['x-pack-core']`. Heavy third-party deps (AWS SDK v2, Google auth/api-client, Azure identity/msal4j, Nimbus OAuth/JOSE, Jackson, Netty, Reactor) — check `build.gradle` and the thirdPartyAudit allowances before adding more.
24
25```bash
26# Unit tests
27./gradlew :x-pack:plugin:inference:test
28./gradlew :x-pack:plugin:inference:test --tests "org.elasticsearch.xpack.inference.ModelConfigurationsTests"
29./gradlew :x-pack:plugin:inference:test --tests "org.elasticsearch.xpack.inference.ModelConfigurationsTests.testSerialization"
30
31# Internal cluster tests (*IT under ...inference.integration)
32./gradlew :x-pack:plugin:inference:internalClusterTest
33
34# YAML REST tests (specs in src/yamlRestTest/resources/rest-api-spec/test/inference/)
35./gradlew :x-pack:plugin:inference:yamlRestTest
36
37# Format / forbidden-API / style gate
38./gradlew :x-pack:plugin:inference:spotlessApply
39./gradlew :x-pack:plugin:inference:precommit
40```
41
42Unit-test base classes: `ESTestCase`, `AbstractWireSerializingTestCase`, `AbstractBWCWireSerializationTestCase`, and `MapperTestCase`/`MapperServiceTestCase` (for `semantic_text`).
43
44### QA suites (`qa/`)
45
46| Suite | Command | Purpose |
47|---|---|---|
48| `test-service-plugin` | (cluster plugin, consumed by others) | **Mock inference service** (`TestInferenceServicePlugin`) — the way to write service-level REST tests without real API keys |
49| `inference-service-tests` | `:qa:inference-service-tests:javaRestTest` | provides `InferenceBaseRestTest` (`putModel`/`infer`/`deleteModel` helpers) |
50| `inference-with-security` | `:qa:inference-with-security:yamlRestTest` | security enabled |
51| `multi-node` | `:qa:multi-node:yamlRestTest` | multi-node cluster |
52| `oauth2` | `:qa:oauth2:javaRestTest` | in-process mock OAuth2 server (skipped on FIPS) |
53| `mixed-cluster`, `rolling-upgrade` | `:qa:<name>:vX.Y.Z#javaRestTest` | BWC / rolling-upgrade |
54
55## Request lifecycle
56
57REST handler (`rest/`) → Transport action (`action/`, base `BaseTransportInferenceAction`, with license check) → `ModelRegistry` lookup of the endpoint config → `InferenceServiceRegistry` dispatch to the concrete service → service runs inference (external HTTP or local) → `InferenceServiceResults` back to the listener. Streaming uses the chunked/streaming variants.
58
59## Adding a new external service integration
60
611. Create `services/<name>/`. Implement the `Model` plus its `<Name>ServiceSettings` (non-secret config), `<Name>TaskSettings` (per-request knobs), and `<Name>SecretSettings` (API keys) — subtypes of the server SPI settings interfaces (base helpers in `services/settings/`).
622. Extend `SenderService` (`services/SenderService.java`, the base `InferenceService` impl) with a concrete `<Name>Service`; implement the `doInfer`/`doChunkedInfer`/`doUnifiedCompletionInfer` abstract methods.
633. Provide a `RequestManager` (builds the outbound HTTP request) and a service-specific `ResponseHandler` (parses the response into `InferenceServiceResults`).
644. Register the service factory in `InferencePlugin` and any `NamedWriteable`s in `InferenceNamedWriteablesProvider`.
655. Respect `ConfigurationParseContext` — `REQUEST` (strict, user API input) vs `PERSISTENT` (lenient, loaded from the index). Use `ServiceUtils`/`ServiceFields` helpers for map extraction and validation.
66
67## External HTTP layer (`external/`) — traced flow
68
69For services that call out over HTTP, a request flows:
70
711. `SenderService` calls `Sender.send(requestManager, inferenceInputs, timeout, listener)`.
722. `HttpRequestSender` lazily starts the async client/executor on first use, then enqueues to `RequestExecutorService`.
733. The executor applies **per-service rate limiting** and dequeues onto the `inference_utility` thread pool.
744. `RequestManager.buildRequest(...)` produces the outbound HTTP request.
755. `RetryingHttpSender` executes it with backoff on 429/5xx/timeouts.
766. `HttpClientManager` (Apache `HttpAsyncClient` connection pool, idle-eviction, XPack SSL) does the actual call.
777. The service's `ResponseHandler` parses the response into `InferenceServiceResults`, returned via the listener (responses delivered on the `inference_response` pool).
78
79## Registry & system indices
80
81`registry/ModelRegistry` persists endpoint configurations and syncs them from cluster state for fast `getModel(inferenceId)` lookups. Three system indices:
82
83- `InferenceIndex` → `.inference` — endpoint configs (service, task_type, service_settings, task_settings, chunking_settings). Root mappings are strict; the settings sub-objects are `dynamic:false` to allow service-specific fields.
84- `InferenceSecretsIndex` → `.inference-secrets` — credentials, kept in a **separate index** from config so secrets can be secured/backed-up independently.
85- An Elastic-Inference-Service cloud-connected-mode index for EIS auth config.
86
87## semantic_text & the search path
88
89`semantic_text` makes embedding generation automatic across ingest and search:
90
91- **Mapping:** `mapper/SemanticTextFieldMapper` defines the field and auto-creates dense- or sparse-vector subfields (plus chunk/text metadata) based on the referenced endpoint's task type.
92- **Ingest:** `action/filter/ShardBulkInferenceActionFilter` intercepts bulk shard requests, batches inference over the configured `inference_fields` (chunking long text, respecting `INDICES_INFERENCE_BATCH_SIZE`), and injects the embeddings into the document source before normal indexing.
93- **Search:** `queries/SemanticQueryBuilder` and the `Semantic{Knn,Match,SparseVector}QueryRewriteInterceptor`s auto-embed the query text at rewrite time and rewrite to native vector/sparse/match queries. `highlight/SemanticTextHighlighter` highlights matched chunks.
94- **Reranking:** `rank/textsimilarity/TextSimilarityRankBuilder` reranks first-pass hits through a rerank endpoint.
95
96## Gotchas
97
98- **Register on both sides.** A new service/model/results type must be registered as a service factory in `InferencePlugin` **and** as `NamedWriteable`(s) in `InferenceNamedWriteablesProvider`, or it silently fails to deserialize from cluster state / the index.
99- **Right parse context.** Honor `ConfigurationParseContext`: `REQUEST` (strict, rejects unknown fields — user API input) vs `PERSISTENT` (lenient — loaded from `.inference`). Using the wrong one breaks either client validation or forward-compat reads.
100- **Secrets stay separate.** Credentials are written to `.inference-secrets`, never `.inference`. Don't fold secret settings into the config document.
101- **SPI is in server.** The `InferenceService`/`Model`/settings/`TaskType` contracts live in server's `org.elasticsearch.inference`, not this plugin and not xpack-core — extend those, don't fork them.
102- **Don't block.** External calls go through the async `Sender`/`HttpRequestSender` path on the inference thread pools; don't call services synchronously from a transport or cluster-state thread.
103
104## Testing conventions specific to inference
105
106- Serialization round-trips use the wire/BWC bases; `semantic_text` mapper tests use `MapperTestCase`/`MapperServiceTestCase`.
107- To test service behavior end-to-end without real credentials, depend on `qa/test-service-plugin` (the mock `TestInferenceServicePlugin`) and extend `InferenceBaseRestTest` from `qa/inference-service-tests`.
108- Internal cluster tests are named `*IT` under `...inference.integration`; YAML specs live in `src/yamlRestTest/resources/rest-api-spec/test/inference/`.
109
elastic/elasticsearch · AGENTS.md
@@ +1 @@
1# Elasticsearch
2
3## Toolchain Snapshot
4- **Java**: JDK 25 via `JAVA_HOME`; use the bundled Gradle wrapper (`./gradlew`).
5- **Build tooling**: Gradle composite build with `build-conventions`, `build-tools`, and `build-tools-internal`; Docker is required for some packaging/tests.
6- **OS packages**: Packaging and QA jobs expect ephemeral hosts; do not run packaging suites on your workstation.
7- **Security**: Default dev clusters enable security; use `elastic-admin:elastic-password` or disable with `-Dtests.es.xpack.security.enabled=false`.
8- **Cursor/Copilot rules**: None provided in repo; follow this guide plus CONTRIBUTING.md.
9
10## Build & Run Commands
11- Refer to BUILDING.md, CONTRIBUTING.md & TESTING.asciidoc for comprehensive build/test instructions.
12
13## Verification & Lint Tasks
14- `./gradlew spotlessJavaCheck` / `spotlessApply` (or `:server:spotlessJavaCheck`): enforce formatter profile in `build-conventions/formatterConfig.xml`.
15- `spotlessApply` also prunes unused imports and reorders imports automatically. Run it instead of manually hunting for unused imports after refactoring.
16
17## Project Structure
18The repository is organized into several key directories:
19* `server`: The core Elasticsearch server. Few third-party dependencies (Lucene plus a handful of small libraries). Key `org.elasticsearch` sub-packages: `cluster` (cluster state machine), `index` (per-index logic), `search` (query execution), `action` (transport actions), `snapshots` (snapshot/restore), plus `indices`, `repositories`, `rest`, `ingest`, etc.
20* `modules`: Features shipped with Elasticsearch by default, but not considered "core" server code. Many modules provide a specific implementation of a pluggable interface defined in `server`, such as `transport-netty4` (the transport layer) or `repository-s3`/`repository-gcs`/`repository-azure` (snapshot repositories). Others integrate with external systems, such as `apm` (Application Performance Monitoring agent integration).
21* `plugins`: Optional, not bundled by default, but officially supported. Examples: `discovery-ec2`/`discovery-gce`/`discovery-azure-classic` (cloud-aware cluster discovery).
22* `libs`: Internal libraries used by multiple parts of the project. Examples: `logging`, `x-content` (JSON/CBOR/YAML/SMILE parsing abstraction).
23* `client`: The official Java REST client.
24* `test`: Test infrastructure used by the rest of the repo. `framework` holds `ESTestCase`/`ESIntegTestCase`/`ESSingleNodeTestCase`; also `test-clusters` and `yaml-rest-runner` (runner for YAML-based REST API tests).
25* `qa`: Integration and multi-version tests. Examples: `rolling-upgrade`, `mixed-cluster`.
26* `rest-api-spec`: JSON spec definitions for the public REST API endpoints.
27* `docs`: Project documentation.
28* `distribution`: Logic for building distribution packages.
29* `x-pack`: Modules, plugins, and commercial features under the Elastic License 2.0. Example sub-plugins: `security`, `ml` (machine learning), `ccr` (cross-cluster replication), `logsdb` (optimized index mode for log data), and `stateless`.
30* `build-conventions`, `build-tools`, `build-tools-internal`: Gradle build logic. Refer to BUILDING.md for details on how these are structured and used.
31
32## Stateless Elasticsearch
33
34Stateless Elasticsearch is a distribution where shard data is stored in an **object store** (e.g., S3, GCS, Azure) rather than local disk. Nodes carry no durable local state. The cluster distinguishes two node roles: **indexing nodes** (`index` role, write path + translog replication to object store) and **search nodes** (`search` role, read-only via shared blob cache). The `DiscoveryNode.STATELESS_ENABLED_SETTING` gates stateless behavior at runtime.
35
36### Plugin `deploymentTarget`
37
38Plugins can set `deploymentTarget` in `build.gradle`. That value tells the node **whether to load the plugin**: **`STATEFUL_ONLY`** (stateful clusters only), **`STATELESS_ONLY`** (stateless mode on only), or **`ALL`** (always loaded; this is the default when the property is omitted).
39
40### Plugin locations
41
42| Plugin | Gradle path | Purpose |
43|---|---|---|
44| `stateless` | `:x-pack:plugin:stateless` | Core stateless — engines, allocation, cache, object store, recovery |
45| `stateless-sigterm` | `:x-pack:plugin:stateless-sigterm` | Clean SIGTERM shutdown for Kubernetes |
46| `stateless-master-failover` | `:x-pack:plugin:stateless-master-failover` | Master failover behavior |
47| `stateless-no-wait-for-active-shards` | `:x-pack:plugin:stateless-no-wait-for-active-shards` | Suppresses wait-for-active-shards |
48| `stateless-health-shards-availability` | `:x-pack:plugin:stateless-health-shards-availability` | Shard availability health indicators |
49
50**Package**: `org.elasticsearch.xpack.stateless.*` throughout.
51
52### Key subsystems
53
54- **Object store** (`objectstore/`): `ObjectStoreService`, bucket config, GC tasks for stale indices and translogs.
55- **Commits** (`commits/`): `StatelessCommitService` manages shard commits to blob store; `HollowShardsService` manages hollow indexing shards.
56- **Cache & prewarming** (`cache/`): `StatelessSharedBlobCacheService`, online prewarming, `SearchCommitPrefetcher`.
57- **Engines** (`engine/`): `IndexEngine` (write path) and `SearchEngine` (read-only); `TranslogReplicator` replicates translog to object store.
58- **Allocation** (`allocation/`): `StatelessExistingShardsAllocator`, separate balancing weights per tier, heap-usage-aware allocation decisions.
59- **Recovery** (`recovery/`): custom primary relocation and unpromotable shard relocation protocols.
60
61## Testing Cheatsheet
62- Standard suite: `./gradlew test` (respects cached results; add `-Dtests.timestamp=$(date +%s)` to bypass caches when reusing seeds).
63- Single project: `./gradlew :server:test` (or other subproject path).
64- Single class: `./gradlew :server:test --tests org.elasticsearch.package.ClassName`.
65- Single package: `./gradlew :server:test --tests 'org.elasticsearch.package.*'`.
66- Single method / repeated runs: `./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N`.
67- Deterministic seed: append `-Dtests.seed=DEADBEEF` (each method uses derived seeds).
68- JVM tuning knobs: `-Dtests.jvms=8`, `-Dtests.heap.size=4G`, `-Dtests.jvm.argline="-verbose:gc"`, `-Dtests.output=always`, etc.
69- Debugging: append `--debug-jvm` to the Gradle test task and attach a debugger on port 5005.
70- CI reproductions: copy the `REPRODUCE WITH` line from CI logs; it includes project path, seed, and JVM flags.
71- Yaml REST tests: `./gradlew ":rest-api-spec:yamlRestTest" --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=<relative_test_file_path>}"`
72- ES|QL CSV tests: `./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.CsvIT.*<csv-file>*"` (e.g. `--tests "...CsvIT.*stats_first_last*"`); append `*<test-name>*` to target a single test within the file.
73- Use the Elasticsearch testing framework where possible for unit and yaml tests and be consistent in style with other elasticsearch tests.
74- Use real classes over mocks or stubs for unit tests, unless the real class is complex then either a simplified subclass should be created within the test or, as a last resort, a mock or stub can be used. Unit tests must be as close to real-world scenarios as possible.
75- Ensure mocks or stubs are well-documented and clearly indicate why they were necessary.
76
77### Test Types
78- Unit Tests: Preferred. Extend `ESTestCase`.
79- Single Node: Extend `ESSingleNodeTestCase` (lighter than full integ test).
80- Integration: Extend `ESIntegTestCase`.
81- REST API: Extend `ESRestTestCase` or `ESClientYamlSuiteTestCase`. **YAML based REST tests are preferred** for integration/API testing.
82
83### Distribution selection for external-module tests
84- Prefer the OSS/minimal distribution over `usesDefaultDistribution` whenever possible. `usesDefaultDistribution` packages the full default distribution, which is significantly more expensive to build and run.
85- Only use `usesDefaultDistribution` when the test genuinely requires a feature that is only available in the default distribution and cannot be replicated with a custom cluster configuration that includes just the needed plugins. Always document the reason in the `usesDefaultDistribution(...)` message.
86
87## Dependency Hygiene
88- Never add a dependency without checking for existing alternatives in the repo.
89
90## Entitlement Policy
91- Never add an entitlement speculatively. Each entry in `entitlement-policy.yaml` must have a specific justification — ideally a concrete `NotEntitledException` that was observed, or at minimum a clear explanation of why the library requires that capability. Entitlements are a least-privilege mechanism; granting one "just in case" defeats the purpose.
92- Every use of `ESTestCase.WithoutEntitlements` must be accompanied by a comment explaining why the entitlement failure is spurious in the test context and would not occur in production.
93
94## Formatting & Imports
95- Absolutely no wildcard imports; keep existing import order and avoid reordering untouched lines.
96- In `switch` statements, do not use `default` as a branch for valid or expected options. Enumerate those cases explicitly and reserve `default` for throwing an exception for unexpected values, or an assertion error if this code branch is unreachable.
97
98## Types, Generics, and Suppressions
99- Prefer type-safe constructs; avoid raw types and unchecked casts.
100- If suppressing warnings, scope `@SuppressWarnings` narrowly (ideally a single statement or method).
101- Document non-obvious casts or type assumptions via Javadoc/comments for reviewers.
102
103## Naming Conventions
104- REST handlers typically use the `Rest*Action` pattern; transport-layer handlers mirror them with `Transport*Action` classes.
105- REST classes expose routes via `RestHandler#routes`; when adding endpoints ensure naming matches existing REST/Transport patterns to aid discoverability.
106- Transport `ActionType` strings encode scope (`indices:data/read/...`, `cluster:admin/...`, etc.); align new names with these conventions to integrate with privilege resolution.
107
108## Logging & Error Handling
109- Elasticsearch should prefer its own logger `org.elasticsearch.logging.LogManager` & `org.elasticsearch.logging.Logger`; declare `private static final Logger logger = LogManager.getLogger(Class.class)`.
110- Always use parameterized logging (`logger.debug("operation [{}]", value)`); never build strings via concatenation.
111- Wrap expensive log-message construction in `() -> Strings.format(...)` suppliers when logging at `TRACE`/`DEBUG` to avoid unnecessary work.
112- Log levels:
113 - `TRACE`: highly verbose developer diagnostics; usually read alongside code.
114 - `DEBUG`: detailed production troubleshooting; ensure volume is bounded.
115 - `INFO`: default-enabled operational milestones; prefer factual language.
116 - `WARN`: actionable problems users must investigate; include context and, if needed, exception stack traces.
117 - `ERROR`: reserve for unrecoverable states (e.g., storage health failures); prefer `WARN` otherwise.
118- Only log client-caused exceptions when the cluster admin can act on them; otherwise rely on API responses.
119- Tests can assert logging via `MockLog` for complex flows.
120
121## Javadoc & Comments
122- New packages/classes/public or abstract methods require Javadoc explaining the "why" rather than the implementation details.
123- Avoid documenting trivial getters/setters; focus on behavior, preconditions, or surprises.
124- For tests, Javadoc can describe scenario setup/expectations to aid future contributors.
125- Do not remove existing comments from code unless the code is also being removed or the comment has become incorrect.
126
127## License Headers
128- Default header (outside `x-pack`): Elastic License 2.0, SSPL v1, or AGPL v3—they are already codified at the top of Java files; copy from existing sources.
129- Files under `x-pack` require the Elastic License 2.0-only header; IDEs configured per CONTRIBUTING.md can insert correct text automatically.
130
131## Generated Files
132- Never hand-edit generated files. Instead, edit the source they are generated from and regenerate.
133- ANTLR-generated files can be regenerated by running the `regen` task on the relevant subproject.
134- Other generated files are regenerated by compiling the project.
135
136## Debugging Missing Tests
137
138When expected test methods are absent from results (not failed, not skipped — simply not present in the XML or binary event stream), check `muted-tests.yml` first. The build translates every entry into a Gradle `TestFilter.excludePattern`, which silently drops matching tests before the randomized runner receives them. A muted test fires no `testStarted` event and leaves no trace in `results-generic.bin`.
139 ```bash
140 grep 'ClassName\|methodName' muted-tests.yml
141 ```
142
143### `No tests found for given includes: [**/*$*.class]`
144
145When a test task fails at execution with `No tests found for given includes: [**/*$*.class](exclude rules)`, it usually does **not** mean Gradle failed to detect the test class. The far more common cause is that **every test method in the targeted class is muted** in `muted-tests.yml`. With all methods excluded, the randomized runner enumerates zero runnable tests.
146
147The behavior is environment-dependent: `MutedTestPlugin` calls `filter.setFailOnNoMatchingTests(buildParams.getCi() == false)`. So an all-muted suite **fails locally** (`ci == false`) with this exact message, but **passes silently in CI** (`ci == true`). This is especially misleading when verifying a freshly migrated or renamed test — it looks like a classpath/detection bug, but the test JVM does start (you'll see native-library and `FeatureFlag` log lines), builds any `@ClassRule` cluster *specs*, then exits in a few seconds without starting the cluster because no test method survived the mute filter.
148
149To confirm: `grep ClassName muted-tests.yml`. To verify the migration/test actually runs, temporarily remove the matching mute entries (or run on a host where `ci` is true), then restore them.
150
151## Best Practices for Automation Agents
152- Never edit unrelated files; keep diffs tightly scoped to the task at hand.
153- Prefer Gradle tasks over ad-hoc scripts.
154- When scripting CLI sequences, leverage `gradlew` task.
155- Unrecognized changes: assume other agent; keep going; focus your changes. If it causes issues, stop + ask user.
156- Do not add "Co-Authored-By" or any AI attribution trailers to commit messages, by any means—including `--trailer`, `-m`, or any other git flag. commit messages should adhere to the 50/72 rule: use a maximum of 50 columns for the commit summary. Your harness may introduce a hook that automatically adds attributions trailers to relevant git commands. Use `bash -lc` or a similar approach in this case to conform to the rule.
157
158## Methods with Required Javadoc Reading
159If you encounter any of the following methods, you must go and read their javadoc before taking any other actions:
160* `fullyLoadedAnalyzer`
161* `TestAnalyzer.statementError`
162* `TestAnalyzer.error`
163* `forciblyCast`
164* `EsqlCapabilities.Cap`
165* `FunctionDefinition.Builder#capabilities`
166
167## ES|QL tests
168If you write or modify ES|QL csv-spec, rest, or yaml tests, read the javadoc for
169`EsqlCapabilities.Cap` and `FunctionDefinition.Builder#capabilities` before proceeding.
170They describe two separate capability mechanisms and the rule for choosing between them.
171
172## Backwards compatibility
173- For changes to a `Writeable` implementation (`writeTo` and constructor from `StreamInput`), add a new `public static final <UNIQUE_DESCRIPTIVE_NAME> = TransportVersion.fromName("<unique_descriptive_name>")` and use it in the new code paths. Confirm the backport branches and then generate a new version file with `./gradlew generateTransportVersion`.
174- Never hand-edit transport version resource files; always use the Gradle tasks. See `docs/internal/Versioning.md` for the full workflow.
175
176Stay aligned with `CONTRIBUTING.md`, `BUILDING.md`, and `TESTING.asciidoc`; this AGENTS guide summarizes—but does not replace—those authoritative docs.
177
178## Documentation
179When building or editing docs, read `docs/AGENTS.md` first.
180
@@ −1 +1 @@
1−# Inference API plugin (x-pack-inference)
1+# Elasticsearch
22
3−Guidance for coding agents working in `x-pack/plugin/inference/` (Gradle `:x-pack:plugin:inference`, esplugin artifact `x-pack-inference`, class `InferencePlugin`). It implements the `_inference` API, integrates many external/internal inference services (see the provider subdirs under `services/`), and provides `semantic_text` and inference-based reranking. The repository-root `AGENTS.md` is authoritative for toolchain, formatting, logging, transport-version, and general testing conventions — this file does not repeat them. Read it first.
3+## Toolchain Snapshot
4+- **Java**: JDK 25 via `JAVA_HOME`; use the bundled Gradle wrapper (`./gradlew`).
5+- **Build tooling**: Gradle composite build with `build-conventions`, `build-tools`, and `build-tools-internal`; Docker is required for some packaging/tests.
6+- **OS packages**: Packaging and QA jobs expect ephemeral hosts; do not run packaging suites on your workstation.
7+- **Security**: Default dev clusters enable security; use `elastic-admin:elastic-password` or disable with `-Dtests.es.xpack.security.enabled=false`.
8+- **Cursor/Copilot rules**: None provided in repo; follow this guide plus CONTRIBUTING.md.
49
5−## This plugin vs. the ML plugin — don't confuse them
10+## Build & Run Commands
11+- Refer to BUILDING.md, CONTRIBUTING.md & TESTING.asciidoc for comprehensive build/test instructions.
612
7−These two plugins both say "inference" but do different things:
13+## Verification & Lint Tasks
14+- `./gradlew spotlessJavaCheck` / `spotlessApply` (or `:server:spotlessJavaCheck`): enforce formatter profile in `build-conventions/formatterConfig.xml`.
15+- `spotlessApply` also prunes unused imports and reorders imports automatically. Run it instead of manually hunting for unused imports after refactoring.
816
9−| | `x-pack/plugin/inference` (this plugin) | `x-pack/plugin/ml` |
10−|---|---|---|
11−| Scope | The `_inference` endpoint: register an *inference endpoint*, call out to a service, get embeddings/completions/reranks | Anomaly detection, datafeeds, data-frame analytics, **local trained-model deployment** (PyTorch via `ml-cpp`) |
12−| Compute | Mostly **calls external HTTP APIs** (OpenAI, Cohere, Bedrock, …); the `elasticsearch` service runs models locally | Runs native processes on ML nodes |
13−| Key feature | `semantic_text` field, semantic query, reranking | jobs, datafeeds, DFA, trained-model assignment |
17+## Project Structure
18+The repository is organized into several key directories:
19+* `server`: The core Elasticsearch server. Few third-party dependencies (Lucene plus a handful of small libraries). Key `org.elasticsearch` sub-packages: `cluster` (cluster state machine), `index` (per-index logic), `search` (query execution), `action` (transport actions), `snapshots` (snapshot/restore), plus `indices`, `repositories`, `rest`, `ingest`, etc.
20+* `modules`: Features shipped with Elasticsearch by default, but not considered "core" server code. Many modules provide a specific implementation of a pluggable interface defined in `server`, such as `transport-netty4` (the transport layer) or `repository-s3`/`repository-gcs`/`repository-azure` (snapshot repositories). Others integrate with external systems, such as `apm` (Application Performance Monitoring agent integration).
21+* `plugins`: Optional, not bundled by default, but officially supported. Examples: `discovery-ec2`/`discovery-gce`/`discovery-azure-classic` (cloud-aware cluster discovery).
22+* `libs`: Internal libraries used by multiple parts of the project. Examples: `logging`, `x-content` (JSON/CBOR/YAML/SMILE parsing abstraction).
23+* `client`: The official Java REST client.
24+* `test`: Test infrastructure used by the rest of the repo. `framework` holds `ESTestCase`/`ESIntegTestCase`/`ESSingleNodeTestCase`; also `test-clusters` and `yaml-rest-runner` (runner for YAML-based REST API tests).
25+* `qa`: Integration and multi-version tests. Examples: `rolling-upgrade`, `mixed-cluster`.
26+* `rest-api-spec`: JSON spec definitions for the public REST API endpoints.
27+* `docs`: Project documentation.
28+* `distribution`: Logic for building distribution packages.
29+* `x-pack`: Modules, plugins, and commercial features under the Elastic License 2.0. Example sub-plugins: `security`, `ml` (machine learning), `ccr` (cross-cluster replication), `logsdb` (optimized index mode for log data), and `stateless`.
30+* `build-conventions`, `build-tools`, `build-tools-internal`: Gradle build logic. Refer to BUILDING.md for details on how these are structured and used.
1431
15−**The seam:** the internal `elasticsearch` service (`services/elasticsearch/`, e.g. ELSER and locally-hosted models) delegates to **ML-deployed trained models** — so a request through this plugin can end up running on the ML plugin's deployment infrastructure. The shared SPI lives in **server** (`org.elasticsearch.inference`); trained-model POJOs live in `xpack.core.ml.inference`.
32+## Stateless Elasticsearch
1633
17−## Where the SPI lives (important)
34+Stateless Elasticsearch is a distribution where shard data is stored in an **object store** (e.g., S3, GCS, Azure) rather than local disk. Nodes carry no durable local state. The cluster distinguishes two node roles: **indexing nodes** (`index` role, write path + translog replication to object store) and **search nodes** (`search` role, read-only via shared blob cache). The `DiscoveryNode.STATELESS_ENABLED_SETTING` gates stateless behavior at runtime.
1835
19−The core inference contracts are **not** in this plugin and **not** in xpack-core — they are in **server** at `server/src/main/java/org/elasticsearch/inference/`: `InferenceService`, `InferenceServiceRegistry`, `Model`, `ModelConfigurations`, `ServiceSettings`/`TaskSettings`/`SecretSettings`, `TaskType`, `InputType`, `InferenceServiceResults`, `ChunkingSettings`, `InferenceServiceExtension`. This plugin provides the *implementations*; other plugins integrate via `InferenceServiceExtension` (loaded through `ExtensiblePlugin`).
36+### Plugin `deploymentTarget`
2037
21−## Build & Test Commands
38+Plugins can set `deploymentTarget` in `build.gradle`. That value tells the node **whether to load the plugin**: **`STATEFUL_ONLY`** (stateful clusters only), **`STATELESS_ONLY`** (stateless mode on only), or **`ALL`** (always loaded; this is the default when the property is omitted).
2239
23−Gradle project `:x-pack:plugin:inference`. Run from repo root. `extendedPlugins = ['x-pack-core']`. Heavy third-party deps (AWS SDK v2, Google auth/api-client, Azure identity/msal4j, Nimbus OAuth/JOSE, Jackson, Netty, Reactor) — check `build.gradle` and the thirdPartyAudit allowances before adding more.
40+### Plugin locations
2441
25−```bash
26−# Unit tests
27−./gradlew :x-pack:plugin:inference:test
28−./gradlew :x-pack:plugin:inference:test --tests "org.elasticsearch.xpack.inference.ModelConfigurationsTests"
29−./gradlew :x-pack:plugin:inference:test --tests "org.elasticsearch.xpack.inference.ModelConfigurationsTests.testSerialization"
42+| Plugin | Gradle path | Purpose |
43+|---|---|---|
44+| `stateless` | `:x-pack:plugin:stateless` | Core stateless — engines, allocation, cache, object store, recovery |
45+| `stateless-sigterm` | `:x-pack:plugin:stateless-sigterm` | Clean SIGTERM shutdown for Kubernetes |
46+| `stateless-master-failover` | `:x-pack:plugin:stateless-master-failover` | Master failover behavior |
47+| `stateless-no-wait-for-active-shards` | `:x-pack:plugin:stateless-no-wait-for-active-shards` | Suppresses wait-for-active-shards |
48+| `stateless-health-shards-availability` | `:x-pack:plugin:stateless-health-shards-availability` | Shard availability health indicators |
3049
31−# Internal cluster tests (*IT under ...inference.integration)
32−./gradlew :x-pack:plugin:inference:internalClusterTest
50+**Package**: `org.elasticsearch.xpack.stateless.*` throughout.
3351
34−# YAML REST tests (specs in src/yamlRestTest/resources/rest-api-spec/test/inference/)
35−./gradlew :x-pack:plugin:inference:yamlRestTest
52+### Key subsystems
3653
37−# Format / forbidden-API / style gate
38−./gradlew :x-pack:plugin:inference:spotlessApply
39−./gradlew :x-pack:plugin:inference:precommit
40−```
54+- **Object store** (`objectstore/`): `ObjectStoreService`, bucket config, GC tasks for stale indices and translogs.
55+- **Commits** (`commits/`): `StatelessCommitService` manages shard commits to blob store; `HollowShardsService` manages hollow indexing shards.
56+- **Cache & prewarming** (`cache/`): `StatelessSharedBlobCacheService`, online prewarming, `SearchCommitPrefetcher`.
57+- **Engines** (`engine/`): `IndexEngine` (write path) and `SearchEngine` (read-only); `TranslogReplicator` replicates translog to object store.
58+- **Allocation** (`allocation/`): `StatelessExistingShardsAllocator`, separate balancing weights per tier, heap-usage-aware allocation decisions.
59+- **Recovery** (`recovery/`): custom primary relocation and unpromotable shard relocation protocols.
4160
42−Unit-test base classes: `ESTestCase`, `AbstractWireSerializingTestCase`, `AbstractBWCWireSerializationTestCase`, and `MapperTestCase`/`MapperServiceTestCase` (for `semantic_text`).
61+## Testing Cheatsheet
62+- Standard suite: `./gradlew test` (respects cached results; add `-Dtests.timestamp=$(date +%s)` to bypass caches when reusing seeds).
63+- Single project: `./gradlew :server:test` (or other subproject path).
64+- Single class: `./gradlew :server:test --tests org.elasticsearch.package.ClassName`.
65+- Single package: `./gradlew :server:test --tests 'org.elasticsearch.package.*'`.
66+- Single method / repeated runs: `./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N`.
67+- Deterministic seed: append `-Dtests.seed=DEADBEEF` (each method uses derived seeds).
68+- JVM tuning knobs: `-Dtests.jvms=8`, `-Dtests.heap.size=4G`, `-Dtests.jvm.argline="-verbose:gc"`, `-Dtests.output=always`, etc.
69+- Debugging: append `--debug-jvm` to the Gradle test task and attach a debugger on port 5005.
70+- CI reproductions: copy the `REPRODUCE WITH` line from CI logs; it includes project path, seed, and JVM flags.
71+- Yaml REST tests: `./gradlew ":rest-api-spec:yamlRestTest" --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=<relative_test_file_path>}"`
72+- ES|QL CSV tests: `./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.CsvIT.*<csv-file>*"` (e.g. `--tests "...CsvIT.*stats_first_last*"`); append `*<test-name>*` to target a single test within the file.
73+- Use the Elasticsearch testing framework where possible for unit and yaml tests and be consistent in style with other elasticsearch tests.
74+- Use real classes over mocks or stubs for unit tests, unless the real class is complex then either a simplified subclass should be created within the test or, as a last resort, a mock or stub can be used. Unit tests must be as close to real-world scenarios as possible.
75+- Ensure mocks or stubs are well-documented and clearly indicate why they were necessary.
4376
44−### QA suites (`qa/`)
77+### Test Types
78+- Unit Tests: Preferred. Extend `ESTestCase`.
79+- Single Node: Extend `ESSingleNodeTestCase` (lighter than full integ test).
80+- Integration: Extend `ESIntegTestCase`.
81+- REST API: Extend `ESRestTestCase` or `ESClientYamlSuiteTestCase`. **YAML based REST tests are preferred** for integration/API testing.
4582
46−| Suite | Command | Purpose |
47−|---|---|---|
48−| `test-service-plugin` | (cluster plugin, consumed by others) | **Mock inference service** (`TestInferenceServicePlugin`) — the way to write service-level REST tests without real API keys |
49−| `inference-service-tests` | `:qa:inference-service-tests:javaRestTest` | provides `InferenceBaseRestTest` (`putModel`/`infer`/`deleteModel` helpers) |
50−| `inference-with-security` | `:qa:inference-with-security:yamlRestTest` | security enabled |
51−| `multi-node` | `:qa:multi-node:yamlRestTest` | multi-node cluster |
52−| `oauth2` | `:qa:oauth2:javaRestTest` | in-process mock OAuth2 server (skipped on FIPS) |
53−| `mixed-cluster`, `rolling-upgrade` | `:qa:<name>:vX.Y.Z#javaRestTest` | BWC / rolling-upgrade |
83+### Distribution selection for external-module tests
84+- Prefer the OSS/minimal distribution over `usesDefaultDistribution` whenever possible. `usesDefaultDistribution` packages the full default distribution, which is significantly more expensive to build and run.
85+- Only use `usesDefaultDistribution` when the test genuinely requires a feature that is only available in the default distribution and cannot be replicated with a custom cluster configuration that includes just the needed plugins. Always document the reason in the `usesDefaultDistribution(...)` message.
5486
55−## Request lifecycle
87+## Dependency Hygiene
88+- Never add a dependency without checking for existing alternatives in the repo.
5689
57−REST handler (`rest/`) → Transport action (`action/`, base `BaseTransportInferenceAction`, with license check) → `ModelRegistry` lookup of the endpoint config → `InferenceServiceRegistry` dispatch to the concrete service → service runs inference (external HTTP or local) → `InferenceServiceResults` back to the listener. Streaming uses the chunked/streaming variants.
90+## Entitlement Policy
91+- Never add an entitlement speculatively. Each entry in `entitlement-policy.yaml` must have a specific justification — ideally a concrete `NotEntitledException` that was observed, or at minimum a clear explanation of why the library requires that capability. Entitlements are a least-privilege mechanism; granting one "just in case" defeats the purpose.
92+- Every use of `ESTestCase.WithoutEntitlements` must be accompanied by a comment explaining why the entitlement failure is spurious in the test context and would not occur in production.
5893
59−## Adding a new external service integration
94+## Formatting & Imports
95+- Absolutely no wildcard imports; keep existing import order and avoid reordering untouched lines.
96+- In `switch` statements, do not use `default` as a branch for valid or expected options. Enumerate those cases explicitly and reserve `default` for throwing an exception for unexpected values, or an assertion error if this code branch is unreachable.
6097
61−1. Create `services/<name>/`. Implement the `Model` plus its `<Name>ServiceSettings` (non-secret config), `<Name>TaskSettings` (per-request knobs), and `<Name>SecretSettings` (API keys) — subtypes of the server SPI settings interfaces (base helpers in `services/settings/`).
62−2. Extend `SenderService` (`services/SenderService.java`, the base `InferenceService` impl) with a concrete `<Name>Service`; implement the `doInfer`/`doChunkedInfer`/`doUnifiedCompletionInfer` abstract methods.
63−3. Provide a `RequestManager` (builds the outbound HTTP request) and a service-specific `ResponseHandler` (parses the response into `InferenceServiceResults`).
64−4. Register the service factory in `InferencePlugin` and any `NamedWriteable`s in `InferenceNamedWriteablesProvider`.
65−5. Respect `ConfigurationParseContext` — `REQUEST` (strict, user API input) vs `PERSISTENT` (lenient, loaded from the index). Use `ServiceUtils`/`ServiceFields` helpers for map extraction and validation.
98+## Types, Generics, and Suppressions
99+- Prefer type-safe constructs; avoid raw types and unchecked casts.
100+- If suppressing warnings, scope `@SuppressWarnings` narrowly (ideally a single statement or method).
101+- Document non-obvious casts or type assumptions via Javadoc/comments for reviewers.
66102
67−## External HTTP layer (`external/`) — traced flow
103+## Naming Conventions
104+- REST handlers typically use the `Rest*Action` pattern; transport-layer handlers mirror them with `Transport*Action` classes.
105+- REST classes expose routes via `RestHandler#routes`; when adding endpoints ensure naming matches existing REST/Transport patterns to aid discoverability.
106+- Transport `ActionType` strings encode scope (`indices:data/read/...`, `cluster:admin/...`, etc.); align new names with these conventions to integrate with privilege resolution.
68107
69−For services that call out over HTTP, a request flows:
108+## Logging & Error Handling
109+- Elasticsearch should prefer its own logger `org.elasticsearch.logging.LogManager` & `org.elasticsearch.logging.Logger`; declare `private static final Logger logger = LogManager.getLogger(Class.class)`.
110+- Always use parameterized logging (`logger.debug("operation [{}]", value)`); never build strings via concatenation.
111+- Wrap expensive log-message construction in `() -> Strings.format(...)` suppliers when logging at `TRACE`/`DEBUG` to avoid unnecessary work.
112+- Log levels:
113+ - `TRACE`: highly verbose developer diagnostics; usually read alongside code.
114+ - `DEBUG`: detailed production troubleshooting; ensure volume is bounded.
115+ - `INFO`: default-enabled operational milestones; prefer factual language.
116+ - `WARN`: actionable problems users must investigate; include context and, if needed, exception stack traces.
117+ - `ERROR`: reserve for unrecoverable states (e.g., storage health failures); prefer `WARN` otherwise.
118+- Only log client-caused exceptions when the cluster admin can act on them; otherwise rely on API responses.
119+- Tests can assert logging via `MockLog` for complex flows.
70120
71−1. `SenderService` calls `Sender.send(requestManager, inferenceInputs, timeout, listener)`.
72−2. `HttpRequestSender` lazily starts the async client/executor on first use, then enqueues to `RequestExecutorService`.
73−3. The executor applies **per-service rate limiting** and dequeues onto the `inference_utility` thread pool.
74−4. `RequestManager.buildRequest(...)` produces the outbound HTTP request.
75−5. `RetryingHttpSender` executes it with backoff on 429/5xx/timeouts.
76−6. `HttpClientManager` (Apache `HttpAsyncClient` connection pool, idle-eviction, XPack SSL) does the actual call.
77−7. The service's `ResponseHandler` parses the response into `InferenceServiceResults`, returned via the listener (responses delivered on the `inference_response` pool).
121+## Javadoc & Comments
122+- New packages/classes/public or abstract methods require Javadoc explaining the "why" rather than the implementation details.
123+- Avoid documenting trivial getters/setters; focus on behavior, preconditions, or surprises.
124+- For tests, Javadoc can describe scenario setup/expectations to aid future contributors.
125+- Do not remove existing comments from code unless the code is also being removed or the comment has become incorrect.
78126
79−## Registry & system indices
127+## License Headers
128+- Default header (outside `x-pack`): Elastic License 2.0, SSPL v1, or AGPL v3—they are already codified at the top of Java files; copy from existing sources.
129+- Files under `x-pack` require the Elastic License 2.0-only header; IDEs configured per CONTRIBUTING.md can insert correct text automatically.
80130
81−`registry/ModelRegistry` persists endpoint configurations and syncs them from cluster state for fast `getModel(inferenceId)` lookups. Three system indices:
131+## Generated Files
132+- Never hand-edit generated files. Instead, edit the source they are generated from and regenerate.
133+- ANTLR-generated files can be regenerated by running the `regen` task on the relevant subproject.
134+- Other generated files are regenerated by compiling the project.
82135
83−- `InferenceIndex` → `.inference` — endpoint configs (service, task_type, service_settings, task_settings, chunking_settings). Root mappings are strict; the settings sub-objects are `dynamic:false` to allow service-specific fields.
84−- `InferenceSecretsIndex` → `.inference-secrets` — credentials, kept in a **separate index** from config so secrets can be secured/backed-up independently.
85−- An Elastic-Inference-Service cloud-connected-mode index for EIS auth config.
136+## Debugging Missing Tests
86137
87−## semantic_text & the search path
138+When expected test methods are absent from results (not failed, not skipped — simply not present in the XML or binary event stream), check `muted-tests.yml` first. The build translates every entry into a Gradle `TestFilter.excludePattern`, which silently drops matching tests before the randomized runner receives them. A muted test fires no `testStarted` event and leaves no trace in `results-generic.bin`.
139+ ```bash
140+ grep 'ClassName\|methodName' muted-tests.yml
141+ ```
88142
89−`semantic_text` makes embedding generation automatic across ingest and search:
143+### `No tests found for given includes: [**/*$*.class]`
90144
91−- **Mapping:** `mapper/SemanticTextFieldMapper` defines the field and auto-creates dense- or sparse-vector subfields (plus chunk/text metadata) based on the referenced endpoint's task type.
92−- **Ingest:** `action/filter/ShardBulkInferenceActionFilter` intercepts bulk shard requests, batches inference over the configured `inference_fields` (chunking long text, respecting `INDICES_INFERENCE_BATCH_SIZE`), and injects the embeddings into the document source before normal indexing.
93−- **Search:** `queries/SemanticQueryBuilder` and the `Semantic{Knn,Match,SparseVector}QueryRewriteInterceptor`s auto-embed the query text at rewrite time and rewrite to native vector/sparse/match queries. `highlight/SemanticTextHighlighter` highlights matched chunks.
94−- **Reranking:** `rank/textsimilarity/TextSimilarityRankBuilder` reranks first-pass hits through a rerank endpoint.
145+When a test task fails at execution with `No tests found for given includes: [**/*$*.class](exclude rules)`, it usually does **not** mean Gradle failed to detect the test class. The far more common cause is that **every test method in the targeted class is muted** in `muted-tests.yml`. With all methods excluded, the randomized runner enumerates zero runnable tests.
95146
96−## Gotchas
147+The behavior is environment-dependent: `MutedTestPlugin` calls `filter.setFailOnNoMatchingTests(buildParams.getCi() == false)`. So an all-muted suite **fails locally** (`ci == false`) with this exact message, but **passes silently in CI** (`ci == true`). This is especially misleading when verifying a freshly migrated or renamed test — it looks like a classpath/detection bug, but the test JVM does start (you'll see native-library and `FeatureFlag` log lines), builds any `@ClassRule` cluster *specs*, then exits in a few seconds without starting the cluster because no test method survived the mute filter.
97148
98−- **Register on both sides.** A new service/model/results type must be registered as a service factory in `InferencePlugin` **and** as `NamedWriteable`(s) in `InferenceNamedWriteablesProvider`, or it silently fails to deserialize from cluster state / the index.
99−- **Right parse context.** Honor `ConfigurationParseContext`: `REQUEST` (strict, rejects unknown fields — user API input) vs `PERSISTENT` (lenient — loaded from `.inference`). Using the wrong one breaks either client validation or forward-compat reads.
100−- **Secrets stay separate.** Credentials are written to `.inference-secrets`, never `.inference`. Don't fold secret settings into the config document.
101−- **SPI is in server.** The `InferenceService`/`Model`/settings/`TaskType` contracts live in server's `org.elasticsearch.inference`, not this plugin and not xpack-core — extend those, don't fork them.
102−- **Don't block.** External calls go through the async `Sender`/`HttpRequestSender` path on the inference thread pools; don't call services synchronously from a transport or cluster-state thread.
149+To confirm: `grep ClassName muted-tests.yml`. To verify the migration/test actually runs, temporarily remove the matching mute entries (or run on a host where `ci` is true), then restore them.
103150
104−## Testing conventions specific to inference
151+## Best Practices for Automation Agents
152+- Never edit unrelated files; keep diffs tightly scoped to the task at hand.
153+- Prefer Gradle tasks over ad-hoc scripts.
154+- When scripting CLI sequences, leverage `gradlew` task.
155+- Unrecognized changes: assume other agent; keep going; focus your changes. If it causes issues, stop + ask user.
156+- Do not add "Co-Authored-By" or any AI attribution trailers to commit messages, by any means—including `--trailer`, `-m`, or any other git flag. commit messages should adhere to the 50/72 rule: use a maximum of 50 columns for the commit summary. Your harness may introduce a hook that automatically adds attributions trailers to relevant git commands. Use `bash -lc` or a similar approach in this case to conform to the rule.
105157
106−- Serialization round-trips use the wire/BWC bases; `semantic_text` mapper tests use `MapperTestCase`/`MapperServiceTestCase`.
107−- To test service behavior end-to-end without real credentials, depend on `qa/test-service-plugin` (the mock `TestInferenceServicePlugin`) and extend `InferenceBaseRestTest` from `qa/inference-service-tests`.
108−- Internal cluster tests are named `*IT` under `...inference.integration`; YAML specs live in `src/yamlRestTest/resources/rest-api-spec/test/inference/`.
158+## Methods with Required Javadoc Reading
159+If you encounter any of the following methods, you must go and read their javadoc before taking any other actions:
160+* `fullyLoadedAnalyzer`
161+* `TestAnalyzer.statementError`
162+* `TestAnalyzer.error`
163+* `forciblyCast`
164+* `EsqlCapabilities.Cap`
165+* `FunctionDefinition.Builder#capabilities`
166+
167+## ES|QL tests
168+If you write or modify ES|QL csv-spec, rest, or yaml tests, read the javadoc for
169+`EsqlCapabilities.Cap` and `FunctionDefinition.Builder#capabilities` before proceeding.
170+They describe two separate capability mechanisms and the rule for choosing between them.
171+
172+## Backwards compatibility
173+- For changes to a `Writeable` implementation (`writeTo` and constructor from `StreamInput`), add a new `public static final <UNIQUE_DESCRIPTIVE_NAME> = TransportVersion.fromName("<unique_descriptive_name>")` and use it in the new code paths. Confirm the backport branches and then generate a new version file with `./gradlew generateTransportVersion`.
174+- Never hand-edit transport version resource files; always use the Gradle tasks. See `docs/internal/Versioning.md` for the full workflow.
175+
176+Stay aligned with `CONTRIBUTING.md`, `BUILDING.md`, and `TESTING.asciidoc`; this AGENTS guide summarizes—but does not replace—those authoritative docs.
177+
178+## Documentation
179+When building or editing docs, read `docs/AGENTS.md` first.
109180
