AGENTS.md

1<!--
2Licensed to the Apache Software Foundation (ASF) under one
3or more contributor license agreements.  See the NOTICE file
4distributed with this work for additional information
5regarding copyright ownership.  The ASF licenses this file
6to you under the Apache License, Version 2.0 (the
7"License"); you may not use this file except in compliance
8with the License.  You may obtain a copy of the License at
9 
10  http://www.apache.org/licenses/LICENSE-2.0
11 
12Unless required by applicable law or agreed to in writing,
13software distributed under the License is distributed on an
14"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15KIND, either express or implied.  See the License for the
16specific language governing permissions and limitations
17under the License.
18-->
19 
20# Flink AI Agent Instructions
21 
22This file provides guidance for AI coding agents working with the Apache Flink codebase.
23 
24## Prerequisites
25 
26- Java 11, 17 (default), or 21. Java 11 syntax must be used in all modules. Java 17 syntax (records, sealed classes, pattern matching) is only permitted in the `flink-tests-java17` module.
27- Maven 3.8.6 (Maven wrapper `./mvnw` included; prefer it)
28- Git
29- Unix-like environment (Linux, macOS, WSL, Cygwin)
30 
31## Commands
32 
33### Build
34 
35- Fast dev build: `./mvnw clean install -DskipTests -Dfast -Pskip-webui-build -T1C`
36- Full build (Java 17 default): `./mvnw clean package -DskipTests -Djdk17 -Pjava17-target`
37- Java 11: `./mvnw clean package -DskipTests -Djdk11 -Pjava11-target`
38- Java 21: `./mvnw clean package -DskipTests -Djdk21 -Pjava21-target`
39- Full build with tests: `./mvnw clean verify`
40- Single module: `./mvnw clean package -DskipTests -pl flink-core-api`
41- Single module with tests: `./mvnw clean verify -pl flink-core-api`
42 
43### Testing
44 
45- Single test class: `./mvnw -pl flink-core-api -Dtest=MemorySizeTest test`
46- Single test method: `./mvnw -pl flink-core-api -Dtest=MemorySizeTest#testParseBytes test`
47 
48### Code Quality
49 
50- Format code (Java + Scala): `./mvnw spotless:apply`
51- Check formatting: `./mvnw spotless:check`
52- Checkstyle: `./mvnw checkstyle:check -T1C`
53- Checkstyle config: `tools/maven/checkstyle.xml`
54 
55## Repository Structure
56 
57Every module from the root pom.xml, organized by function. Flink provides three main user-facing APIs (recommended in this order: SQL, Table API, DataStream API) plus a newer DataStream v2 API.
58 
59### Core Infrastructure
60 
61- `flink-annotations` — Stability annotations (`@Public`, `@PublicEvolving`, `@Internal`, `@Experimental`) and `@VisibleForTesting`
62- `flink-core-api` — Core API interfaces (functions, state, types) shared by all APIs
63- `flink-core` — Core implementation (type system, serialization, memory management, configuration)
64- `flink-runtime` — Distributed runtime (JobManager, TaskManager, scheduling, network, state)
65- `flink-clients` — CLI and client-side job submission
66- `flink-rpc/` — RPC framework
67  - `flink-rpc-core` — RPC interfaces
68  - `flink-rpc-akka`, `flink-rpc-akka-loader` — Pekko-based RPC implementation
69 
70### SQL / Table API (recommended API for most users)
71 
72- `flink-table/`
73  - `flink-sql-parser` — SQL parser (extends Calcite SQL parser)
74  - `flink-table-common` — Shared types, descriptors, catalog interfaces
75  - `flink-table-api-java` — Table API for Java
76  - `flink-table-api-scala` — Table API for Scala
77  - `flink-table-api-bridge-base`, `flink-table-api-java-bridge`, `flink-table-api-scala-bridge` — Bridges between Table and DataStream APIs
78  - `flink-table-api-java-uber` — Uber JAR for Table API
79  - `flink-table-planner` — SQL/Table query planning and optimization (Calcite-based)
80  - `flink-table-planner-loader`, `flink-table-planner-loader-bundle` — Classloader isolation for planner
81  - `flink-table-runtime` — Runtime operators for Table/SQL queries
82  - `flink-table-calcite-bridge` — Bridge to Apache Calcite
83  - `flink-sql-gateway-api`, `flink-sql-gateway` — SQL Gateway for remote SQL execution
84  - `flink-sql-client` — Interactive SQL CLI
85  - `flink-sql-jdbc-driver`, `flink-sql-jdbc-driver-bundle` — JDBC driver for SQL Gateway
86  - `flink-table-code-splitter` — Code generation utilities
87  - `flink-table-test-utils` — Test utilities for Table/SQL
88 
89### DataStream API (original streaming API)
90 
91- `flink-streaming-java` — DataStream API and stream processing operator implementations
92 
93### DataStream API v2 (newer event-driven API)
94 
95- `flink-datastream-api` — DataStream v2 API definitions
96- `flink-datastream` — DataStream v2 API implementation
97 
98### Connectors (in-tree)
99 
100- `flink-connectors/`
101  - `flink-connector-base` — Base classes for source/sink connectors
102  - `flink-connector-files` — Unified file system source and sink
103  - `flink-connector-datagen` — DataGen source for testing
104  - `flink-connector-datagen-test` — Tests for DataGen connector
105  - `flink-hadoop-compatibility` — Hadoop InputFormat/OutputFormat compatibility
106  - `flink-file-sink-common` — Common file sink utilities
107- Most connectors (Kafka, JDBC, Elasticsearch, etc.) live in separate repos under [github.com/apache](https://github.com/apache); see README.md for the full list
108 
109### Formats
110 
111- `flink-formats/`
112  - `flink-json`, `flink-csv`, `flink-avro`, `flink-parquet`, `flink-orc`, `flink-protobuf` — Serialization formats
113  - `flink-avro-confluent-registry` — Avro with Confluent Schema Registry
114  - `flink-sequence-file`, `flink-compress`, `flink-hadoop-bulk`, `flink-orc-nohive` — Hadoop-related formats
115  - `flink-format-common` — Shared format utilities
116  - `flink-sql-json`, `flink-sql-csv`, `flink-sql-avro`, `flink-sql-parquet`, `flink-sql-orc`, `flink-sql-protobuf` — SQL-layer format integrations
117  - `flink-sql-avro-confluent-registry` — SQL-layer Avro with Confluent Schema Registry
118 
119### State Backends
120 
121- `flink-state-backends/`
122  - `flink-statebackend-rocksdb` — RocksDB state backend
123  - `flink-statebackend-forst` — ForSt state backend (experimental; a fork of RocksDB)
124  - `flink-statebackend-heap-spillable` — Heap-based spillable state backend
125  - `flink-statebackend-changelog` — Changelog state backend
126  - `flink-statebackend-common` — Shared state backend utilities
127- `flink-dstl/flink-dstl-dfs` — State changelog storage (DFS-based persistent changelog for incremental checkpointing)
128 
129### File Systems
130 
131- `flink-filesystems/`
132  - `flink-hadoop-fs` — Hadoop FileSystem abstraction
133  - `flink-s3-fs-native`, `flink-s3-fs-hadoop`, `flink-s3-fs-presto`, `flink-s3-fs-base` — S3 file systems
134  - `flink-oss-fs-hadoop` — Alibaba OSS
135  - `flink-azure-fs-hadoop` — Azure Blob Storage
136  - `flink-gs-fs-hadoop` — Google Cloud Storage
137  - `flink-fs-hadoop-shaded` — Shaded Hadoop dependencies
138 
139### Queryable State
140 
141- `flink-queryable-state/`
142  - `flink-queryable-state-runtime` — Server-side queryable state service
143  - `flink-queryable-state-client-java` — Client for querying operator state from running jobs
144 
145### Deployment
146 
147- `flink-kubernetes` — Kubernetes integration
148- `flink-yarn` — YARN integration
149- `flink-dist`, `flink-dist-scala` — Distribution packaging
150- `flink-container` — Container entry-point and utilities for containerized deployments
151 
152### Metrics
153 
154- `flink-metrics/`
155  - `flink-metrics-core` — Metrics API and core implementation
156  - Reporter implementations: `flink-metrics-jmx`, `flink-metrics-prometheus`, `flink-metrics-datadog`, `flink-metrics-statsd`, `flink-metrics-graphite`, `flink-metrics-influxdb`, `flink-metrics-slf4j`, `flink-metrics-dropwizard`, `flink-metrics-otel`
157 
158### Libraries
159 
160- `flink-libraries/`
161  - `flink-cep` — Complex Event Processing
162  - `flink-state-processing-api` — Offline state access (savepoint reading/writing)
163 
164### Other
165 
166- `flink-models` — AI model integration (sub-modules: `flink-model-openai`, `flink-model-triton`)
167- `flink-python` — PyFlink (Python API)
168- `flink-runtime-web` — Web UI for JobManager dashboard
169- `flink-external-resources` — External resource management (e.g., GPU)
170- `docs/` — Documentation content (Hugo site). This is where user-facing docs are written.
171- `flink-docs` — Documentation build module (auto-generated config reference docs)
172- `flink-examples` — Example programs
173- `flink-quickstart` — Maven archetype for new projects
174- `flink-walkthroughs` — Tutorial walkthrough projects
175 
176### Testing
177 
178- `flink-tests` — Integration tests
179- `flink-end-to-end-tests` — End-to-end tests
180- `flink-test-utils-parent` — Test utility classes
181- `flink-yarn-tests` — YARN-specific tests
182- `flink-fs-tests` — FileSystem tests
183- `flink-architecture-tests` — ArchUnit architectural boundary tests
184- `tools/ci/flink-ci-tools` — CI tooling
185 
186## Architecture Boundaries
187 
1881. **Client** submits jobs to the cluster. Submission paths include the CLI (`bin/flink run` via `flink-clients`), the SQL Client (`bin/sql-client.sh` via `flink-sql-client`), the SQL Gateway (`flink-sql-gateway`, also accessible via JDBC driver), the REST API (direct HTTP to JobManager), programmatic execution (`StreamExecutionEnvironment.execute()` or `TableEnvironment.executeSql()`), and PyFlink (`flink-python`, wraps the Java APIs).
1892. **JobManager** (`flink-runtime`) orchestrates execution: receives jobs, creates the execution graph, manages scheduling, coordinates checkpoints, and handles failover. Never runs user code directly.
1903. **TaskManager** (`flink-runtime`) executes the user's operators in task slots. Manages network buffers, state backends, and I/O.
1914. **Table Planner** (`flink-table-planner`) translates SQL/Table API programs into DataStream programs. The planner is loaded in a separate classloader (`flink-table-planner-loader`) to isolate Calcite dependencies.
1925. **Connectors** communicate with external systems. Source connectors implement the `Source` API (FLIP-27); sinks implement the `Sink` API (package `sink2`). Most connectors are externalized to separate repositories.
1936. **State Backends** persist keyed state and operator state. RocksDB is the primary backend for production use.
1947. **Checkpointing** provides exactly-once guarantees. The JobManager coordinates barriers through the data stream; TaskManagers snapshot local state to a distributed file system.
195 
196Key separations:
197 
198- **Planner vs Runtime:** The table planner generates code and execution plans; the runtime executes them. Changes to planning logic live in `flink-table-planner`; changes to runtime operators live in `flink-table-runtime` or `flink-streaming-java`.
199- **Codegen vs hand-written operators:** Per-record expression logic (casts, projections, filters, function calls) is generated at planning time by cast rules in `flink-table-planner/.../functions/casting/` and call generators in `flink-table-planner/.../codegen/calls/`, then compiled by Janino into the surrounding operator class. Operators with fixed structure (joins, aggregations, source/sink runtime) are hand-written Java in `flink-table-runtime` or `flink-streaming-java`. New scalar functions usually only need a `BuiltInFunctionDefinitions` entry plus a `BuiltInScalarFunction` subclass - the planner wires up codegen automatically. New cast behaviour or a custom call shape needs a cast rule or call generator.
200- **API vs Implementation:** Public API surfaces (`flink-core-api`, `flink-datastream-api`, `flink-table-api-java`) are separate from implementation modules. API stability annotations control what users can depend on.
201- **ArchUnit enforcement:** `flink-architecture-tests/` contains ArchUnit tests that enforce module boundaries. New violations should be avoided; if unavoidable, follow the freeze procedure in `flink-architecture-tests/README.md`.
202 
203## Common Change Patterns
204 
205This section maps common types of Flink changes to the modules they touch and the verification they require.
206 
207### Adding a new SQL built-in function
208 
2091. Register in `flink-table-common` in `BuiltInFunctionDefinitions.java` (definition, input/output type strategies, runtime class reference)
2102. Implement in `flink-table-runtime` under `functions/` (extend the appropriate base class: `BuiltInScalarFunction`, `BuiltInTableFunction`, `BuiltInAggregateFunction`, or `BuiltInProcessTableFunction`)
2113. Add tests in `flink-table-planner` and `flink-table-runtime`
2124. Extend Table API support
2135. Document in `docs/`
2146. See [flink-table/flink-table-planner/AGENTS.md](flink-table/flink-table-planner/AGENTS.md) and [flink-table/flink-table-runtime/AGENTS.md](flink-table/flink-table-runtime/AGENTS.md) for detailed patterns
215 
216### Adding a new configuration option
217 
2181. Define `ConfigOption<T>` in the relevant config class (e.g., `ExecutionConfigOptions.java` in `flink-table-api-java`)
2192. Use `ConfigOptions.key("table.exec....")` builder with type, default value, and description
2203. Add `@Documentation.TableOption` annotation for auto-generated docs
2214. Document in `docs/` if user-facing
2225. Verify: unit test for default value, ITCase for behavior change
223 
224### Adding a new table operator (e.g., join type, aggregate)
225 
2261. Involves `flink-table-runtime` (operator), `flink-table-planner` (ExecNode, physical/logical rules), and tests across both
2272. See [flink-table/flink-table-planner/AGENTS.md](flink-table/flink-table-planner/AGENTS.md) and [flink-table/flink-table-runtime/AGENTS.md](flink-table/flink-table-runtime/AGENTS.md) for detailed development order and testing patterns
228 
229### Adding a new connector (Source or Sink)
230 
2311. Implement the `Source` API (`flink-connector-base`): `SplitEnumerator`, `SourceReader`, `SourceSplit`, serializers (`SimpleVersionedSerializer`)
2322. Or implement the `Sink` API (package `sink2`) for sinks
2333. Most new connectors go in separate repos under `github.com/apache`, not in the main Flink repo
2344. Verify: unit tests + ITCase with real or embedded external system
235 
236### Modifying state serializers
237 
2381. Changes to `TypeSerializer` require a corresponding `TypeSerializerSnapshot` for migration
2392. Bump version in `getCurrentVersion()`, handle old versions in `readSnapshot()`
2403. Snapshot must have no-arg constructor for reflection-based deserialization
2414. Implement `resolveSchemaCompatibility()` for upgrade paths
2425. Verify: serializer snapshot migration tests, checkpoint restore tests across versions
243 
244### Introducing or changing user-facing APIs (`@Public`, `@PublicEvolving`, `@Experimental`)
245 
2461. New user-facing API requires a voted FLIP (Flink Improvement Proposal); this applies to `@Public`, `@PublicEvolving`, and `@Experimental` since users build against all three
2472. Every user-facing API class and method must carry a stability annotation
2483. Changes to existing `@Public` or `@PublicEvolving` API must maintain backward compatibility
2494. `@Internal` APIs can be changed freely; users should not depend on them
2505. Update JavaDoc on the changed class/method
2516. Add to release notes
2527. Verify: ArchUnit tests pass, no new architecture violations
253 
254## Coding Standards
255 
256- **Format Java files with Spotless immediately after editing:** `./mvnw spotless:apply`. Uses google-java-format with AOSP style.
257- **Scala formatting:** Spotless + scalafmt (config at `.scalafmt.conf`, maxColumn 100).
258- **Checkstyle:** `tools/maven/checkstyle.xml` (version defined in root `pom.xml` as `checkstyle.version`). Some modules (flink-core, flink-optimizer, flink-runtime) are not covered by checkstyle enforcement, but conventions should still be followed.
259- **No new Scala code.** All Flink Scala APIs are deprecated per FLIP-265. Write all new code in Java.
260- **Apache License 2.0 header** required on all new files (enforced by Apache Rat). Use an HTML comment for markdown files.
261- **API stability annotations:** Every user-facing API class and method must have a stability annotation. `@Public` (stable across minor releases), `@PublicEvolving` (may change in minor releases), `@Experimental` (may change at any time). These are all part of the public API surface that users build against. `@Internal` marks APIs with no stability guarantees that users should not depend on.
262- **Logging:** Use parameterized log statements (SLF4J `{}` placeholders), never string concatenation.
263- **No Java serialization** for new features (except internal RPC message transport).
264- **Use `final`** for variables and fields where applicable.
265- **Comments:** Do not add unnecessary comments that restate what the code does. Add comments that explain "the why" where relevant.
266- **Reuse existing code.** Before implementing new utilities or abstractions, search for existing ones in the codebase. Prioritize architecture consistency and code reusability.
267- Full code style guide: https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/
268 
269## Testing Standards
270 
271- Add tests for new behavior, covering success, failure, and edge cases.
272- Use **JUnit 5** + **AssertJ** assertions. Do not use JUnit 4 or Hamcrest in new test code.
273- Prefer real test implementations over Mockito mocks where possible.
274- **Integration tests:** Name classes with `ITCase` suffix (e.g., `MyFeatureITCase.java`).
275- **Red-green verification:** For bug fixes, verify that new tests actually fail without the fix before confirming they pass with it.
276- **Test location** mirrors source structure within each module.
277- Follow the testing conventions at https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing
278 
279## Commits and PRs
280 
281### Commit message format
282 
283- `[FLINK-XXXX][component] Description` where FLINK-XXXX is the JIRA issue number
284- `[hotfix][component] Description` for typo fixes without JIRA
285- Each commit must have a meaningful message including the JIRA ID. If you don't know the ticket number, ask.
286- Separate cleanup/refactoring from functional changes into distinct commits
287- When AI tools were used: add `Generated-by: <Tool Name and Version>` trailer per [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html)
288 
289### Pull request conventions
290 
291- Title format: `[FLINK-XXXX][component] Title of the pull request`
292- A corresponding JIRA issue is required (except hotfixes for typos)
293- Fill out the PR template completely but concisely: describe purpose, change log, testing approach, impact assessment
294- Each PR should address exactly one issue
295- Ensure `./mvnw clean verify` passes before opening a PR
296- Always push to your fork, not directly to `apache/flink`
297- Rebase onto the latest target branch before submitting
298- For user-visible behaviour changes, breaking changes, new SQL features, or new config options: fill in the **Release Notes** field on the JIRA ticket. The release manager consolidates these when cutting a release. The next version's `docs/content/release-notes/flink-X.Y.md` will be generated based of the jira tickets, so make sure to fill them in properly.
299 
300### AI-assisted contributions
301 
302- Disclose AI usage by checking the AI disclosure checkbox and uncommenting the `Generated-by` line in the PR template
303- Add `Generated-by: <Tool Name and Version>` to commit messages
304- Never add `Co-Authored-By` with an AI agent as co-author; agents are assistants, not authors
305- You must be able to explain the design, code, and tests, debug them, and respond to review feedback substantively
306- Reviewer-ready quality bar: the author owns PR quality. PRs that look AI-generated without author refinement (walls of unreviewed prose, scaffolding without behaviour, tests that do not exercise the change, padded commit messages) will be closed without review
307 
308## Code Review Guidelines
309 
310When reviewing a PR or diff against this repo:
311 
312- Look for opportunities to simplify the code, scoped to the diff itself (not pre-existing code outside the change).
313- Flag comments that are obvious (restate what the code already says) or overly verbose.
314- In test code, look for potential flakiness — e.g. `Thread.sleep` used outside a retry/poll loop, or similar timing-dependent, non-deterministic patterns. Where applicable, suggest clock injection (e.g. a manually-advanced `Clock`/`ManualClock`) instead of relying on wall-clock time, or waiting for the actual condition in a loop with a timeout, for deterministic tests.
315- Check that each commit message conforms to Flink conventions: it must start with `[FLINK-XXXX]` or `[hotfix]`, and must specify a subsystem/component (e.g. `[FLINK-XXXX][runtime] Description`).
316- If a change introduces a new feature controlled by a config option/flag, check that the resolved state (enabled/disabled, and the effective value) is logged at INFO level when the feature initializes/activates.
317- Consider whether a change should be hidden behind a feature flag, especially if it's non-trivial (touches core paths, changes default behavior, or is hard to reason about in isolation). This is mandatory if the change is risky (correctness, performance, backward-compatibility, or data-safety risk) and no flag/kill-switch already exists.
318- For changes to configuration options, check if the corresponding documentation has been regenerated (this should be covered by tests but flagging it earlier speeds up development)
319 
320## Boundaries
321 
322### Ask first
323 
324- Adding or changing `@Public`, `@PublicEvolving`, or `@Experimental` annotations (these are user-facing API commitments requiring a FLIP)
325- Large cross-module refactors
326- New dependencies
327- Changes to serialization formats (affects state compatibility)
328- Changes to checkpoint/savepoint behavior
329- Changes that could impact performance on hot paths (per-record processing, serialization, state access)
330 
331### Never
332 
333- Commit secrets, credentials, or tokens
334- Push directly to `apache/flink`; always work from your fork
335- Mix unrelated changes into one PR
336- Use Java serialization for new features
337- Edit generated files by hand when a generation workflow exists
338- Use the legacy `SourceFunction` or `SinkFunction` interfaces for connectors; use the `Source` API (FLIP-27) and `Sink` API (package `sink2`) instead
339- Add `Co-Authored-By` with an AI agent as co-author in commit messages; AI agents are assistants, not authors. Use `Generated-by: <Tool Name and Version>` instead.
340- Suppress or bypass checkstyle rules (no `CHECKSTYLE:ON`/`CHECKSTYLE:OFF` comments, no adding entries to `tools/maven/suppressions.xml`, no `@SuppressWarnings`). Fix the code to satisfy checkstyle instead.
341- Add, change, or remove classes outside the `org.apache.flink.*` package (for example, classes copied from Calcite)
342- Modify `Parser.jj` (Calcite's generated parser grammar; expected to be removed in future Calcite upgrades)
343- Use destructive git operations unless explicitly requested
344 
345## References
346 
347- [README.md](README.md) — Build instructions and project overview
348- [DEVELOPMENT.md](DEVELOPMENT.md) — IDE setup and development environment
349- [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md) — Contribution process
350- [.github/PULL_REQUEST_TEMPLATE.md](.github/PULL_REQUEST_TEMPLATE.md) — PR checklist
351- [Code Style Guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/) — Detailed coding guidelines
352- [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) — AI tooling policy
353