| Dimension | Shared | Only in A | Only in B | Overlap |
|---|---|---|---|---|
| Sections | 0 | 25 | 18 | 0% |
| Commands | 0 | 10 | 3 | 0% |
| Section tags | 1 | 8 | 5 | 7% |
What each file covers
Sections
0 shared · 25 only in A · 18 only in B- − Agent Instructions for vLLM
- − 1. Contribution Policy (Mandatory)
- − Duplicate-work checks
- − No low-value busywork PRs
- − Accountability
- − Fail-closed behavior
- − 2. Development Workflow
- − Environment setup
- − Install `uv` if you don't have it already:
- − Always use `uv` for Python environment management:
- − Always make sure `pre-commit` and its hooks are installed:
- − Installing dependencies
- − If you are only making Python changes:
- − If you are also making C/C++ changes:
- − Tests
- − Install test dependencies (use cuda.in on non-x86_64):
- − Run a specific test file:
- − Running linters
- − Run all pre-commit hooks on staged files:
- − Run on all files:
- − Run a specific hook:
- − Run mypy as it is in CI:
- − Coding style guidelines
- − Commit messages
- − Domain-Specific Guides
- + AGENTS.md
- + Project Overview
- + Build & Test
- + Build release binary (rust/target/release/vllm-bench)
- + Run all tests
- + Run ignored integration tests (requires network for tokenizer download)
- + Architecture
- + Key Design Decisions
- + Common Issues
- + Typical Usage
- + Embedding benchmark (openai-embeddings, 8 inputs batched per request)
- + vLLM rerank benchmark (one query + 8 documents per request)
- + Prefix-cache stress (10 shared prefixes, 256+256 tokens)
- + Custom JSONL workload ({"prompt": ..., "output_tokens": ...} per line)
- + Random dataset
- + Random multimodal dataset (VLM benchmark)
- + HuggingFace dataset (WildChat)
- + HuggingFace dataset (LongBench with subset)
Commands
0 shared · 10 only in A · 3 only in B- − gh issue view <issue_number> --repo vllm-project/vllm --comments
- − gh pr list --repo vllm-project/vllm --state open --search "<issue_number> in:body"
- − gh pr list --repo vllm-project/vllm --state open --search "<short area keywords>"
- − uv venv --python 3.12
- − uv pip install -r requirements/lint.txt
- − uv pip install -e . --torch-backend=auto
- − uv pip install -r requirements/test/cuda.in
- − python3
- − pip
- − pip install
- + cargo build -p vllm-bench --release
- + cargo test -p vllm-bench
- + cargo test -p vllm-bench -- --ignored
Section tags
1 shared · 8 only in A · 5 only in B- − setup
- − lint-format
- − code-style
- − git-pr
- − dependencies
- − do-not
- − agent-behaviour
- − docs
- + build
- + architecture
- + performance
- + deployment
- + monorepo
- test
Line diff
vllm-project/vllm · AGENTS.md
@@ −1 @@
1# Agent Instructions for vLLM
2
3> These instructions apply to **all** AI-assisted contributions to `vllm-project/vllm`.
4> Breaching these guidelines can result in automatic banning.
5
6## 1. Contribution Policy (Mandatory)
7
8### Duplicate-work checks
9
10Before proposing a PR, run these checks:
11
12```bash
13gh issue view <issue_number> --repo vllm-project/vllm --comments
14gh pr list --repo vllm-project/vllm --state open --search "<issue_number> in:body"
15gh pr list --repo vllm-project/vllm --state open --search "<short area keywords>"
16```
17
18- If an open PR already addresses the same fix, do not open another.
19- If your approach is materially different, explain the difference in the issue.
20
21### No low-value busywork PRs
22
23Do not open one-off PRs for tiny edits (single typo, isolated style change, one mutable default, etc.). Mechanical cleanups are acceptable only when bundled with substantive work.
24
25### Accountability
26
27- Pure code-agent PRs are **not allowed**. A human submitter must understand and defend the change end-to-end.
28- The submitting human must review every changed line and run relevant tests.
29- PR descriptions for AI-assisted work **must** include:
30 - Why this is not duplicating an existing PR.
31 - Test commands run and results.
32 - Model evaluation results when the change affects output, accuracy, or serving.
33 - Clear statement that AI assistance was used.
34
35### Fail-closed behavior
36
37If work is duplicate/trivial busywork, **do not proceed**. Return a short explanation of what is missing.
38
39---
40
41## 2. Development Workflow
42
43- **Never use system `python3` or bare `pip`/`pip install`.** All Python commands must go through `uv` and `.venv/bin/python`.
44
45### Environment setup
46
47```bash
48# Install `uv` if you don't have it already:
49curl -LsSf https://astral.sh/uv/install.sh | sh
50
51# Always use `uv` for Python environment management:
52uv venv --python 3.12
53source .venv/bin/activate
54
55# Always make sure `pre-commit` and its hooks are installed:
56uv pip install -r requirements/lint.txt
57pre-commit install
58```
59
60### Installing dependencies
61
62```bash
63# If you are only making Python changes:
64VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
65
66# If you are also making C/C++ changes:
67uv pip install -e . --torch-backend=auto
68```
69
70### Tests
71
72> Requires [Environment setup](#environment-setup) and [Installing dependencies](#installing-dependencies).
73
74```bash
75# Install test dependencies (use cuda.in on non-x86_64):
76uv pip install -r requirements/test/cuda.in
77
78# Run a specific test file:
79.venv/bin/python -m pytest tests/path/to/test_file.py -v
80```
81
82When adding tests:
83
84- **Design before you write.** Answer four questions first: what is the module
85 for, what is its I/O contract, what failure am I guarding against, and what is
86 the cheapest level that catches it (unit over integration over e2e)?
87- **Reuse before create.** Extend existing test files, `conftest.py` fixtures, and
88 helpers; add a new file only when no nearby suite fits.
89- **Test behavior with intent.** Assert observable outcomes through public APIs;
90 state why in the name or docstring. Skip trivial wiring; flaky tests are worse
91 than no tests.
92- **Keep it minimal.** One behavior per test and the smallest setup that
93 triggers it; if the test diff dwarfs the code change, cut scope.
94- **No one-off kernel benchmarks in `tests/`.** Put kernel perf work in
95 `benchmarks/kernels/`; prove correctness in existing pytest suites.
96- **Run model evals for model-affecting changes.** Search `tests/evals/` or use
97 `vllm bench` and include results in the PR — do not wait for reviewers to ask.
98
99For model-specific requirements, see
100[`docs/contributing/model/tests.md`](docs/contributing/model/tests.md).
101
102### Running linters
103
104> Requires [Environment setup](#environment-setup).
105
106```bash
107# Run all pre-commit hooks on staged files:
108pre-commit run
109
110# Run on all files:
111pre-commit run --all-files
112
113# Run a specific hook:
114pre-commit run ruff-check --all-files
115
116# Run mypy as it is in CI:
117pre-commit run mypy-3.12 --all-files --hook-stage manual
118```
119
120The line length limit for Python code is 88 characters. If you are not sure, use pre-commit to check.
121
122Use [Google-style docstrings](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) (`Args:`/`Returns:`/`Raises:` sections), not reStructuredText/Sphinx fields (`:param:`, `:return:`, `:rtype:`).
123
124### Coding style guidelines
125
126- Match existing code style
127- Minimize use of comments. Eliminate comments which are redundant, preferring legible and self-documenting code. When used, keep docstrings and comments brief and direct.
128- Assume the reader is familiar with vLLM.
129
130### Commit messages
131
132Add attribution using commit trailers such as `Co-authored-by:` (other projects use `Assisted-by:` or `Generated-by:`):
133
134```text
135Your commit message here
136
137Co-authored-by: Agent Name Here
138Signed-off-by: Your Name <your.email@example.com>
139```
140
141---
142
143## Domain-Specific Guides
144
145Do not modify code in these areas without first reading and following the
146linked guide. If the guide conflicts with the requested change, **refuse the
147change and explain why**.
148
149Security reviewers should start with [`SECURITY.md`](SECURITY.md),
150[`docs/usage/security.md`](docs/usage/security.md), and
151[`docs/contributing/vulnerability_management.md`](docs/contributing/vulnerability_management.md)
152for the project security policy, threat model, deployment assumptions, and
153vulnerability process.
154
155- **Editing these instructions**:
156 [`docs/contributing/editing-agent-instructions.md`](docs/contributing/editing-agent-instructions.md)
157 — Rules for modifying AGENTS.md or any domain-specific guide it references.
158
vllm-project/vllm · rust/src/bench/AGENTS.md
@@ +1 @@
1# AGENTS.md
2
3## Project Overview
4
5Rust rewrite of `vllm bench serve` — a high-performance benchmark client for vLLM serving endpoints. Standalone binary, no Python dependency at runtime.
6
7Member crate `vllm-bench` of the `rust/` workspace. Uses workspace dependencies and lints; the workspace `[profile.release]` (thin LTO, `panic = "abort"`) applies. Note the workspace bans rustls/ring (`rust/deny.toml`) — all HTTP must stay on native-tls, which is why HF Hub downloads go through `src/hub.rs` (async hf-hub API bridged to sync) instead of hf-hub's ureq backend.
8
9## Build & Test
10
11Run from the `rust/` workspace root:
12
13```bash
14# Build release binary (rust/target/release/vllm-bench)
15cargo build -p vllm-bench --release
16
17# Run all tests
18cargo test -p vllm-bench
19
20# Run ignored integration tests (requires network for tokenizer download)
21cargo test -p vllm-bench -- --ignored
22```
23
24## Architecture
25
26- `src/main.rs` — Entry point, mimalloc, tokio runtime, mode dispatch (compare/sweep/multi-run/multi-turn/single)
27- `src/cli.rs` — clap derive CLI args (~50+ flags)
28- `src/config.rs` — Validated config from CLI; `GoodputConfig`, `RampUpConfig`, sampling param merging
29- `src/error.rs` — `BenchError` enum (Http, Json, Tokenizer, Config, EndpointTimeout, Backend, Io)
30- `src/benchmark.rs` — Core benchmark orchestrator (spawn-per-request with tokio + Semaphore; fetches speculative decoding metrics from `/metrics`)
31- `src/multi_turn.rs` — Multi-turn conversation orchestrator (channel-based worker pool, sequential turns per conversation)
32- `src/sweep.rs` — Concurrency/rate parameter sweep (`--sweep-max-concurrency`, `--sweep-request-rate`)
33- `src/multi_run.rs` — N-run aggregation with mean/std/min/max/CV (`--num-runs`)
34- `src/compare.rs` — Side-by-side diff of two result JSON files (`--compare`)
35- `src/tokenizer.rs` — `TokenizerKind` enum: Local(HuggingFace), Tiktoken, OR Server-side `/tokenize`+`/detokenize` fallback
36- `src/tiktoken.rs` — Tiktoken BPE loader (`.tiktoken`/`.model` files; built-in encodings o200k_base/cl100k_base; pat_str extraction from Python source)
37- `src/hub.rs` — `HubRepo`: sync facade over hf-hub's async (reqwest/native-tls) API — per-download thread with its own runtime; the sync ureq backend is unusable here because it pulls rustls, which `rust/deny.toml` bans
38- `src/rate_control.rs` — Gamma/Poisson request scheduling + linear/exponential ramp-up
39- `src/ready_checker.rs` — Endpoint readiness with retry
40- `src/backends/` — Backend implementations (enum dispatch, not trait objects)
41 - `mod.rs` — `Backend` enum, `RequestFuncInput`/`RequestFuncOutput` (includes `messages` field for multi-turn)
42 - `streaming.rs` — SSE parser (`StreamedResponseHandler`) with speculative JSON parse for split TCP segments
43 - `openai_completions.rs` — `/v1/completions` backend
44 - `openai_chat.rs` — `/v1/chat/completions` backend (uses `input.messages` when set; zero-copy raw JSON payload for multimodal)
45 - `pooling.rs` — Non-streaming pooling/embedding backends: `openai-embeddings`, `openai-embeddings-chat`, `vllm-pooling`, `vllm-rerank`
46- `src/datasets/random.rs` — Random dataset generation with rayon parallelism
47- `src/datasets/random_mm.rs` — Random multimodal dataset (synthetic JPEG images, bucket config sampling, pre-serialized JSON fragments); `--enable-multimodal-chat` pre-builds the chat `messages` array at dataset time (mirrors Python's `apply_multimodal_chat_transformation`)
48- `src/datasets/sharegpt.rs` — ShareGPT JSON loader + HuggingFace Hub auto-download with caching
49- `src/datasets/sonnet.rs` and `src/datasets/sonnet.txt` — Sonnet dataset (built-in Shakespeare sonnets via `include_str!("sonnet.txt")`; controllable token length + shared prefix; mirrors Python `SonnetDataset`)
50- `src/datasets/speed_bench.rs` — NVIDIA SPEED-Bench loader (HF datasets-server API, 6 configs, 11 categories, local cache)
51- `src/datasets/hf_dataset.rs` — Generic HuggingFace dataset loader (datasets-server API, column auto-detection)
52- `src/datasets/custom.rs` — Custom JSONL dataset (`{"prompt": ..., "output_tokens": ...}` per line; `--custom-output-len -1` uses per-line output_tokens; prompts always sent raw — no client-side chat template)
53- `src/datasets/prefix_repetition.rs` — Prefix repetition dataset (N shared prefixes × fresh random suffixes, standard prefix-cache stress; mirrors Python `PrefixRepetitionRandomDataset`)
54- `src/datasets/random_rerank.rs` — Random rerank dataset (one query + batched documents per request for `vllm-rerank`; `--no-reranker` for embedding-based scoring; mirrors Python `RandomDatasetForReranking`)
55- `src/datasets/multi_turn.rs` — Multi-turn synthetic generator + ShareGPT multi-turn loader (3-tier prefix sharing: global/conversation/unique-suffix; `per_turn_input_len`)
56- `src/metrics/mod.rs` — `BenchmarkMetrics` and `MultiTurnMetrics` structs
57- `src/metrics/calculator.rs` — TTFT/TPOT/ITL/E2EL/throughput stats, goodput SLO checking, peak concurrency, `calculate_multi_turn_metrics`
58- `src/metrics/steady_state.rs` — Steady-state window detection (in-flight concurrency plateau via two-pointer start/end merge) + plateau throughput/TTFT/TPOT; gated on `--max-concurrency` set + `--request-rate inf` (closed-loop)
59- `src/output/console.rs` — Terminal output matching Python format + multi-turn per-turn breakdown
60- `src/output/json.rs` — JSON result file (compatible with Python schema) + multi-turn JSON with `per_turn_metrics`
61
62## Key Design Decisions
63
64- **Enum dispatch** for backends (avoids async trait object issues with `dyn`)
65- **reqwest http1_only()** to match Python aiohttp behavior
66- **rayon** for parallel dataset generation (key perf win over Python)
67- **mimalloc** global allocator to reduce contention at 1400+ concurrency (page-agnostic; works on aarch64 64K-page kernels where jemalloc aborts with `LG_PAGE=12` builds)
68- **Arc\<str\> prompts** zero-copy sharing across tokio tasks (~3GB savings at 100k prompts with 8k-token inputs)
69- **Spawn-per-request** `tokio::spawn` + `Semaphore` (matches Python asyncio pattern)
70- **Speculative JSON parse** in SSE handler — detects complete JSON before `\n\n` arrives, improving TTFT/ITL accuracy when TCP segments split
71- **Tokenizer fallback chain**: Local HF → Tiktoken (`.tiktoken`/`.model` + built-in encodings) → Server-side `/tokenize`+`/detokenize`. Blocking HTTP in rayon threads for server fallback.
72- **hf-hub** for downloading tokenizers and datasets from HuggingFace Hub
73- **Pre-serialized mm fragments** (`Arc<str>`) for multimodal: image content stored as JSON strings, zero-copy concatenated into payload — avoids deep-cloning ~200KB+ base64 per request
74- **Steady-state metrics** (default-on in closed-loop): measure throughput/TTFT/TPOT only over the saturated plateau to cut run-to-run variance at high concurrency; `steady_state` is an `Option` in JSON (`#[serde(default)]` for backward compat), null when the scope gate fails or `--no-steady-state`
75- **`--prompt-token-ids`** (random dataset only): send token-ID arrays instead of text to skip server-side tokenization; also skips the token-length verification pass (counts exact by construction)
76- **`--random-range-ratio`** follows Python semantics: lengths sampled uniformly from `[len*(1-r), len*(1+r)]`, default `0.0` = fixed; accepts a float in `[0,1)` or `'{"input": r1, "output": r2}'`. (The pre-2026-07 Rust-only form `[len*r, len]` with default 1.0 is rejected with a migration hint.)
77- **`prompt_list`** (`Arc<[Arc<str>]>` on `SampleRequest`/`RequestFuncInput`): multiple inputs per request for pooling backends — embeddings batches (`--random-batch-size`) send `"input": [...]`, rerank sends `[0]` as query + `[1..]` as documents
78- JSON output schema must match Python `vllm bench serve` exactly
79
80## Common Issues
81
82- **localhost vs 127.0.0.1**: Some systems resolve `localhost` to IPv6 `::1` while vLLM listens on IPv4 only. Use `127.0.0.1` or the actual hostname.
83- **Models without tokenizer.json** (e.g., `nvidia/Kimi-K2.5-NVFP4`): Automatically falls back to server-side tokenization. Can also use `--tokenizer` to point to a model with `tokenizer.json`.
84- **usage.completion_tokens parsing**: vLLM sends final usage chunk with `"choices":[]` (empty array). The usage `if` must be separate from the choices `if` (not `else if`).
85
86## Typical Usage
87
88```bash
89# Embedding benchmark (openai-embeddings, 8 inputs batched per request)
90./target/release/vllm-bench \
91 --backend openai-embeddings \
92 --base-url http://gb200-10:30000 \
93 --model BAAI/bge-large-en-v1.5 \
94 --dataset-name random \
95 --random-input-len 512 \
96 --random-batch-size 8 \
97 --num-prompts 1000 \
98 --save-result
99
100# vLLM rerank benchmark (one query + 8 documents per request)
101./target/release/vllm-bench \
102 --backend vllm-rerank \
103 --base-url http://gb200-10:30000 \
104 --model BAAI/bge-reranker-v2-m3 \
105 --dataset-name random-rerank \
106 --random-input-len 512 \
107 --random-batch-size 8 \
108 --num-prompts 500 \
109 --save-result
110
111# Prefix-cache stress (10 shared prefixes, 256+256 tokens)
112./target/release/vllm-bench \
113 --backend vllm \
114 --base-url http://gb200-10:30000 \
115 --model nvidia/Kimi-K2.5-NVFP4 \
116 --dataset-name prefix_repetition \
117 --prefix-repetition-prefix-len 256 \
118 --prefix-repetition-suffix-len 256 \
119 --prefix-repetition-num-prefixes 10 \
120 --num-prompts 1000
121
122# Custom JSONL workload ({"prompt": ..., "output_tokens": ...} per line)
123./target/release/vllm-bench \
124 --backend openai-chat \
125 --base-url http://gb200-10:30000 \
126 --model nvidia/Kimi-K2.5-NVFP4 \
127 --dataset-name custom \
128 --dataset-path workload.jsonl \
129 --custom-output-len -1 \
130 --num-prompts 1000
131
132# Random dataset
133./target/release/vllm-bench \
134 --backend vllm \
135 --base-url http://gb200-10:30000 \
136 --model nvidia/Kimi-K2.5-NVFP4 \
137 --dataset-name random \
138 --random-input-len 8192 \
139 --random-output-len 1024 \
140 --ignore-eos \
141 --num-prompts 4096 \
142 --percentile-metrics "ttft,tpot,itl,e2el" \
143 --save-result \
144 --max-concurrency 1400
145
146# Random multimodal dataset (VLM benchmark)
147./target/release/vllm-bench \
148 --backend openai-chat \
149 --base-url http://gb200-10:30000 \
150 --model Qwen/Qwen2.5-VL-7B-Instruct \
151 --dataset-name random-mm \
152 --random-input-len 512 \
153 --random-output-len 128 \
154 --num-prompts 100 \
155 --random-mm-base-items-per-request 1 \
156 --random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
157 --random-mm-bucket-config '{(1024, 800, 1): 1.0}'
158
159# HuggingFace dataset (WildChat)
160./target/release/vllm-bench \
161 --backend openai-chat \
162 --base-url http://gb200-10:30000 \
163 --model nvidia/Kimi-K2.5-NVFP4 \
164 --dataset-name hf \
165 --dataset-path allenai/WildChat-4.8M \
166 --hf-split train \
167 --num-prompts 1000 \
168 --save-result
169
170# HuggingFace dataset (LongBench with subset)
171./target/release/vllm-bench \
172 --backend openai-chat \
173 --base-url http://gb200-10:30000 \
174 --model nvidia/Kimi-K2.5-NVFP4 \
175 --dataset-name hf \
176 --dataset-path THUDM/LongBench \
177 --hf-subset narrativeqa \
178 --hf-split test \
179 --hf-output-len 512 \
180 --num-prompts 200
181```
182
@@ −1 +1 @@
1−# Agent Instructions for vLLM
1+# AGENTS.md
22
3−> These instructions apply to **all** AI-assisted contributions to `vllm-project/vllm`.
4−> Breaching these guidelines can result in automatic banning.
3+## Project Overview
54
6−## 1. Contribution Policy (Mandatory)
5+Rust rewrite of `vllm bench serve` — a high-performance benchmark client for vLLM serving endpoints. Standalone binary, no Python dependency at runtime.
76
8−### Duplicate-work checks
7+Member crate `vllm-bench` of the `rust/` workspace. Uses workspace dependencies and lints; the workspace `[profile.release]` (thin LTO, `panic = "abort"`) applies. Note the workspace bans rustls/ring (`rust/deny.toml`) — all HTTP must stay on native-tls, which is why HF Hub downloads go through `src/hub.rs` (async hf-hub API bridged to sync) instead of hf-hub's ureq backend.
98
10−Before proposing a PR, run these checks:
9+## Build & Test
1110
12−```bash
13−gh issue view <issue_number> --repo vllm-project/vllm --comments
14−gh pr list --repo vllm-project/vllm --state open --search "<issue_number> in:body"
15−gh pr list --repo vllm-project/vllm --state open --search "<short area keywords>"
16−```
11+Run from the `rust/` workspace root:
1712
18−- If an open PR already addresses the same fix, do not open another.
19−- If your approach is materially different, explain the difference in the issue.
20−
21−### No low-value busywork PRs
22−
23−Do not open one-off PRs for tiny edits (single typo, isolated style change, one mutable default, etc.). Mechanical cleanups are acceptable only when bundled with substantive work.
24−
25−### Accountability
26−
27−- Pure code-agent PRs are **not allowed**. A human submitter must understand and defend the change end-to-end.
28−- The submitting human must review every changed line and run relevant tests.
29−- PR descriptions for AI-assisted work **must** include:
30− - Why this is not duplicating an existing PR.
31− - Test commands run and results.
32− - Model evaluation results when the change affects output, accuracy, or serving.
33− - Clear statement that AI assistance was used.
34−
35−### Fail-closed behavior
36−
37−If work is duplicate/trivial busywork, **do not proceed**. Return a short explanation of what is missing.
38−
39−---
40−
41−## 2. Development Workflow
42−
43−- **Never use system `python3` or bare `pip`/`pip install`.** All Python commands must go through `uv` and `.venv/bin/python`.
44−
45−### Environment setup
46−
4713 ```bash
48−# Install `uv` if you don't have it already:
49−curl -LsSf https://astral.sh/uv/install.sh | sh
14+# Build release binary (rust/target/release/vllm-bench)
15+cargo build -p vllm-bench --release
5016
51−# Always use `uv` for Python environment management:
52−uv venv --python 3.12
53−source .venv/bin/activate
17+# Run all tests
18+cargo test -p vllm-bench
5419
55−# Always make sure `pre-commit` and its hooks are installed:
56−uv pip install -r requirements/lint.txt
57−pre-commit install
20+# Run ignored integration tests (requires network for tokenizer download)
21+cargo test -p vllm-bench -- --ignored
5822 ```
5923
60−### Installing dependencies
24+## Architecture
6125
62−```bash
63−# If you are only making Python changes:
64−VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
26+- `src/main.rs` — Entry point, mimalloc, tokio runtime, mode dispatch (compare/sweep/multi-run/multi-turn/single)
27+- `src/cli.rs` — clap derive CLI args (~50+ flags)
28+- `src/config.rs` — Validated config from CLI; `GoodputConfig`, `RampUpConfig`, sampling param merging
29+- `src/error.rs` — `BenchError` enum (Http, Json, Tokenizer, Config, EndpointTimeout, Backend, Io)
30+- `src/benchmark.rs` — Core benchmark orchestrator (spawn-per-request with tokio + Semaphore; fetches speculative decoding metrics from `/metrics`)
31+- `src/multi_turn.rs` — Multi-turn conversation orchestrator (channel-based worker pool, sequential turns per conversation)
32+- `src/sweep.rs` — Concurrency/rate parameter sweep (`--sweep-max-concurrency`, `--sweep-request-rate`)
33+- `src/multi_run.rs` — N-run aggregation with mean/std/min/max/CV (`--num-runs`)
34+- `src/compare.rs` — Side-by-side diff of two result JSON files (`--compare`)
35+- `src/tokenizer.rs` — `TokenizerKind` enum: Local(HuggingFace), Tiktoken, OR Server-side `/tokenize`+`/detokenize` fallback
36+- `src/tiktoken.rs` — Tiktoken BPE loader (`.tiktoken`/`.model` files; built-in encodings o200k_base/cl100k_base; pat_str extraction from Python source)
37+- `src/hub.rs` — `HubRepo`: sync facade over hf-hub's async (reqwest/native-tls) API — per-download thread with its own runtime; the sync ureq backend is unusable here because it pulls rustls, which `rust/deny.toml` bans
38+- `src/rate_control.rs` — Gamma/Poisson request scheduling + linear/exponential ramp-up
39+- `src/ready_checker.rs` — Endpoint readiness with retry
40+- `src/backends/` — Backend implementations (enum dispatch, not trait objects)
41+ - `mod.rs` — `Backend` enum, `RequestFuncInput`/`RequestFuncOutput` (includes `messages` field for multi-turn)
42+ - `streaming.rs` — SSE parser (`StreamedResponseHandler`) with speculative JSON parse for split TCP segments
43+ - `openai_completions.rs` — `/v1/completions` backend
44+ - `openai_chat.rs` — `/v1/chat/completions` backend (uses `input.messages` when set; zero-copy raw JSON payload for multimodal)
45+ - `pooling.rs` — Non-streaming pooling/embedding backends: `openai-embeddings`, `openai-embeddings-chat`, `vllm-pooling`, `vllm-rerank`
46+- `src/datasets/random.rs` — Random dataset generation with rayon parallelism
47+- `src/datasets/random_mm.rs` — Random multimodal dataset (synthetic JPEG images, bucket config sampling, pre-serialized JSON fragments); `--enable-multimodal-chat` pre-builds the chat `messages` array at dataset time (mirrors Python's `apply_multimodal_chat_transformation`)
48+- `src/datasets/sharegpt.rs` — ShareGPT JSON loader + HuggingFace Hub auto-download with caching
49+- `src/datasets/sonnet.rs` and `src/datasets/sonnet.txt` — Sonnet dataset (built-in Shakespeare sonnets via `include_str!("sonnet.txt")`; controllable token length + shared prefix; mirrors Python `SonnetDataset`)
50+- `src/datasets/speed_bench.rs` — NVIDIA SPEED-Bench loader (HF datasets-server API, 6 configs, 11 categories, local cache)
51+- `src/datasets/hf_dataset.rs` — Generic HuggingFace dataset loader (datasets-server API, column auto-detection)
52+- `src/datasets/custom.rs` — Custom JSONL dataset (`{"prompt": ..., "output_tokens": ...}` per line; `--custom-output-len -1` uses per-line output_tokens; prompts always sent raw — no client-side chat template)
53+- `src/datasets/prefix_repetition.rs` — Prefix repetition dataset (N shared prefixes × fresh random suffixes, standard prefix-cache stress; mirrors Python `PrefixRepetitionRandomDataset`)
54+- `src/datasets/random_rerank.rs` — Random rerank dataset (one query + batched documents per request for `vllm-rerank`; `--no-reranker` for embedding-based scoring; mirrors Python `RandomDatasetForReranking`)
55+- `src/datasets/multi_turn.rs` — Multi-turn synthetic generator + ShareGPT multi-turn loader (3-tier prefix sharing: global/conversation/unique-suffix; `per_turn_input_len`)
56+- `src/metrics/mod.rs` — `BenchmarkMetrics` and `MultiTurnMetrics` structs
57+- `src/metrics/calculator.rs` — TTFT/TPOT/ITL/E2EL/throughput stats, goodput SLO checking, peak concurrency, `calculate_multi_turn_metrics`
58+- `src/metrics/steady_state.rs` — Steady-state window detection (in-flight concurrency plateau via two-pointer start/end merge) + plateau throughput/TTFT/TPOT; gated on `--max-concurrency` set + `--request-rate inf` (closed-loop)
59+- `src/output/console.rs` — Terminal output matching Python format + multi-turn per-turn breakdown
60+- `src/output/json.rs` — JSON result file (compatible with Python schema) + multi-turn JSON with `per_turn_metrics`
6561
66−# If you are also making C/C++ changes:
67−uv pip install -e . --torch-backend=auto
68−```
62+## Key Design Decisions
6963
70−### Tests
64+- **Enum dispatch** for backends (avoids async trait object issues with `dyn`)
65+- **reqwest http1_only()** to match Python aiohttp behavior
66+- **rayon** for parallel dataset generation (key perf win over Python)
67+- **mimalloc** global allocator to reduce contention at 1400+ concurrency (page-agnostic; works on aarch64 64K-page kernels where jemalloc aborts with `LG_PAGE=12` builds)
68+- **Arc\<str\> prompts** zero-copy sharing across tokio tasks (~3GB savings at 100k prompts with 8k-token inputs)
69+- **Spawn-per-request** `tokio::spawn` + `Semaphore` (matches Python asyncio pattern)
70+- **Speculative JSON parse** in SSE handler — detects complete JSON before `\n\n` arrives, improving TTFT/ITL accuracy when TCP segments split
71+- **Tokenizer fallback chain**: Local HF → Tiktoken (`.tiktoken`/`.model` + built-in encodings) → Server-side `/tokenize`+`/detokenize`. Blocking HTTP in rayon threads for server fallback.
72+- **hf-hub** for downloading tokenizers and datasets from HuggingFace Hub
73+- **Pre-serialized mm fragments** (`Arc<str>`) for multimodal: image content stored as JSON strings, zero-copy concatenated into payload — avoids deep-cloning ~200KB+ base64 per request
74+- **Steady-state metrics** (default-on in closed-loop): measure throughput/TTFT/TPOT only over the saturated plateau to cut run-to-run variance at high concurrency; `steady_state` is an `Option` in JSON (`#[serde(default)]` for backward compat), null when the scope gate fails or `--no-steady-state`
75+- **`--prompt-token-ids`** (random dataset only): send token-ID arrays instead of text to skip server-side tokenization; also skips the token-length verification pass (counts exact by construction)
76+- **`--random-range-ratio`** follows Python semantics: lengths sampled uniformly from `[len*(1-r), len*(1+r)]`, default `0.0` = fixed; accepts a float in `[0,1)` or `'{"input": r1, "output": r2}'`. (The pre-2026-07 Rust-only form `[len*r, len]` with default 1.0 is rejected with a migration hint.)
77+- **`prompt_list`** (`Arc<[Arc<str>]>` on `SampleRequest`/`RequestFuncInput`): multiple inputs per request for pooling backends — embeddings batches (`--random-batch-size`) send `"input": [...]`, rerank sends `[0]` as query + `[1..]` as documents
78+- JSON output schema must match Python `vllm bench serve` exactly
7179
72−> Requires [Environment setup](#environment-setup) and [Installing dependencies](#installing-dependencies).
80+## Common Issues
7381
74−```bash
75−# Install test dependencies (use cuda.in on non-x86_64):
76−uv pip install -r requirements/test/cuda.in
82+- **localhost vs 127.0.0.1**: Some systems resolve `localhost` to IPv6 `::1` while vLLM listens on IPv4 only. Use `127.0.0.1` or the actual hostname.
83+- **Models without tokenizer.json** (e.g., `nvidia/Kimi-K2.5-NVFP4`): Automatically falls back to server-side tokenization. Can also use `--tokenizer` to point to a model with `tokenizer.json`.
84+- **usage.completion_tokens parsing**: vLLM sends final usage chunk with `"choices":[]` (empty array). The usage `if` must be separate from the choices `if` (not `else if`).
7785
78−# Run a specific test file:
79−.venv/bin/python -m pytest tests/path/to/test_file.py -v
80−```
86+## Typical Usage
8187
82−When adding tests:
83−
84−- **Design before you write.** Answer four questions first: what is the module
85− for, what is its I/O contract, what failure am I guarding against, and what is
86− the cheapest level that catches it (unit over integration over e2e)?
87−- **Reuse before create.** Extend existing test files, `conftest.py` fixtures, and
88− helpers; add a new file only when no nearby suite fits.
89−- **Test behavior with intent.** Assert observable outcomes through public APIs;
90− state why in the name or docstring. Skip trivial wiring; flaky tests are worse
91− than no tests.
92−- **Keep it minimal.** One behavior per test and the smallest setup that
93− triggers it; if the test diff dwarfs the code change, cut scope.
94−- **No one-off kernel benchmarks in `tests/`.** Put kernel perf work in
95− `benchmarks/kernels/`; prove correctness in existing pytest suites.
96−- **Run model evals for model-affecting changes.** Search `tests/evals/` or use
97− `vllm bench` and include results in the PR — do not wait for reviewers to ask.
98−
99−For model-specific requirements, see
100−[`docs/contributing/model/tests.md`](docs/contributing/model/tests.md).
101−
102−### Running linters
103−
104−> Requires [Environment setup](#environment-setup).
105−
10688 ```bash
107−# Run all pre-commit hooks on staged files:
108−pre-commit run
89+# Embedding benchmark (openai-embeddings, 8 inputs batched per request)
90+./target/release/vllm-bench \
91+ --backend openai-embeddings \
92+ --base-url http://gb200-10:30000 \
93+ --model BAAI/bge-large-en-v1.5 \
94+ --dataset-name random \
95+ --random-input-len 512 \
96+ --random-batch-size 8 \
97+ --num-prompts 1000 \
98+ --save-result
10999
110−# Run on all files:
111−pre-commit run --all-files
100+# vLLM rerank benchmark (one query + 8 documents per request)
101+./target/release/vllm-bench \
102+ --backend vllm-rerank \
103+ --base-url http://gb200-10:30000 \
104+ --model BAAI/bge-reranker-v2-m3 \
105+ --dataset-name random-rerank \
106+ --random-input-len 512 \
107+ --random-batch-size 8 \
108+ --num-prompts 500 \
109+ --save-result
112110
113−# Run a specific hook:
114−pre-commit run ruff-check --all-files
111+# Prefix-cache stress (10 shared prefixes, 256+256 tokens)
112+./target/release/vllm-bench \
113+ --backend vllm \
114+ --base-url http://gb200-10:30000 \
115+ --model nvidia/Kimi-K2.5-NVFP4 \
116+ --dataset-name prefix_repetition \
117+ --prefix-repetition-prefix-len 256 \
118+ --prefix-repetition-suffix-len 256 \
119+ --prefix-repetition-num-prefixes 10 \
120+ --num-prompts 1000
115121
116−# Run mypy as it is in CI:
117−pre-commit run mypy-3.12 --all-files --hook-stage manual
118−```
122+# Custom JSONL workload ({"prompt": ..., "output_tokens": ...} per line)
123+./target/release/vllm-bench \
124+ --backend openai-chat \
125+ --base-url http://gb200-10:30000 \
126+ --model nvidia/Kimi-K2.5-NVFP4 \
127+ --dataset-name custom \
128+ --dataset-path workload.jsonl \
129+ --custom-output-len -1 \
130+ --num-prompts 1000
119131
120−The line length limit for Python code is 88 characters. If you are not sure, use pre-commit to check.
132+# Random dataset
133+./target/release/vllm-bench \
134+ --backend vllm \
135+ --base-url http://gb200-10:30000 \
136+ --model nvidia/Kimi-K2.5-NVFP4 \
137+ --dataset-name random \
138+ --random-input-len 8192 \
139+ --random-output-len 1024 \
140+ --ignore-eos \
141+ --num-prompts 4096 \
142+ --percentile-metrics "ttft,tpot,itl,e2el" \
143+ --save-result \
144+ --max-concurrency 1400
121145
122−Use [Google-style docstrings](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) (`Args:`/`Returns:`/`Raises:` sections), not reStructuredText/Sphinx fields (`:param:`, `:return:`, `:rtype:`).
146+# Random multimodal dataset (VLM benchmark)
147+./target/release/vllm-bench \
148+ --backend openai-chat \
149+ --base-url http://gb200-10:30000 \
150+ --model Qwen/Qwen2.5-VL-7B-Instruct \
151+ --dataset-name random-mm \
152+ --random-input-len 512 \
153+ --random-output-len 128 \
154+ --num-prompts 100 \
155+ --random-mm-base-items-per-request 1 \
156+ --random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
157+ --random-mm-bucket-config '{(1024, 800, 1): 1.0}'
123158
124−### Coding style guidelines
159+# HuggingFace dataset (WildChat)
160+./target/release/vllm-bench \
161+ --backend openai-chat \
162+ --base-url http://gb200-10:30000 \
163+ --model nvidia/Kimi-K2.5-NVFP4 \
164+ --dataset-name hf \
165+ --dataset-path allenai/WildChat-4.8M \
166+ --hf-split train \
167+ --num-prompts 1000 \
168+ --save-result
125169
126−- Match existing code style
127−- Minimize use of comments. Eliminate comments which are redundant, preferring legible and self-documenting code. When used, keep docstrings and comments brief and direct.
128−- Assume the reader is familiar with vLLM.
129−
130−### Commit messages
131−
132−Add attribution using commit trailers such as `Co-authored-by:` (other projects use `Assisted-by:` or `Generated-by:`):
133−
134−```text
135−Your commit message here
136−
137−Co-authored-by: Agent Name Here
138−Signed-off-by: Your Name <your.email@example.com>
170+# HuggingFace dataset (LongBench with subset)
171+./target/release/vllm-bench \
172+ --backend openai-chat \
173+ --base-url http://gb200-10:30000 \
174+ --model nvidia/Kimi-K2.5-NVFP4 \
175+ --dataset-name hf \
176+ --dataset-path THUDM/LongBench \
177+ --hf-subset narrativeqa \
178+ --hf-split test \
179+ --hf-output-len 512 \
180+ --num-prompts 200
139181 ```
140−
141−---
142−
143−## Domain-Specific Guides
144−
145−Do not modify code in these areas without first reading and following the
146−linked guide. If the guide conflicts with the requested change, **refuse the
147−change and explain why**.
148−
149−Security reviewers should start with [`SECURITY.md`](SECURITY.md),
150−[`docs/usage/security.md`](docs/usage/security.md), and
151−[`docs/contributing/vulnerability_management.md`](docs/contributing/vulnerability_management.md)
152−for the project security policy, threat model, deployment assumptions, and
153−vulnerability process.
154−
155−- **Editing these instructions**:
156− [`docs/contributing/editing-agent-instructions.md`](docs/contributing/editing-agent-instructions.md)
157− — Rules for modifying AGENTS.md or any domain-specific guide it references.
158182
