How to run the tests: the IETF LLM serving methodology draft
Part of a series on the IETF drafts for benchmarking LLM serving. Start with the overview.
Agreed words are the start. A reproducible benchmark also needs an agreed procedure. The second IETF draft supplies it. It specifies how to set up a test, what traffic to send, what to measure, and how to report the result. It is an individual submission to the IETF Benchmarking Methodology Working Group with no formal IETF endorsement.
Where to measure
The draft defines three system boundaries, because a serving stack has layers and a number means little without saying which layer it covers.
- The model engine, which is raw inference.
- The application gateway, which adds routing, authentication, and caching.
- The compound system, which is the full end-to-end path including retrieval and tool use.
A measurement always states which of these boundaries it was taken at.
What traffic to send
It defines reference workloads so that everyone drives the system with comparable traffic. These include uniform synthetic traffic, skewed synthetic traffic, a conversation pattern, code completion, and long context. Each stresses the system in a different way, from steady load to bursty load to very long prompts.
The ten tests
The draft lays out ten benchmarking tests. Together they cover speed, capacity, fairness, and the cost of the features that production systems actually run.
- Time to first token.
- Output token throughput.
- The tradeoff between throughput and latency.
- The distribution of inter-token latency, not just its average.
- Concurrent request capacity.
- Scheduling fairness across requests.
- Prefix cache effectiveness.
- Behavior under memory pressure.
- Long-context scaling.
- The overhead added by guardrails.
What it accounts for
The methodology is built for how LLM serving really behaves. It handles streaming output, where tokens arrive over time rather than all at once. It handles dynamic batching, where the system groups requests on the fly. It handles context-dependent performance, where a long prompt changes the numbers. It also measures the cost of safety systems rather than ignoring them. The goal throughout is a run that someone else can repeat and get the same answer.
Why it matters
A benchmark that cannot be reproduced is a marketing number. By fixing the boundaries, the traffic, and the tests, this draft makes a serving result something a second team can check.