Machine-readable signals, not prose

An agent should not have to parse a paragraph of English to learn how to behave. The API should hand it structured, actionable signals instead. A rule expressed as a field the agent can read is worth more than the same rule buried in a sentence it has to interpret.

The description is the contract

For an agent, the API description is the thing it actually programs against. Two properties follow. Operation identifiers must be stable and meaningful, because the agent selects operations by their names and descriptions. The description of each operation should say plainly when to use it and what side effects it has. A side effect is a change the call makes to the world, such as charging a card or sending a message.

Consistency and predictability

The API should look the same everywhere. Naming, pagination, error formats, and authentication should follow one convention across every operation. An agent that learns the pattern once can then apply it everywhere, which saves the limited context it has to work with.

Respect the context budget

An agent can only hold so much text at a time. A response that dumps everything eats into that budget and crowds out the agent’s actual reasoning. The draft asks that responses be bounded by default, with controls for field selection or verbosity so the agent can ask for only what it needs.

Errors the agent can recover from

When a call fails, the failure should be machine-readable. The draft points to RFC 9457 Problem Details, a standard JSON format for HTTP errors. A good error says whether retrying is worthwhile and gives field-level feedback on what was invalid, so the agent can fix the request rather than guess.

Safe writes

Agents retry often, so a write operation must tolerate being repeated. The draft calls for idempotency keys, which let the agent mark a request so that sending it twice has the same effect as sending it once. It also calls for dry-run and preview modes, so an agent can check what a write would do before committing to it.

Tell the agent when to slow down

Rate limits should be expressed as machine-readable backoff hints. When the service wants the client to wait, it should say so in a form the agent can read directly, so the agent can throttle itself rather than hammer the endpoint and get blocked.

Why it matters

Agents are becoming a primary audience for APIs, and they have real constraints: a small context window, frequent retries, and selection by description. An API that ignores these makes agents slow, expensive, and error-prone to work with. The profile gives API designers a concrete checklist for serving autonomous clients while keeping security, reliability, and operational visibility intact.

Read the draft

Design Considerations and Profile for HTTP APIs Consumed by AI Agents.

How many should there be

There is a well-known way to guess how many Fibonacci primes lie below some cutoff. The Fibonacci numbers grow at a steady multiplicative rate, so a Fibonacci number deep in the sequence has a predictable size. A number of that size is prime with a chance that shrinks as the number grows, in the usual way primes thin out. A classical divisibility fact narrows the search further. A Fibonacci number can only be prime when its position is itself prime. Combining these gives a clean prediction. The number of Fibonacci primes up to a cutoff grows in step with the logarithm of that cutoff, times a fixed constant.

This is the Grantham and Granville heuristic (Grantham and Granville 2024). Run it out to the current search depth and it predicts about 60 positions. We observe 57. The agreement is close.

There is a wrinkle worth airing. A quick fit of the constant comes out about 19% higher than the value the theory predicts. The study tracks the gap down. Fitting the constant against the Fibonacci data alone lands within 3% of the theoretical value. Pooling several related sequences gives a range that comfortably contains it. The apparent excess turns out to be a mix of a bookkeeping choice in the formula and ordinary small-sample noise. With only 57 points, the data cannot even tell the theoretical constant apart from a nearby look-alike value. Settling that would take far more data than the search has produced.

Predicting the next one

If the positions are spread out with no memory of each other, you can still say something useful about the next one. Look at the gaps between consecutive positions on a logarithmic scale. Those gaps behave like the waiting times in a process that has no memory, the same statistics that govern how long you wait for the next random arrival. Standard tests back this up.

That observation supports a prediction with an error bar, which the earlier work never attempted. From the data available up to a point, the method gives a 90% window for where the next position should fall. The position that was actually found next landed inside the window. Replaying this one step at a time across the whole list, the true position fell inside its predicted window about 87% of the time. For a method with a single tunable number, landing near the promised 90% is about as well as one can ask.

Looking for hidden patterns

The study then hunts for structure in the positions. It checks whether they favour certain remainders, whether neighbours influence each other, whether some special families of primes are over- or under-represented, and whether the leading digits follow the lopsided distribution that many natural datasets show. Every test comes back empty.

As a single overall check, the authors generated three thousand fake datasets from the no-pattern model and compared nine summary measurements. The real data looked unremarkable on all nine. The honest conclusion is stated plainly. At 57 points there is no pattern to find. The biases people detect in primes are measured across millions of cases. A list of 57 is far too short to see effects that faint.

Where structure does live

The positions of the primes carry no usable structure. The way primes divide these sequences is a different story, and here the picture is exact.

For the Fibonacci numbers, ask which is the first position a given prime divides. A classical result of Lagarias says that for two thirds of primes that first position is even (Lagarias 1985). The study confirms this to high precision across many primes. The finer details of this divisibility follow from a known proof built on a symmetry argument from algebra (Cubre and Rouse 2014).

The study then moves to relatives of the Fibonacci numbers built from longer recurrences, where each term sums the previous three, four, or more. These no longer have the clean divisibility property. Their behaviour is instead controlled by a symmetry group attached to the recurrence, and a classical theorem turns that symmetry into exact proportions for how the sequence repeats when reduced by a prime. The study confirms these proportions numerically for the three-term and four-term cases.

For the three-term case there is one quantity the study can only estimate. It measures it at about 0.40 and is careful to flag that the exact value is not known. The study also corrects a claim in an earlier draft that said these finer densities were computable for any length of recurrence. For the general case they are open, with a partial result under a standard unproven hypothesis as the best that is currently known (Järviniemi 2022).

What is known and what is new

The point of the study is the line it draws between the two. The count estimate, the matching constant, the two-thirds divisibility result, the symmetry-driven proportions, and the repetition structure of the longer sequences are established mathematics with citations. The new contributions are smaller and clearly marked. There is a prediction window with a measured success rate. There is a reusable battery of tests for telling structure apart from noise. There is a fully cited reproduction that anyone can run.

Reproducibility

Every result comes from self-contained Python scripts using standard scientific libraries. The input data are the published index lists for the Fibonacci, Lucas, and related sequences. The code is at github.com/krimler/k-bonacci.

Read the paper

The full paper, with the precise statements, the constants, and the proofs it draws on, is here: Fibonacci and k-bonacci prime statistics (PDF and details).

References

Cubre, Paul, and Jeremy Rouse. 2014. “Divisibility Properties of the Fibonacci Entry Point.” Proceedings of the American Mathematical Society 142 (11): 3771–85. https://arxiv.org/abs/1212.6221.

Grantham, Jon, and Andrew Granville. 2024. Fibonacci Primes, Primes of the Form , and Beyond. https://arxiv.org/abs/2307.07894.

Järviniemi, Olli. 2022. “Positive Lower Density for Prime Divisors of Generic Linear Recurrences.” Mathematical Proceedings of the Cambridge Philosophical Society. https://arxiv.org/abs/2102.04042.

Lagarias, Jeffrey C. 1985. “The Set of Primes Dividing the Lucas Numbers Has Density 2/3.” Pacific Journal of Mathematics 118 (2): 449–61.

What the paper does

Large AI training jobs are synchronous. Every GPU waits for the slowest transfer in a step. So one bad link can stall the whole job. At 100,000 GPUs, network failures stop being rare. They become a daily event.

The paper offers a three-part answer.

First, a new transport called MRC, short for Multipath RC. It extends RoCEv2. It sprays each queue pair across many paths instead of pinning it to one. It balances load with ECN signals. It recovers losses with selective ACK and packet trimming. It turns off PFC and runs the fabric in best-effort lossy mode.

Second, a multi-plane Clos topology. Take an 800 Gb/s NIC and split it into 8 lanes of 100 Gb/s. Build 8 parallel planes from the same switches. Now a 100,000 GPU cluster needs only two switch tiers, not three. This lowers latency, lowers cost and power, and shrinks the blast radius of any one link.

Third, static source routing with SRv6. The endpoint writes the full path into each packet using uN micro-segments. Dynamic routing in the switches is turned off completely. So the endpoint owns failure handling, not the switch.

This runs in production at OpenAI and Microsoft. It has trained frontier models including ChatGPT and Codex. It is implemented on NVIDIA CX-8, AMD Pollara and Vulcano, and Broadcom Thor Ultra NICs. The spec is released openly through OCP.

What works well

The design choices are bold and well argued. Turning off dynamic routing sounds backwards at first. Turning off PFC sounds risky too. But the reasoning is clean. Two adaptive systems fighting each other add no value. MRC already routes around failures on its own. So let it do that job alone.

The blast-radius argument is concrete and convincing. Losing one T0-to-T1 link in an 800 Gb/s single plane cuts a node’s capacity by about 3 percent. The same loss in a 100 Gb/s plane costs only about 0.4 percent. So the job barely notices a link flap. The team even leaves flapping links in service and repairs them later.

The observability insight is elegant. With static source routing, a probe takes the exact same path a data packet would take. So a Clustermapper probe gives ground truth on the forwarding plane. Switch telemetry cannot match this. This is why they can run a relaxed repair posture and still feel safe.

The multi-vendor breadth is a real systems achievement. The same protocol works across three NIC vendors and several switch platforms. That is strong evidence the design generalizes.

The honesty is refreshing. The team flags the single points of failure that remain. They flag the cases where their load-balancing rule does not behave well.

A few friendly suggestions

The MRC versus RoCE comparison is uneven, and that is the thing I would fix first. MRC’s headline results come from 42,000 to 75,000 GPU jobs. The head-to-head against RoCE runs on tiny testbeds of 16 and 64 GPUs, called Cluster C and Cluster D. The authors say plainly they have no large deployment for a direct comparison.

That gap matters for two reasons. The RoCE baseline is configured by the authors, as a single plane with DCQCN. The authors also call DCQCN very hard to tune. So a skeptic could read the comparison as a gentle strawman. Even a partial at-scale A or B test would make the case airtight.

There is no at-scale before-and-after. The paper shows MRC works well at scale. It shows RoCE struggles on small testbeds. It never shows the old approach at the same scale. So the size of the win is asserted, not measured. The strongest possible number would be the job-interruption rate before MRC and after MRC at a fixed scale.

Clustermapper’s own cost deserves a paragraph. Every node runs an agent. Every agent probes every directly connected link every millisecond. That is powerful. But the total probe traffic and control overhead at 100,000 nodes is not characterized. There is also a small tension to resolve. The paper says MRC makes denylists unnecessary. Yet it keeps Clustermapper for observability and some policy calls. One sentence reconciling the two would help.

Several key figures are single incidents. Figure 6 is one transceiver glitch that cost about 25 percent throughput. Figure 8 is one T1 reboot. These traces are vivid and useful. But a distribution across many failures would tell readers the typical case, not just the illustrative one. For example, a histogram of throughput dips and recovery times across all flaps in a job.

The remaining single point of failure is flagged, and it should be quantified. An 800 Gb/s NIC optic splits into 4 lanes of 200 Gb/s. If the NIC transceiver itself flaps, all ports go down at once. Then the queue pairs fail and the job cannot ride it out. The authors say this is rare and that a different optical design might avoid it. How often does it happen at scale, and what does it cost the job. That number would tell readers how close to fully resilient the system really is.

Generalizability is narrow by design. MRC needs SRv6 uN micro-segment support in the switches. It needs packet trimming in the switches. It needs specific NICs. It needs a multi-plane topology. So it suits a 100,000 GPU frontier cluster, where you can co-design the whole stack. A short note on what a smaller operator could adopt step by step would widen the paper’s reach.

The relation to UET could be sharper. MRC is described as a minimal extension to RoCE that borrows from Ultra Ethernet Transport. So a crisp line on what is new in MRC beyond UET would help. The novelty seems to be the SRv6 static-routing plus multi-plane co-design plus the production experience itself. Saying so directly would place the contribution well.

Bottom line

This is a bold, important, and candid paper. The static-routing-for-observability argument is a standout. The multi-vendor production deployment is rare and valuable. Turning off both dynamic routing and PFC is exactly the kind of principled risk-taking the field needs.

The main opportunity is evidence symmetry. MRC’s at-scale wins are weighed against small-testbed RoCE. They are also weighed against an unquantified before. So the size of the benefit is the one thing a reader has to take partly on faith. Close that gap and a very persuasive paper becomes an unarguable one.

What the paper does

Azure runs a lot of network policy in the hypervisor. Think private virtual networks, ACLs, load balancing, NAT, metering, and QoS. Running all of this on host CPUs costs cores. Those cores could be sold to customers instead.

AccelNet moves this work onto an FPGA SmartNIC. The NIC sits between the host NIC and the top-of-rack switch. The authors call this a bump-in-the-wire design.

The clever part is the Generic Flow Table, or GFT. The first packet of a flow goes to software. The VFP stack reads the policy, compiles it into one exact-match rule, and installs that rule on the FPGA. Every packet after that is handled by hardware. The host CPU does almost nothing for the rest of the flow.

The numbers are strong. VM to VM latency drops from about 50 microseconds to about 17 microseconds. A single TCP connection reaches about 32 Gbps. Host CPU use for the datapath falls under 1 percent. This has been running on more than 1 million hosts since 2015.

What works well

The production evidence is the best part. These are real numbers from a real fleet over several years. Most papers cannot offer this.

The first-packet-in-software idea is simple and it ages well. You keep the full flexibility of software for policy. You keep the speed of hardware for the bulk of the traffic. For example, a typical TCP flow has at least 7 to 10 packets including the handshake. So processing only the first packet in software still offloads the large majority.

The four-way hardware comparison is honest. The team weighs ASICs, multicore SoCs, FPGAs, and plain host CPUs. They say plainly when a choice rests on judgment rather than measurement.

The operational story is unusually complete. They cover FPGA reconfiguration without losing flows. They cover live migration. They cover transparent VF bonding through NetVSC so a revoked VF does not break the VM. They even cover a failsafe path for DPDK and RDMA apps.

A few friendly suggestions

The FPGA versus ASIC versus SoC case could use real measurements. Right now it leans on judgment. It also leans on the fact that Catapult FPGAs were already deployed. A head-to-head test on the same policy workload would settle it. As written, a reader cannot tell if FPGAs won on merit or on circumstance.

The cost story is only half told. The price of a CPU core is laid out carefully, around 0.10 to 0.11 dollars per hour. The FPGA side is missing. We do not see FPGA unit cost. We do not see the aggregate power of 17 to 19 watts across more than 1 million hosts weighed against the cores it saves. A rough two-sided model would make the don’t-burn-cores thesis much stronger.

The software path is the part I would most like to see stressed. The first packet of every flow goes to software. So the real limit is the new-flow rate, not the packet rate. The paper measures long-lived elephant flows. It does not measure the corner cases.

Here are concrete cases I would want tested. How many new connections per second can the VFP exception path handle before it saturates. What happens under a SYN flood, where an attacker opens many short connections on purpose. What happens when the flow count blows past the cache. The L1 cache holds only 2048 flows. The L2 cache lives in DRAM and holds about 1 million flows. The eviction policy and the miss cost are never described.

The AWS and GCP comparison is a single snapshot from November 2017. Competitors keep improving. So a point-in-time fastest-in-the-cloud claim is fragile. A note on the number of runs, the tuning, and the variance would help. A CDF would help more than P50, P99, and P99.9 bars alone.

The security framing is upbeat, and one tradeoff is left out. The Spectre and Meltdown resilience point is fair, because the datapath skips the host CPU. But SR-IOV hands a VF straight to an untrusted guest. That opens new surface, like guest-to-hardware DMA paths and side channels on a shared FPGA. One honest paragraph would round out the operational story.

The reliability claims would be better with numbers. The paper says failure rates stay low and that DRAM fails most often. Both are believable. The team has the fleet telemetry to give an annual failure rate, so it would be nice to see it.

Bottom line

This is a landmark paper and it earned its influence. Its strength is production evidence at real scale. Its soft spots are evaluation. The FPGA-versus-alternatives case is argued, not measured. The cost model is one-sided. The first-packet path is never stressed under high churn or attack. None of this takes away from a fine piece of engineering. These are the notes of an admiring colleague who would love to see the next version close the loop on the corner cases.

The idea

AEGIS is a framework that sits one layer above existing architecture tools. The name stands for Architecture with Evidence, Governance, Intent, and Safety. The core move is simple to state. AEGIS makes evidence and obligations into things you can model directly, the same way TOGAF and ArchiMate already let you model services and data. Once an obligation is a modelled object, a machine can ask whether the evidence for it is still valid. Once that check is automatic, it can run all the time.

The paper makes three contributions. It defines the new building blocks and how they attach to existing architecture models. It replaces the periodic review cycle with a continuous one. It maps the new building blocks to actual regulation, so a conformance report can be generated rather than written by hand.

The new building blocks

AEGIS introduces seven kinds of object. Each one names something that governance of an AI system needs but that current architecture languages cannot express.

• Evidence is a record that something is true, stamped with who issued it, when it was issued, and when it expires. Evidence can be valid, expired, superseded, or revoked.
• Obligation is a requirement placed on part of the system, such as “this service must keep an audit log”. An obligation says what must be true, what would prove it, and which regulation it comes from.
• Policy is a named, versioned bundle of obligations.
• Risk is a structured record of something that could go wrong, linked to the obligations meant to control it.
• RuntimeSignal is an event from a running system, such as an enforcement log or an audit result. Once it is linked to the right part of the model, it becomes a piece of evidence.
• Exception is a recorded, time-limited waiver from an obligation, with a reason and an owner.
• ArchitectureDecisionRecord is the record of a design decision, made queryable so the obligations and risks behind a choice stay attached to it.

These objects connect to the services, agents, and data assets you already model. AEGIS is built as an extension, so existing architecture models do not need to be redrawn.

The rule that does the work

One simple rule drives everything. A part of the system that is required to do something is in violation the moment there is no unexpired evidence that it is doing it. The framework checks this rule continuously rather than at review time.

A short example shows why this matters. A service is required to keep personal data from leaving the organisation unless a redaction step has run first. This requirement comes from the EU AI Act. A piece of evidence confirms the redaction step works, and that evidence is set to expire on a fixed date. The day after it expires, the rule fires on its own. The service is now in violation, a signal is raised, and the responsible owner is notified. No one had to remember to re-check.

The same idea gives a health score for any part of the system. The score is the share of a part’s mandatory obligations that currently have valid evidence behind them. When the score drops below a set level, the framework re-checks the affected area and refreshes whatever has gone stale.

What this lets you catch

The paper shows three things the framework can do that current tools cannot. None of these are measured on a live system. They are demonstrations of what the model can express.

It can express every control in the ISO/IEC 42001 standard as a checkable obligation, so a conformance report becomes a query rather than a manual exercise.

It can catch three ways governance silently drifts. Evidence can expire without anyone re-checking it. A rule enforced at the level of an individual agent can be changed while the organisation-wide policy is not, so the two no longer agree. A regulation can be updated so that an obligation now points at an old version of the law. Today’s architecture diagrams and code-level checks miss all three.

It can pull the rules enforced on individual AI agents up to the organisation level and show where two agents’ rules conflict with each other, a conflict that is invisible when each agent is governed on its own.

Limits

The three demonstrations show what the model can represent. They do not report results from a deployed system. The mapping to regulation covers the EU AI Act and ISO/IEC 42001. Other regimes, such as GDPR or medical-device rules, would need their own mappings. The reasoning engine behind the checks has not been tested at the scale of a large organisation. A full working implementation is the main piece of future work, and a real evaluation against existing tools depends on building it first.

Read the paper

The full paper, with the formal model and the complete regulatory mapping, is here: AEGIS (PDF and details).

References

Retrieval as routing

The paper reframes the problem. Choosing which store to read is a routing decision, and routing deserves to be a first-class part of the design. Stated plainly, for each request the agent should decide which store or stores to consult, and that decision should weigh two things against each other. One is accuracy, the quality of the eventual answer. The other is retrieval cost, measured in context tokens. A good router reads the store that holds the answer and skips the rest.

What the evidence shows

To measure the ceiling, the paper uses an oracle router. An oracle router is an idealized router that always picks the right store, used here to show what perfect routing would buy. The oracle achieves better downstream question-answering accuracy while spending far fewer context tokens than reading from all stores. Both numbers move in the good direction at once. Selective retrieval is not only cheaper, it gives better answers, because irrelevant context was hurting quality in the first place.

The oracle is a measurement tool rather than a deployable system, since it relies on knowing the right store in advance. The paper presents this as early-stage work and argues for learned routing, where the agent learns to make the store choice on its own, as the path to scalable multi-store systems.

Why it matters

As agents accumulate more kinds of memory, reading all of it per request scales badly in both cost and quality. Turning store selection into an explicit, cost-aware decision is what keeps a multi-store agent affordable and sharp as the number of stores grows.

Read the paper

Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agents (arXiv:2603.15658).

The gap in the theory

The existing analysis of DAM studies the limit where the network is infinitely large and the stored patterns are drawn at random. That setting is clean, and it tells you the capacity scales. It does not tell you what happens in a real network of a fixed, finite size. It gives no convergence rate, so you do not know how many steps recall takes. It does not say how the memory behaves when an adversary, rather than chance, chooses the corruption. This work fills those gaps with an algorithmic analysis, meaning it studies the actual retrieval procedure step by step under conditions you can check.

How fast recall settles

The first result is about speed. Recall in these networks runs by updating one unit at a time, which is called asynchronous updating. The paper assumes two checkable conditions. The stored patterns are well separated from each other, and the interference between them stays bounded even when the memory is heavily loaded. Under these, retrieval converges geometrically. Geometric convergence means the error shrinks by a constant fraction at every step. Once the state enters the basin of attraction, the region around a stored pattern that pulls inputs toward it, the number of steps needed to lock on grows only with the logarithm of the network size. That is fast. Doubling the network adds only a constant number of steps.

How much corruption it survives

The second result is about robustness against a deliberate attacker. The paper states an explicit margin condition. A margin here is the safety gap between the correct pattern and its nearest competitor. The condition turns that gap into a concrete number: how many corrupted bits the memory can absorb in a single pass and still recover the right pattern. This makes robustness a quantity you can compute for a given network rather than a hope.

Why it always settles

The third result explains why retrieval never gets stuck cycling. The paper shows that DAM retrieval is a potential game. A potential game is a system where every local update improves a single global score. Because each unit’s update can only raise that score, and the score cannot rise forever, the process must come to rest. It settles at a stable configuration, which in game terms is a pure Nash equilibrium and in memory terms is a recalled pattern. The capacity results survive this finite-size treatment, matching the classical scaling for random patterns and holding up to small logarithmic factors in the worst case.

Why it matters

Dense associative memory underlies the attention mechanism in modern networks, so its guarantees are not a curiosity. Turning capacity, speed, and robustness from statements about an infinite limit into checkable conditions on a real network is what lets you reason about these systems as engineered components.

Read the paper

Algorithmic Analysis of Dense Associative Memory, ICLR 2026 Workshop. OpenReview.

The words

The first draft is a terminology document. It defines a shared vocabulary for LLM serving performance, covering latency, throughput, resource use, and quality. It gives precise definitions for a large set of metrics. These include time to first token, the delay before the first output appears, and inter-token latency, the gap between successive output tokens. It separates the two phases of generation, prefill where the prompt is processed and decode where tokens are produced. It covers throughput, latency percentiles and jitter, multi-tenant concerns such as fairness and head-of-line blocking, optimizations such as prefix caching and speculative decoding, compound systems such as retrieval-augmented generation and agents, and quality signals such as policy violations and false refusals. It defines terms only. It prescribes no method and sets no pass-or-fail thresholds.

See the Terminology draft.

The method

The second draft is a methodology. It says how to run the tests, set up the parameters, take the measurements, and report the results. It defines three system boundaries to test at: the model engine, the application gateway, and the full compound system. It defines reference workloads, including uniform and skewed synthetic traffic, conversation, code completion, and long context. It defines ten tests, among them time to first token, output token throughput, the throughput-and-latency tradeoff, the distribution of inter-token latency, concurrent request capacity, scheduling fairness, prefix cache effectiveness, behavior under memory pressure, long-context scaling, and the overhead added by guardrails. The goal is reproducible and fair comparison that accounts for streaming output, dynamic batching, and the cost of safety systems.

See the Methodology draft.

The measurement boundaries

The third draft defines profiles. A profile binds the terms and the procedures to a concrete role in the system and a concrete workload pattern, and it declares exactly where the measurement starts and stops. This matters because a metric like time to first token changes depending on whether you include the gateway, the firewall, and the network. Without a declared boundary the numbers are not comparable. The draft defines infrastructure profiles for the model engine, the AI gateway, the AI firewall, and the compound system, along with workload profiles for a chatbot and a compound workflow. It specifies profiles only, not new metrics or thresholds.

See the Profiles draft.

The workloads

The fourth draft defines the workloads themselves. It gives 25 standard workload profiles organized into six groups, so that everyone tests against the same representative tasks. The groups run from non-generative work such as embeddings, reranking, and scoring, through minimal-output tasks such as classification and function calling, interactive streaming such as chat and code completion, prefill-heavy tasks such as summarization and document question answering, decode-heavy tasks such as content generation and reasoning, and multi-step agentic workflows. Each profile is described along five dimensions: the ratio of input to output, the constraints on the output, the latency sensitivity, the concurrency pattern, and how much of the prompt prefix is shared across requests.

See the Serving Workload Profiles draft.

Why it matters

Benchmarks only mean something when everyone measures the same thing the same way. These four drafts supply the missing pieces in order. The vocabulary removes ambiguity in the terms. The methodology makes a run reproducible. The profiles declare what was and was not included in a measurement. The workloads make sure systems are compared on representative tasks. Together they make LLM serving numbers something you can actually trust when you compare them.

Five dimensions

Every workload is described along five dimensions. Together they capture what makes a task cheap or expensive to serve.

• Input to output ratio. Whether the task is mostly prompt, mostly generation, or balanced.
• Output constraints. Whether the output is non-token, minimal, structured, semi-structured, or free-form.
• Latency sensitivity. Whether the task is interactive, near-real-time, or throughput-oriented.
• Concurrency pattern. Whether requests are single-stream, bursty, sustained and high, or chained in sequence.
• Prefix sharing. How much of the prompt prefix is shared across requests, from none, through a shared system prompt, up to a shared document or schema.

Twenty-five profiles in six groups

The draft defines 25 standard workloads, sorted into six groups by what they demand of the system.

• Group A, non-generative. Embeddings, cross-encoder reranking, and logprob scoring.
• Group B, minimal output. Classification, short question answering, and function calling.
• Group C, interactive streaming. Chat, code completion, and roleplay.
• Group D, prefill-heavy. Summarization, document question answering, and extraction.
• Group E, decode-heavy. Content generation, reasoning, and translation.
• Group F, multi-step. Agentic workflows and planning.

Each profile maps to the metrics and test procedures from the companion methodology and profiles drafts, so a workload, the way to measure it, and the boundary it is measured at all line up.

Why it matters

Without standard workloads, every benchmark picks its own favourable task and the results stop being comparable. A fixed, representative set means a serving system is judged on the same range of work that production actually throws at it.

Read the draft

Benchmarking Workload Profiles for Large Language Model Serving.

What a profile is

A profile binds the vocabulary and the procedures to a concrete role in the system and a concrete workload pattern. Its job is to declare the boundary. A profile says exactly where a measurement starts and stops, so two results taken under the same profile are measuring the same thing.

The profiles it defines

The draft defines infrastructure profiles for the distinct roles in a serving stack.

• Model engine. Raw inference: prefill, decode, and management of the key-value cache.
• AI gateway. The overhead of routing, authentication, and caching.
• AI firewall. The latency and accuracy of security inspection, including detection rates and false positive rates.
• Compound system. End-to-end completion for orchestrated multi-step workflows such as retrieval-augmented generation and agents.

It also defines workload profiles for how the system is driven, including a chatbot pattern, which is multi-turn and human-paced, and a compound workflow pattern, which is multi-step and machine-paced.

The draft specifies profiles only. It introduces no new metrics and no thresholds. It is the layer that makes the terminology and the methodology comparable across real deployments, where a request passes through several components before it reaches the model.

Why it matters

Production systems are built from a gateway, a firewall, an engine, and more. Most published benchmarks never say which of these the numbers include. Profiles remove that ambiguity by making the boundary an explicit, declared part of every result.

Read the draft

Performance Benchmarking Profiles for Large Language Model Serving Systems.

What it defines

The draft gives precise definitions for a large set of metrics, on the order of seventy, grouped into about a dozen categories. The categories cover the parts of serving that people measure and disagree about.

• Request and response timing. Time to first token, the delay before the first output appears. Inter-token latency, the gap between successive output tokens. Normalized latency, which adjusts for output length.
• Phase-specific timing. Prefill latency, the cost of processing the prompt, and decode latency, the cost of generating tokens. These two phases behave differently, so the draft keeps them separate.
• Throughput. Token rates and request rates, defined so it is clear what is being counted.
• Latency distribution. Percentiles and jitter, since an average hides the tail that users actually feel.
• Multi-tenancy and scheduling. Fairness across tenants and head-of-line blocking, where one large request stalls the queue behind it.
• Optimizations. Prefix caching, which reuses work across requests that share a prompt, and speculative decoding, which guesses tokens ahead and verifies them.
• Compound systems. Retrieval-augmented generation and agentic workflows, where a request triggers several model and tool calls.
• Quality enforcement. Policy violations and false refusals, so safety behavior is measurable rather than anecdotal.

What it leaves out, on purpose

The draft defines terms only. It prescribes no method for measuring them and sets no pass-or-fail thresholds. That restraint is the point. A vocabulary that everyone agrees on has to be separable from any one way of testing. The method comes in the next draft, and the thresholds are left to whoever runs a benchmark.

Why it matters

Comparable benchmarks start with agreed words. Once time to first token and inter-token latency mean one thing across papers and products, the rest of the benchmarking effort has a foundation to stand on.

Read the draft

Benchmarking Terminology for Large Language Model Serving.

Where to measure

The draft defines three system boundaries, because a serving stack has layers and a number means little without saying which layer it covers.

• The model engine, which is raw inference.
• The application gateway, which adds routing, authentication, and caching.
• The compound system, which is the full end-to-end path including retrieval and tool use.

A measurement always states which of these boundaries it was taken at.

What traffic to send

It defines reference workloads so that everyone drives the system with comparable traffic. These include uniform synthetic traffic, skewed synthetic traffic, a conversation pattern, code completion, and long context. Each stresses the system in a different way, from steady load to bursty load to very long prompts.

The ten tests

The draft lays out ten benchmarking tests. Together they cover speed, capacity, fairness, and the cost of the features that production systems actually run.

1. Time to first token.
2. Output token throughput.
3. The tradeoff between throughput and latency.
4. The distribution of inter-token latency, not just its average.
5. Concurrent request capacity.
6. Scheduling fairness across requests.
7. Prefix cache effectiveness.
8. Behavior under memory pressure.
9. Long-context scaling.
10. The overhead added by guardrails.

What it accounts for

The methodology is built for how LLM serving really behaves. It handles streaming output, where tokens arrive over time rather than all at once. It handles dynamic batching, where the system groups requests on the fly. It handles context-dependent performance, where a long prompt changes the numbers. It also measures the cost of safety systems rather than ignoring them. The goal throughout is a run that someone else can repeat and get the same answer.

Why it matters

A benchmark that cannot be reproduced is a marketing number. By fixing the boundaries, the traffic, and the tests, this draft makes a serving result something a second team can check.

Read the draft

Benchmarking Methodology for Large Language Model Serving.

Two kinds of field need two kinds of protection

The key observation is that not all data is equally risky, and the right protection depends on how often a value appears.

Rare fields are the dangerous ones. A value that shows up only a few times can identify its source, so it needs strong hiding. AlignDP applies PAC indistinguishability to rare fields. The practical meaning is that an observer cannot reliably tell whether a given rare value was present, so the rare information stays concealed.

Common fields are different. You often want to keep useful aggregate statistics about them, such as how frequent each value is. AlignDP protects these with RAPPOR, a well-known method for local differential privacy. RAPPOR randomizes each individual report so no single response is revealing, while still allowing accurate frequency estimates once many reports are combined. The frequent values stay usable in aggregate without exposing any one contributor.

A global aggregator sits over both tiers. It manages composition, meaning the way many small privacy releases add up, and it tracks the overall budget so the total exposure stays bounded.

What the paper shows

The work establishes theoretical results for both tiers. It extends the PAC guarantee to the global aggregation step and derives error bounds for the RAPPOR estimates. Simulations match the design intent. Rare categories remain hidden, while frequent categories can be recovered with small error. The paper is explicit about the tradeoff between privacy and utility, since that tension is the heart of any differential privacy scheme.

Why it matters

Defending a model after it has been copied is a weak position. AlignDP moves the control to the point where data enters, and it tailors the strength of protection to the risk each field carries. Strong hiding goes where identification is possible. Lighter, aggregate-preserving protection goes where only statistics matter.

Read the paper

AlignDP: Hybrid Differential Privacy with Rarity-Aware Protection for LLMs (arXiv:2512.17251).

Spending a privacy budget per query

The core mechanism is an adaptive budget. Differential privacy is a way of adding calibrated random noise so that an output reveals very little about any single underlying record. The strength of that protection is a budget, and spending more budget buys more accuracy at the cost of more exposure. Most systems fix one budget for everything. AVEC sets the budget per query, based on three things: how sensitive the query is, how confident the local model is on its own, and how much budget the user has already spent in the past.

To keep the spending honest, AVEC tracks it with an odometer. An odometer is a running tally of privacy budget consumed across many queries, so the system always knows how much protection remains. The accounting uses Rényi differential privacy, a version of the theory that composes cleanly when you add up many small releases.

What it proves it can and cannot do

The paper does the unglamorous work of marking the boundaries. It establishes utility ceilings, which are limits on how good the answers can be at a given level of privacy. It establishes delegation-leakage bounds, which cap how much the handoff to the remote model can reveal. It also proves impossibility results. Two tempting shortcuts do not work on their own. Deterministic gating, where a fixed rule decides what to send, cannot provide the guarantee. Hash-only certification, where a hash is used to prove integrity, is not enough by itself.

Why it matters

Local models are sold on privacy, and the delegation step quietly undermines that promise. AVEC takes the step seriously. It treats each outward query as a metered privacy event, verifies the transformation on the device, and is honest about the ceilings and the dead ends. That gives later empirical work a clear target.

Read the paper

AVEC: Bootstrapping Privacy for Local LLMs (arXiv:2509.10561).

Canonicalization

The tool is called Opal, and it does canonicalization. A canonical form is a standard way of writing something so that two things that are really equal end up looking identical. Given two preference objectives, Opal returns one of two answers. Either it produces a canonical form that shows the two are algebraically the same, or it produces a concrete witness that they differ. A witness here is a specific case where the two objectives disagree, so the difference is demonstrated rather than asserted.

What it finds

Run Opal across the popular methods and a clear picture emerges. Many widely used objectives optimize the same underlying thing. Their apparent differences are reparameterizations, meaning the same objective dressed in different variables. Some methods, by contrast, are provably distinct, and Opal isolates a small set of structural mechanisms that produce real differences rather than cosmetic ones.

One finding is worth calling out because it is easy to miss. Batch normalization can cause the same response pair to receive different gradients depending on which other examples share its batch. The training signal for a pair stops being a property of the pair alone. It becomes a property of the batch it landed in. That is a genuine difference in objective, and it comes from a detail most people treat as bookkeeping.

Why it matters

A field that keeps proposing objectives needs a way to tell novelty from notation. Opal turns that into a decidable check. It tells you when a new method is a real departure and when it is an old method in new clothes. That saves effort, and it points attention at the handful of structural choices that actually change what the model learns.

Read the paper

When Are Two RLHF Objectives the Same? (arXiv:2509.11298).

A compass that points wrong in some places

The setup is misspecified feedback. Human feedback is systematically wrong on certain kinds of input. A useful picture is a compass that points true almost everywhere but points the wrong way in a few specific regions. The problem is not random error, which averages out. It is consistent error in particular places.

Two numbers describe how bad it is. One is the fraction of contexts where the feedback is biased. The other is the strength of that bias. The trouble lives in the corner of the input space where both are nonzero.

The hard result

The paper proves a lower bound. Suppose two candidate reward functions agree everywhere except on the biased contexts. To tell them apart from feedback alone, any learning algorithm needs a number of samples that grows exponentially in the prevalence of the bias and the square of its strength. Rare edge cases with subtly wrong feedback create a learning problem that is exponentially hard. You can pour in data and still not know which objective you are actually optimizing.

This is the sense in which the gap always wins. The gap is the distance between what you optimize, a proxy built from human feedback, and what you actually want, the true objective. When the problematic contexts are rare and the feedback there is quietly biased, no amount of ordinary data closes that gap.

The way out

There is one escape, and it is specific. Suppose you have a calibration oracle, a way to identify where the feedback is unreliable. Then you can spend your limited questions on exactly those contexts. The paper shows that this collapses the cost from exponential to polynomial. You no longer need to find the needle by luck, because you have been told which haystack it is in.

The lesson is practical. Throwing more feedback at an aligned system does not help where the feedback itself is broken. Finding and targeting the broken regions does.

What is settled and what is open

The results are theoretical. The paper proves the lower bound and the oracle speedup. It invites collaboration on large-scale experiments and on building the benchmarks that would test these effects in practice.

Read the paper

Murphy’s Laws of AI Alignment: Why the Gap Always Wins (arXiv:2509.05381).

The gap in ordinary alerting

Standard monitoring detects individual device or link failures. Each alert is local. It says nothing about the aggregate risk that several failures together push a datacenter or a region past its capacity limit. Operators are left to piece that picture together by hand, often while the alerts are still arriving. The risk that matters is the cascading one, and it lives above the level of any single alert.

What ANSC measures

ANSC produces a color-coded health score for each datacenter and region. The score ranks urgency by the probability of an imminent capacity violation rather than by the impact of what has already happened. Two ingredients go into it. The first is the current residual capacity, meaning how much headroom is left right now. The second is the probability of additional failures on top of the current state. The score combines these and normalizes them at the datacenter level and the regional level, so a green datacenter and a red one mean the same thing wherever you look.

What it changes for operators

The paper reports ANSC running across more than 400 datacenters and 60 regions. The effect is to cut noise and point attention at the places most likely to fail soon. Instead of a flat stream of equal-looking alerts, an operator sees a ranked view of where capacity is genuinely at risk. Remediation effort goes to the highest risk first, which is where it does the most good.

Why it matters

Reliability at this scale is a prioritization problem. There are always more alerts than people. A score that orders work by the probability of the next failure, rather than by the noise of the last one, is what lets a small team hold a large fleet.

Read the paper

ANSC: Probabilistic Capacity Health Scoring for Datacenter-Scale Reliability (arXiv:2508.16119).

The two quantities

NPO names two things and turns each into a number.

The first is alignment loss. This is how far the model’s behavior sits from what was intended. Driving it down is the usual goal.

The second is meta-alignment, and it is the idea that gives the paper its name. Meta-alignment asks whether your monitoring is itself trustworthy. A system that reports “everything looks aligned” is only useful if that report tracks reality. NPO calls this monitoring fidelity. The paper shows that meta-alignment reduces to ordinary alignment once you fix a threshold on the monitor. In plain terms, checking the checker is the same kind of problem as the original check, so the same machinery handles both.

The feedback that drives it

The loop runs on structured human feedback. This is more than a thumbs up. It includes likes, explicit overrides where a person corrects the system, and abstentions where a person declines to answer. Each kind of signal carries different information, and NPO ingests all of them. Around this sit the operational pieces a real deployment needs: scoring scenarios, tuning thresholds, and validating policies before they take effect.

Why the loop is stable

A feedback loop is only useful if it settles rather than drifts. NPO proves that under noisy feedback both quantities converge, and that they do so together. The alignment loss and the monitoring fidelity both approach their targets as more feedback arrives. The paper backs the theory with results from a large deployment, along with simulation artifacts and ablation studies that isolate which parts of the design carry the weight.

Why it matters

The shift here is from a single alignment event to a controlled process. Once alignment and the honesty of its monitoring are both numbers that provably go down under feedback, you can run alignment the way you run any other reliability metric. You watch it, you set targets, and you route effort to where the number is worst.

Read the paper

NPO: Learning Alignment and Meta-Alignment through Structured Human Feedback (arXiv:2507.21131).