Each decode step reads the model's weights from memory once, no matter how many people are waiting on an answer — the arithmetic for every user shares that single read. So adding users barely adds memory traffic; it mostly adds math, and the card has math to spare. Per-user speed falls, but the combined rate keeps climbing until the compute units saturate.
The measured sweep
Persistent raw-engine batching, 128-token prompts, 1,024-token answers, temperature 0, two repeats per point. Combined counts every token after each request's first; per user is the combined rate divided by the number of users. Methodology & data · raw data.
| Users at once | Combined (tok/s) | Per user (tok/s) | Notes |
|---|---|---|---|
| 1 | 87.64 | 87.64 | graph-captured single-user path |
| 2 | 77.19 | 38.59 | runs off-graph; dips below one user |
| 4 | 149.48 | 37.37 | off-graph |
| 8 | 276.77 | 34.60 | off-graph |
| 16 | 488.65 | 30.54 | off-graph |
| 32 | 823.15 | 25.72 | graph-captured |
| 64 | 1,039.41 | 16.24 | graph-captured · determinism open |
A newer treatment measured separately (never joined to the curve above) reached 90.91 tok/s for one user and 1,052.87 combined at 64. The dip from 1 to 2 users is a property of this tuned stack — its XPU Graphs are captured only at 1, 32, and 64 users, so in-between sizes pay eager-mode overhead.
Loading the projected curve from mlbottleneck.com…
Dashed line: what the ML Bottleneck physics engine projects for stock software on this exact setup. It is a projection, not an independent check: the engine's batched-compute constant for Intel vLLM is calibrated partly against this sweep's 32- and 64-user anchors. Solid line: the lab's measured tuned stack, whose win over stock is biggest at one user; tuning of the batched path continues.
How to read it
If you are one person, the number that matters is the single-user rate (87.6–90.9 tok/s here — the same class as the promoted single-user recipes on the benchmark board). If a household, class, or office shares the box — or one person fans out agent jobs — the combined rate is the budget: at 64 streams each one still gets ~16 tok/s, which is faster than most people read, while the card as a whole does the work of a small serving cluster.
Three honest caveats. 1) This is experimental evidence, not a promoted recipe: speeds repeat cleanly, but fixed-seed outputs at 64 users are not yet bit-identical across repeats (27–29 of 64 sequences matched). 2) The 2–16-user dip is real for this stack — the XPU Graph captures exist only at batch 1/32/64; a deployment expecting 8 concurrent users would want a capture at 8. 3) Numbers are for 128-token prompts and 1,024-token answers; long prompts shift work toward prefill and lower every row.
Want the same curve for a different model, card count, or user count? Open this setup in ML Bottleneck and change anything — its "throughput vs concurrent users" chart is the projected version of this page, and this sweep is part of the measured evidence that keeps it honest.
Two cards, many people: Qwen3.8-27B INT4 with speculation
The Qwen3.8-27B AutoRound INT4 lane (two B70s, vLLM XPU, lab kernel stack, XPU graph capture) was swept the same way over HTTP on 2026-09-06, with every response compared token-for-token against its own single-user oracle, at four speculation settings. Warm pass; the count in brackets is how many of the concurrent answers were byte-identical to the oracle. Recipe · raw data (R284-R290).
| Users at once | No speculation | MTP depth 1 | MTP depth 2 | MTP depth 4 |
|---|---|---|---|---|
| 1 | 50.0 | - | - | 112.4 |
| 2 | 94.6 (2/2) | 147.3 (2/2) | - | 191.0 (2/2) |
| 4 | 178.5 (4/4) | 268.6 (4/4) | - | 294.7 (4/4) |
| 8 | 327.0 (8/8) | 456.0 (7/8) | - | 422.6 (7/8) |
| 16 | 533.8 (16/16) | 711.0 (16/16) | 591.0 (16/16) | 574.3 (16/16) |
| 32 | 815.0 (32/32) | 854.4 (32/32) | 723.4 (29/32) | 641.3 (32/32) |
| 64 | 991.4 (64/64) | 842.0 (61/64) | 815.2 (59/64) | 591.4 (58/64) |
| 128 | 1,085.3 (128/128) | 894.8 (121/128) | 815.0 (121/128) | 584.7 (117/128) |
Speculative decoding wins while the verify batch is small: depth 4 for one to four users, depth 1 from about eight to thirty-two. Past that every depth plateaus (the verify batch exceeds the ~32-row tier of the INT4 matmul) and plain decoding keeps climbing to ~1,000 combined tok/s at 64 users and ~1,085 at 128, still byte-identical to the single-user answers. The few non-identical answers above 32 users are always the same prompts diverging at the same token, a near-tie in the next-token scores that the wider matmul tier rounds the other way; they are valid answers, but the recipe withholds those points from its identity-qualified profiles. One card alone peaks at 514 combined tok/s at 32 users without speculation and 465 at 16 with depth 1.
Why answers change when the server gets busy, and what fixes it
Every table above reports how many of the concurrent answers came back byte-identical to the answer the same server gives one user at a time. That count is usually perfect at low concurrency and slips at high concurrency: a handful of prompts come back different. The answers are not wrong, they are a second valid continuation. Measurements on four lanes in September 2026 pinned down why, and one configuration removes it entirely.
It is the matmul, not the model. When a kernel sums a row differently depending on how many rows are in the batch, any token whose top two candidates are exactly tied can fall the other way. Kernels that keep one summation order regardless of batch size do not have the problem. On these cards that is the INT4 W4A16 path.
Two models show it failing in two different ways on the same stack, same launcher, same 12 prompts, and both are clean once the weights are INT4:
| Model and check | FP8 weights | INT4 W4A16 weights |
|---|---|---|
| Qwen3.5-4B, two fresh servers, one user | 11/12, 9/12, 11/12 over three tries | 12/12 |
| Qwen3.5-9B, 16 users at once | 15/16 | 16/16, both passes |
| Qwen3.5-9B, 64 users at once | 59/64 | 64/64, both passes |
| Qwen3.5-9B, one user, tokens per second | 98.3 | 113.6 |
| Qwen3.5-9B on llama.cpp (Q8), two fresh servers | 8/12 agree, 47.2 tok/s | |
The 4B case is the clearest evidence that this is arithmetic and not load: two servers started from the same files, with one request each and nothing concurrent, still disagreed on three of twelve prompts. Each of those prompts had exactly two possible answers and each server picked one, never a third. That is what an exact tie looks like.
A second, independent check. Concurrency is not the only thing that changes the batch shape. These models can draft several tokens ahead and then verify them in one step, and a deeper draft means more rows in that step, without any extra users. On the FP8 weights, drafting four ahead instead of three broke a third of the answers; five and six broke the same third. On the INT4 weights, drafting three, four, five or six ahead all returned byte-identical answers. Same models, same prompts, same servers. So the thing that decides whether answers are reproducible is not how many people are connected and not whether drafting is switched on: it is whether the matmul sums the same way when the batch changes shape.
It is not a quirk of one engine either. The same 9B model served by llama.cpp from a Q8 file, a different runtime with a different quantization and different kernels, agreed on only 8 of 12 prompts between two fresh servers. A 26B model on that same runtime is worse still: two fresh servers, greedy decoding, identical prompts, and not one of twelve answers came back the same - some diverging within the first thirty characters. We checked whether drafting was to blame by turning it off entirely, and the answer was still zero of twelve, so it is the model's own decoding, not the speed trick layered on top. Reproducible output is not something a stack has by default. On these cards it is something the INT4 path provides and the alternatives do not.
One card, though. Everything above was measured on a single GPU. Add a second card and the guarantee weakens: the same INT4 build on two cards matched 63 of 64 answers at sixty-four users, in both passes, and the 4B model held only to thirty-two users. The kernel has not changed and is still the same row-count invariant matmul, so whatever varies is something the second card adds. We tested the obvious candidate - the step that combines the two cards' partial results - by forcing it to run the same way it does for a single request, and then tested it together with the one other batch-size-dependent step we found. Neither helped at all. The reason turned up in the same measurement: send the identical question sixty-four times at once, and the copies do not all come back the same. They share a batch, a size, and every input, so how many requests are present cannot be what separates them - which is why making the arithmetic independent of that number changed nothing. Something about where a request sits in the batch matters, and we do not yet know what. What is settled is the practical part. If you need the byte-exact guarantee, one card gives it and two do not.
The obvious knob does not help. Padding the decode batch to fixed row counts, so the matmul always sees the same shapes, is inert below its threshold and costs 13% of throughput at 64 users above it, while changing nothing about how many answers match. So the flips that remain when speculative decoding is switched on come from somewhere other than the matmul's row count, and that thread is still open.
What to do with this. If you need identical answers no matter how busy the server is, serve the INT4 build and leave speculative decoding off: that combination was byte-exact at every level from one to sixty-four users, twice in a row, at about 1,270 combined tokens per second on a single card. If you want the fastest answer for one person, turn speculation on and accept that a rare tied token may fall either way once many people share the server. The recipes state which of their numbers are identity-qualified and withhold the ones that are not.