Host tuning

Why the same recipe runs at different speeds on different hosts

During decode, the runtime launches a long list of small GPU kernels for every token. Our shape profile of the Qwen3.8 FP8 TP2 MTP1 stack counted 2,080 GEMMs and 1,056 all-reduces per rank across eight decode iterations, so roughly 130 two-card all-reduces and 260 GEMMs per step, plus norms, attention, and sampling. R61 shape profile

Each of those launches costs CPU time before the GPU sees it. If the GPU finishes a kernel before the CPU has queued the next one, the GPU idles. The runtime also waits on the host at points (synchronize, all-reduce, token readback). That is why two boxes with the same cards, image, model, and weights can disagree on tok/s.

The lab measured this directly with one probe run on both of its hosts inside the same container image. Lab-measured

Sources: four-B70 host replay note and the publishing host's probe data file (3.13 µs launch, 25.98 µs launch+sync, 41.47 µs rms_norm). The replay host's launch figures were read twice, 5.18 and 6.2 µs, and its launch+sync twice, 30.2 and 33.0 µs.

The CPU alone is about 28 percent slower single-thread. The GPU submission path is 1.65x slower per launch, launch-heavy ops are 3.2x slower, and the two-card collective is about 4x slower. With about 130 exposed host round trips per token, that adds up to the whole decode gap:

All six prompt classes on the replay host sat within 27.5–30.6 tok/s on MTP1, so the factor is uniform and host-wide, not a per-prompt effect. Replay note

Graph on vs graph off

vLLM can record the kernel launch sequence for a decode step once, per captured batch size, and then replay the whole recording as a single submission. On CUDA this is a CUDA graph; on Intel XPU the same mechanism is exposed as an XPU graph (VLLM_XPU_ENABLE_XPU_GRAPH=1). The knobs live in the compilation config:

• cudagraph_mode: PIECEWISE captures the parts of the step between non-capturable ops; FULL_DECODE_ONLY captures the whole decode step and leaves prefill eager.
• cudagraph_capture_sizes and max_cudagraph_capture_size: which batch sizes get a recording. Anything larger runs eagerly.

Capture removes the per-launch CPU cost that the section above describes. So the question is simply: how much of your decode step is that cost?

On a fast host, graph capture gained nothing

R58 turned on a size-one XPU graph on the publishing host's deterministic MTP1 profile. Outputs stayed 12/12 exact against the MTP0 oracle. Decode went from 51.81 to 51.23 tok/s, a 1.1 percent loss, which missed the preregistered 99 percent non-inferiority floor by 0.06 tok/s. Graph-off stayed selected. Lab-measured R58 note · CURRENT.md

On a slow host, it gave 90–95 percent back

The MTP1 row with only size [1] matters. With one captured size, the single-row target step is recorded but the two-row verification step still runs eagerly, so the draft's launch cost cancels its token gain. You must capture every batch shape the decode step actually uses. Across 2K–32K real-content depth, the graph-on replay host held 90–94 percent of the published MTP1 curve and 93–95 percent of the MTP0 curve, with every MTP1 array matching its own MTP0 oracle. Replay note

What graph capture costs you

• Fixed capture sizes. Only recorded batch sizes replay. Under concurrency, a batch that does not match a captured size is padded to the next one or bypasses the graph and runs eagerly.
• Startup time and memory. Every captured size is a recording that takes time to build at startup and holds its own memory. The lab's four-B70 Flash-Next runs report full-graph capture as a separate startup phase (52 seconds on all four ranks in one attempt). A52 note
• Identity under concurrency. At one request the outputs were byte-identical to graph-off in every lab check: 12/12 strict digests on the replay host, and 64/64 c1 oracles for both profiles on the publishing host (R163). Under load the picture is mixed. Graph-on MTP1 on the replay host was exact through c16, then 29/32 and 59/64; graph-on MTP0 there was exact through c8, then 15/16 at c16. On the publishing host, R163 graph-on MTP0 was exact through c32 (63/64 at c64) and graph-on MTP1 was 15/16 at c16 and 57/64 at c64, while the published graph-off MTP0 profile (R156) is 64/64 at c64. Lab-measured Replay ladders · R163 result · CURRENT.md
• Less visibility. Inside a replay, individual kernels are not separately submitted, so per-kernel profiling and tracing lose resolution. Diagnose graph-off; then re-enable capture.
• Support status. vLLM warns that XPU graph is experimental and officially supports single-GPU execution. The lab's TP2 arms passed their gates, but you are past the documented envelope.

How to profile your own host in five minutes

The lab's probe is a short pure-Python script that runs inside the recipe's container image on one card. It reports three numbers that matter and two that are specific to the image. Probe script

# inside the R139 (or later) image, one B70 visible
python3 qwen38-fp8-host-submission-latency-probe.py probe.json
cat probe.json

How to read it:

• launch_us_async: microseconds per back-to-back tiny kernel launch (2,000 one-element adds, no sync). This is your pure submission cost.
• launch_plus_sync_us: launch then synchronize, 500 times. This is the cost of every host-waited point in the step.
• ir_rms_norm_m2_us: a real launch-heavy op at decode shape. This amplifies the first number.
• w8a16_m2_qkv_us: a decode-shaped W8A16 GEMM. On the publishing host this returned a scale-layout error rather than a time; treat it as optional.

Reference points from the lab: 3.1 µs / 26 µs / 41 µs on the host that ran graph-off at full published speed; 5.2 µs / 30 µs / 134 µs on the host that needed graph-on to get within 5 percent. If your launch number is at or above about 5 µs, or your rms_norm number is several times 41 µs, expect graph-off to underperform and try graph capture first. Between those points we have no measurement; probe, then compare decode both ways. For a two-card recipe, also time the exact-shape all-reduce the model uses: the lab saw 13 µs versus 48–51 µs for a [1,5120] BF16 reduce. Replay note

Quick host checks

lscpu | grep -E 'Model name|^CPU\(s\)|MHz|NUMA'
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/module/pcie_aspm/parameters/policy
xpu-smi discovery
scripts/report-pcie-topology.sh            # live and max link state for every hop
sudo lspci -vv -s 21:00.0 | grep -E 'LnkSta|LnkCap'   # the card's upstream switch port, not the GPU
sudo dmesg | grep -i -E 'aer|corrected|xe '

Host fixes the lab actually needed

Each row is something the lab did on its own B70 hosts, with the symptom, the fix, and what it measurably changed. Every claim links to the note. Lab-measured

Decision table