Qwen3.5 9B FP8-dynamic with its own MTP head on one Intel Arc Pro B70 — 98.1 tok/s measured

What we measured Lab-measured

98.1tok/s MTP depth 3 with the draft-only INT4 lm_head, one card, strict completions suite (center of two fresh servers, 98.251 / 98.027)

Center of two fresh-server class-balanced medians (98.251 / 98.027) over the strict fixed 12-prompt six-class suite through the completions API, 512-token cap, separate empty compile caches, 12/12 complete token arrays exact versus a same-configuration MTP0 oracle (50.165 / 50.173), canaries on every server, cache zero; one B70, R276 image, qwen3_5_mtp depth 3, draft-only INT4 lm_head (VLLM_XPU_DRAFT_LM_HEAD_INT4=1), full decode-only XPU graph capture. proof file

MTP = multi-token prediction, a small draft the main model verifies · oracle = the fixed prompt set whose exact outputs are checked

Open the full guide Submit an improvement

Model: Qwen3.5 9B FP8-dynamic (compressed-tensors, per-channel FP8 weights, dynamic activations) with the publisher's MTP head
Publisher: RedHatAI (FP8-dynamic of Alibaba / Qwen)
Checkpoint: RedHatAI/Qwen3.5-9B-FP8-dynamic
Compression: FP8-dynamic (compressed-tensors, per-channel FP8 weights, dynamic activations)
Software: vLLM XPU
Cards: 1× Intel Arc Pro B70 32 GiB
Model weight bytes: 14.0 GB
Operating systems: Linux
Delivery: Docker / container
Good for: general coding
Published: 2026-09-07
Clean-host replay: not yet

Still missing before this becomes an install guide

clean-host replay

Measured performance profiles Lab-measured

Qwen3.5-9B FP8 one card, MTP depth 3 with the draft INT4 head: decode over active context (2K-32K) Lab-measured

Campaign c7 (2026-09-07): one B70, R276 image, one-slot vLLM HTTP completions at exactly 2K, 4K, 8K, 16K, 24K and 32K active context of unrepeated real content (technical prose, Python, structured documents; three requests per depth, median shown), 128 output tokens, max-model-len 33024, max-num-batched-tokens 4096, cache zero, canaries before and after; every answer matched the same-configuration MTP0 oracle (18/18). No value is interpolated. evidence

Qwen3.5-9B FP8 one card, no speculation: decode over active context (2K-32K) Lab-measured

Campaign c7 MTP0 arm (the oracle for the depth-3 profile), same workload and shape. evidence

Qwen3.5-9B FP8 two cards (TP2), MTP depth 3 with the draft INT4 head: decode over active context (2K-32K) Lab-measured

Campaign c8 (2026-09-07): two B70s (TP2), otherwise as the one-card context profile; 18/18 answers matched the two-card MTP0 oracle. evidence

Many people at once Lab-measured

Qwen3.5-9B FP8 one card, MTP depth 3 with the draft INT4 head and XPU graph capture, identity-qualified aggregate decode vs concurrent users (c1-c8; warm pass) Lab-measured

Campaign c2 (2026-09-07): one B70, R276 image, strict launcher env, FULL_DECODE_ONLY capture sizes 1-64, qwen3_5_mtp depth 3, VLLM_XPU_DRAFT_LM_HEAD_INT4=1, max-model-len 256, max-num-seqs 64, max-num-batched-tokens 512, 128 returned raw token IDs per response on the small-context suite, two passes on one server, warm pass shown. c1-c8 output-identity-qualified in both passes. Measured but withheld: c16 738.4 (15/16), c32 883.6 (30/32), c64 831.2 (57/64). evidence

Qwen3.5-9B FP8 one card, no speculation, XPU graph capture, identity-qualified aggregate decode vs concurrent users (c1-c8; warm pass) Lab-measured

Campaign c2 MTP0 ladder (as above without speculation). c1-c8 output-identity-qualified in both passes. Measured but withheld: c16 634.7 (15/16), c32 1055.2 (31/32), c64 1253.8 (59/64). evidence

Qwen3.5-9B FP8 two cards (TP2), MTP depth 3 with the draft INT4 head and XPU graph capture, identity-qualified aggregate decode vs concurrent users (c1-c8; warm pass) Lab-measured

Campaign c5 (2026-09-07): two B70s (TENSOR_PARALLEL_SIZE=2), otherwise as the one-card depth-3 profile. c1-c8 output-identity-qualified in both passes. Measured but withheld: c16 1156.7 (15/16; first pass 16/16 at 969.3), c32 1422.2 (29/32), c64 1578.1 (57/64). evidence

Qwen3.5-9B FP8 two cards (TP2), no speculation, XPU graph capture, identity-qualified aggregate decode vs concurrent users (c1-c8; warm pass) Lab-measured

Campaign c5 MTP0 ladder on two B70s. c1-c8 output-identity-qualified in both passes. Measured but withheld: c16 1013.9 (13/16), c32 1680.7 (27/32), c64 2079.9 (59/64). evidence

How much faster could this get? Projected — not measured

The ML Bottleneck physics engine projects what stock software, a tuned run, and the physical ceiling look like for this exact model, compression, card count, and software. The grade is optimization headroom against the tuned-run target, not model quality.