Qwen3.8 27B official FP8 on two Intel Arc Pro B70 cards
Alibaba's Qwen3.8 27B official block-FP8 weights on two Arc Pro B70 cards. The R187 profile (the R156 row-invariant W8A16 kernel and mixed-step GDN split, served with one whole-graph torch.compile instead of vLLM's piecewise split) is clean-boot-qualified at 54.935 tok/s MTP1 (FP16 target verifier, draft-only INT4 head), 70.142 tok/s MTP depth 2, 79.183 tok/s MTP depth 3, 82.396 tok/s MTP depth 4, 86.182 tok/s MTP depth 5, and 33.097 tok/s MTP0; every pair 12/12 against a same-configuration MTP0 oracle, repeat-exact at 224-300-token prompts, MTP0 output-identical to a single request through 64 concurrent users, MTP1 and depths 3-5 through 16, depth 2 through 4. On the piecewise compile MTP depth 2 emitted a phantom first token on one request in 64; on the whole-graph compile no pass has shown it (R182-R193, 2026-09-03). The cause is an unfixed upstream vLLM defect that also occurs on the unmodified image (R192/R194), so this is a configuration that avoids it on our deterministic build, not a fix; no patch, no image rebuild. A prebuilt copy of the exact image is on GitHub Container Registry (ghcr.io/steveseguin/vllm-openai-xpu-qwen38-fp8, digest sha256:173660ec…, equal to the image id the launchers verify); the source build in the guide remains the authoritative route.
Median of two clean-boot fresh-server class-balanced medians over the complete fixed 12-prompt/six-class natural-512 suite (48-78-token prompts), separate empty compile caches, 12/12 complete token arrays exact versus a same-configuration MTP0 oracle (depth-5 target-verified speculative decoding; MTP1 54.935, depth 4 82.396, MTP0 33.097 on the same line), canaries before and after, cache zero; repeat-exact at 100-300-token prompts; output-identical through 16 concurrent users (MTP0: through 64). proof file
MTP = multi-token prediction, a small draft the main model verifies · oracle = the fixed prompt set whose exact outputs are checked
Still missing before this becomes an install guide
tested clean-host Intel driver and Docker installation
independent host-driver/Docker installation and strict endpoint replay
beginner recovery flow
MTP1 output identity above 16 concurrent users (MTP0 is exact through 64; the MTP1 residual is not in any censused kernel)
Measured performance profiles Lab-measured
R187 FP8 TP2 MTP0 HTTP decode over exact active context Lab-measured
Measured on the published R187 line (R156 image, whole-graph torch.compile; 2026-09-03, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. MTP0 (target only). No value is interpolated. evidence
Measured on the published R139 row-invariant W8A16 image (2026-09-02, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. Derived as active context tokens divided by HTTP TTFT; MTP0. evidence
R139 FP8 TP2 MTP0 HTTP TTFT over exact active context Lab-measured
Measured on the published R139 row-invariant W8A16 image (2026-09-02, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. Direct HTTP TTFT, MTP0. evidence
R187 FP8 TP2 MTP1 HTTP decode over exact active context Lab-measured
Measured on the published R187 line (R156 image, whole-graph torch.compile; 2026-09-03, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. MTP1 with FP16 target verifier and draft-only INT4 head; every output matched the same-image MTP0 oracle (18/18 complete arrays). evidence
Measured on the published R139 row-invariant W8A16 image (2026-09-02, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. Derived as active context tokens divided by HTTP TTFT; MTP1. evidence
R139 FP8 TP2 MTP1 HTTP TTFT over exact active context Lab-measured
Measured on the published R139 row-invariant W8A16 image (2026-09-02, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. Direct HTTP TTFT, MTP1. evidence
R187 FP8 TP2 MTP depth-2 HTTP decode over exact active context Lab-measured
Measured on the published R187 line (R156 image, whole-graph torch.compile; 2026-09-04, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. MTP1 with FP16 target verifier and draft-only INT4 head; every output matched the same-image MTP0 oracle (18/18 complete arrays). MTP depth 2 matched the same-configuration MTP0 oracle on 18/18 complete arrays. evidence
R187 FP8 TP2 MTP depth-3 HTTP decode over exact active context Lab-measured
Measured on the published R187 line (R156 image, whole-graph torch.compile; 2026-09-04, clean boot): one-slot vLLM HTTP completions on two B70s at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, cache zero, canaries before and after. MTP1 with FP16 target verifier and draft-only INT4 head; every output matched the same-image MTP0 oracle (18/18 complete arrays). MTP depth 3 matched the same-configuration MTP0 oracle on 18/18 complete arrays. evidence
Measured vLLM HTTP completions on two B70s with the R187 image (MTP1, FP16 verifier, draft-only INT4 head), FP16 activations/KV, 64 service slots, 256-token total request capacity, max_num_batched_tokens=512, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite. Every point shown is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (harness --require-output-identity). c32 and c64 were measured but are withheld: 1/32 and 8/64 near-tie prompts took a different valid branch. One server, one pass per point; not a promoted concurrency speed record. evidence
Measured vLLM HTTP completions on two B70s with the R187 image (MTP0, target only), FP16 activations/KV, 64 service slots, 256-token total request capacity, max_num_batched_tokens=512, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite. Every point shown is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (harness --require-output-identity). All seven points including c32 and c64 passed 64/64. One server, one pass per point; not a promoted concurrency speed record. evidence
Official FP8 TP2 HTTP median TTFT under concurrency Lab-measured
Measured on the previous R50/R62 image (natural oneDNN W8A16 kernel), not yet re-measured on R139. Median request TTFT from the same two fresh-server, output-audited direct-P2P attempts. The service has 64 active slots, so c1-c64 are unqueued. Each point is the median of the two per-attempt p50 values; worst latency range across every reported p50/p95 metric was 4.404%. No point is interpolated or extrapolated. evidence
Official FP8 TP2 HTTP p95 TTFT under concurrency Lab-measured
Measured on the previous R50/R62 image (natural oneDNN W8A16 kernel), not yet re-measured on R139. p95 request TTFT from the same two fresh-server, output-audited direct-P2P attempts. The service has 64 active slots, so c1-c64 are unqueued. Each point is the median of the two per-attempt p95 values; c64 reached 1.526 seconds. No point is interpolated or extrapolated. evidence
Measured vLLM HTTP completions on two B70s with the R187 image (MTP1, FP16 verifier, draft-only INT4 head), FP16 activations/KV, 64 service slots, 256-token total request capacity, max_num_batched_tokens=512, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite. Every point shown is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (harness --require-output-identity). c32 and c64 were measured but are withheld: 1/32 and 8/64 near-tie prompts took a different valid branch. One server, one pass per point; not a promoted concurrency speed record. evidence
Measured vLLM HTTP completions on two B70s with the R187 image (MTP1, FP16 verifier, draft-only INT4 head), FP16 activations/KV, 64 service slots, 256-token total request capacity, max_num_batched_tokens=512, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite. Every point shown is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (harness --require-output-identity). c32 and c64 were measured but are withheld: 1/32 and 8/64 near-tie prompts took a different valid branch. One server, one pass per point; not a promoted concurrency speed record. evidence
Measured vLLM HTTP completions on two B70s with the R187 image (MTP1, FP16 verifier, draft-only INT4 head), FP16 activations/KV, 64 service slots, 256-token total request capacity, max_num_batched_tokens=512, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite. Every point shown is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (harness --require-output-identity). c32 and c64 were measured but are withheld: 1/32 and 8/64 near-tie prompts took a different valid branch. One server, one pass per point; not a promoted concurrency speed record. evidence
Measured vLLM HTTP completions on two B70s with the R187 image (MTP1, FP16 verifier, draft-only INT4 head), FP16 activations/KV, 64 service slots, 256-token total request capacity, max_num_batched_tokens=512, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite. Every point shown is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (harness --require-output-identity). c32 and c64 were measured but are withheld: 1/32 and 8/64 near-tie prompts took a different valid branch. One server, one pass per point; not a promoted concurrency speed record. evidence
How much faster could this get? No projection
This package's measured workload does not map cleanly onto a single model + compression + card-count shape, so no like-for-like projection is shown. The measured numbers above stand on their own.