Qwen3.8 27B AutoRound INT4, fixed-K batch-invariant profile, on two Intel Arc Pro B70 cards
devan-carlin's AutoRound INT4 tensors served through vLLM's plain-GPTQ oneDNN W4A16 path with a rebuilt kernel library that pins a two-tier fixed-K GEMM strategy, FP16 linears in 32-row pieces, single-split attention and size-independent Inductor reductions, on the FP8 lane's whole-graph deterministic stack. Single-request output is repeat-exact and speculative decoding is lossless against the MTP0 oracle at every depth measured; MTP0 output is byte-identical to a single request through 64 concurrent users (a near-tie prompt can differ in some runs), speculative depths through 16.
Public headline Pending
Research-status (2026-09-06). Headline: TP2 depth 4 with XPU graph capture and the draft-only INT4 lm_head (R257, R256 image) 112.362/112.325 tok/s, G2 12/12 and G3 12/12 vs the eager MTP0 oracle, acceptance 3.51; depth 5 109.971/110.069, depth 6 108.340/108.373 (R258). Graphs without the draft head (R247/R250/R253): MTP0 49.833/49.887, depth 1 76.723/76.629, depth 4 91.004/91.012, depth 5 88.844/89.117, depth 6 84.060/83.982. Eager R239 matrix: TP2 MTP0 34.210/35.640, d1 51.098/50.088, d2 61.140/61.541, d3 67.613/67.831, d4 68.552/67.789; TP1 MTP0 32.960/32.945, d1 49.640/49.468, d2 56.505/56.449, d3 58.471/58.474, d4 56.292/56.251; every pair 12/12 vs each other and vs the MTP0 oracle. Identity ladders: MTP0 exact c1-c64 (eager and under graphs); speculative depths exact through c16 (eager TP2) / c8 (TP1, graphs), c32 >= 29/32, c64 >= 58/64. R258/R259 (depths 5-6 and ladders on the headline configuration) pending. No promotion or LocalMaxxing submission yet. Diagnostic measurements remain in the guide and evidence, but none is presented as the package headline.
Still missing before this becomes an install guide
clean-host replay
Measured performance profiles Lab-measured
INT4 fixed-K TP2 MTP0 (graphs) HTTP decode over exact active context Lab-measured
Measured 2026-09-06 (R260b, clean boot) on the headline configuration (R256 image, two B70s, XPU graph capture FULL_DECODE_ONLY sizes 1-8, strict launcher env): one-slot vLLM HTTP completions at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, max-model-len 33024, max-num-batched-tokens 4096, cache zero, canaries before and after. MTP0 (target only). No value is interpolated. evidence
INT4 fixed-K TP2 MTP depth 4 (graphs, draft-only INT4 head) HTTP decode over exact active context Lab-measured
Measured 2026-09-06 (R260b, clean boot) on the headline configuration (R256 image, two B70s, XPU graph capture FULL_DECODE_ONLY sizes 1-8, strict launcher env): one-slot vLLM HTTP completions at exactly 2K, 4K, 8K, 16K, 24K, and 32K active context, unrepeated technical prose, Python, and structured documents (three requests per depth, median shown), 128 output tokens, max-model-len 33024, max-num-batched-tokens 4096, cache zero, canaries before and after. MTP depth 4 with the FP16 target verifier and draft-only INT4 head; every output matched the same-configuration MTP0 oracle (18/18 complete arrays). Class spread is wide at every depth (Python code accepts more draft tokens); the median of the three classes is shown. evidence
Measured vLLM HTTP completions on two B70s with the R224 image (fixed-K W4A16, FP16 row-chunk linears) and VLLM_BATCH_INVARIANT=0 (corrected 2026-09-06: vLLM's own batch-invariant switch was never in effect on this lane; the strict launchers pin it to 0 and vLLM refuses to boot the GDN backend with it set), no speculation, FP16 activations/KV, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite (R226). Every point is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte. One server, one pass per point; not a promoted concurrency speed record. evidence
As above with MTP depth 4 on the R228 image, VLLM_BATCH_INVARIANT=0 (corrected 2026-09-06: vLLM's own batch-invariant switch was never in effect on this lane; the strict launchers pin it to 0 and vLLM refuses to boot the GDN backend with it set) and Inductor split_reductions=false (R232). Points through c16 are output-identity-qualified; c32 (31/32) and c64 (63/64) were measured but are withheld because near-tie prompts took a different valid branch. evidence
One B70 (TENSOR_PARALLEL_SIZE=1), R228 image, VLLM_BATCH_INVARIANT=0 (corrected 2026-09-06: vLLM's own batch-invariant switch was never in effect on this lane; the strict launchers pin it to 0 and vLLM refuses to boot the GDN backend with it set), Inductor split_reductions=false, no speculation, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite (R239 TP1 depth-1 campaign MTP0 ladder). Every point is output-identity-qualified against its sequential oracle. One server, one pass per point. evidence
INT4 fixed-K TP2 MTP0 with XPU graph capture, identity-qualified aggregate decode vs concurrent users (c1-c64) Lab-measured
Two B70s, R228 image, VLLM_BATCH_INVARIANT=0 (corrected 2026-09-06: vLLM's own batch-invariant switch was never in effect on this lane; the strict launchers pin it to 0 and vLLM refuses to boot the GDN backend with it set), split_reductions=false, VLLM_XPU_ENABLE_XPU_GRAPH=1 with FULL_DECODE_ONLY capture sizes 1-8, no speculation, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite (R251). Every point is output-identity-qualified against its sequential oracle. One server, one pass per point. evidence
INT4 fixed-K TP2 MTP depth 4 with graph capture and the draft-only INT4 head, identity-qualified aggregate decode vs concurrent users (c1-c16, warm pass) Lab-measured
Two B70s, R256 image, split_reductions=false, VLLM_XPU_ENABLE_XPU_GRAPH=1 (FULL_DECODE_ONLY, sizes 1-8), VLLM_XPU_DRAFT_LM_HEAD_INT4=1, MTP depth 4, cache disabled, 128 returned raw token IDs per response on the 64-prompt small-context suite; every rung run twice on one server and the second (warm) pass shown (R265b, 2026-09-06): the first pass of a fresh server carries a one-time Dynamo recompilation on the first multi-user batch. Every point through c16 is output-identity-qualified (16/16 at every rung). Above 16 users the corrected re-measurement with the W4A16 pad switch forwarded (R281) gives c32 632.6 (30/32) and c64 603.7 (62/64, admission-limited at max-model-len 256), withheld because near-tie prompts take a different valid branch; the R276 image with capture sizes to 320 (R282) adds +22%/+10% at two/four users (191.0/294.7). evidence
R284 (2026-09-06): two B70s, R276 image, XPU graph capture FULL_DECODE_ONLY sizes to 320, strict launcher env with the W4A16 pad switch forwarded off, max-num-seqs 128, max-model-len 512, max-num-batched-tokens 1024, no speculation, cache disabled, 128 returned raw token IDs per response on the small-context suite; every rung run twice on one server, warm pass shown. c16/c32/c64 are output-identity-qualified in both passes (16/16, 32/32, 64/64). Measured but withheld: c96 915.4 (95/96 warm, 96/96 first pass) and c128 1085.3 (128/128 warm, 127/128 first pass), one near-tie prompt each. evidence
R284 (2026-09-06): as the MTP0 profile above with qwen3_next_mtp depth 4, FP16 target verifier and the draft-only INT4 lm_head. c16 and c32 are output-identity-qualified in both passes (16/16, 32/32). Measured but withheld: c64 591.4 (58/64), c96 590.6 (89/96), c128 584.7 (117/128); the same prompts diverge at the same token positions in every rung of 64 and above (near-tie flips in the >32-row W4A16 GEMM tier), and the depth-4 aggregate plateaus at 585-641 tok/s from c32 while MTP0 keeps scaling to 991 (c64) and 1085 (c128). Serve more than about 32 users without speculation. evidence
R285 (2026-09-06): one B70 (TENSOR_PARALLEL_SIZE=1, GPU_MEMORY_UTILIZATION=0.96), R276 image, XPU graph capture FULL_DECODE_ONLY sizes to 320, strict launcher env with the W4A16 pad switch forwarded off, max-num-seqs 128, max-model-len 512, max-num-batched-tokens 1024, no speculation, cache disabled, 128 returned raw token IDs per response on the small-context suite; every rung run twice on one server, warm pass shown. Every rung is output-identity-qualified in both passes (16/16 through 128/128). Peak 514.0 tok/s at 32 users; from 64 users the batch budget queues prefill (ttft_max 13-27 s). evidence
R285 (2026-09-06): as the one-card MTP0 profile above with qwen3_next_mtp depth 4, FP16 target verifier and the draft-only INT4 lm_head. c16 and c32 are output-identity-qualified in both passes (16/16, 32/32). Measured but withheld: c64 246.2 (62/64), c96 243.4 (90/96), c128 243.9 (123/128), near-tie divergences as on TP2. One card saturates at ~245 tok/s with depth 4 from 16 users; MTP0 on the same card reaches 514 at 32 users, so serve more than about 8-16 one-card users without speculation. evidence
R287 (2026-09-06): as the R284 profiles (two B70s, R276 image, capture sizes to 320, pad off, max-num-seqs 128, max-model-len 512, max-num-batched-tokens 1024, two passes, warm pass shown) with qwen3_next_mtp depth 1. c2, c4 and c32 are output-identity-qualified in both passes (R288/R287); c16 in the warm pass (16/16; the first pass of the fresh server had one near-tie flip, 15/16). Measured but withheld: c64 842.0 (61/64), c96 906.3 (92/96), c128 894.8 (121/128). Depth 1 is the fastest setting through 32 users (depth 2: 723.4, depth 4: 641.3, no speculation: 815.0 at c32); from 64 users no speculation is faster and exact (992 at c64). R288 adds c2 147.3, c4 268.6 and c8 456.0 (7/8 in both passes, one near-tie prompt, withheld); depth 4 (R282) is faster at 1-4 users (191.0/294.7), depth 1 from 8 users. evidence
R289 (2026-09-06): as the one-card R285 profiles (TENSOR_PARALLEL_SIZE=1, R276 image, capture sizes to 320, pad off, max-num-seqs 128, max-model-len 512, max-num-batched-tokens 1024, two passes, warm pass shown) with qwen3_next_mtp depth 1. c4, c8 and c16 are output-identity-qualified in both passes; c32 in the warm pass (32/32; first pass 30/32) and already behind no speculation there (371.9 vs 513.8, prefill queueing with ttft_max 7.5 s). One card: depth 1 from 4 to 16 users (464.7 at 16 vs 345.7 without speculation), no speculation from 32 users. evidence
Measured vLLM HTTP completions on two B70s with the R293 image and VLLM_XPU_FP16_LINEAR_CLASSPAD=1 (R295, 2026-09-11): the served R276 configuration (capture sizes to 320, INT4 draft head, pad off) with every unquantized FP16 linear kept in one verified oneDNN M-class instead of 32-row pieces. No speculation, 128 returned raw token IDs per response on the 64-prompt small-context suite, second of two passes. Every point is output-identity-qualified: each concurrent output equals its own sequential oracle byte for byte (128/128 at 64 users). One server; not a promoted concurrency speed record. With the 5 ms admission stagger the same rung is 640/640 over ten passes at 1014.4 tok/s (R297). evidence
How much faster could this get? No projection
This package's measured workload does not map cleanly onto a single model + compression + card-count shape, so no like-for-like projection is shown. The measured numbers above stand on their own.