# Qwen3.6 27B FP8 on 4x Intel Arc Pro B70

Date: 2026-05-04
Host: Ubuntu 24.04.4 LTS, AMD EPYC 9015, 16 GB RAM, 4x Intel Arc Pro B70 32GB
Runtime: vLLM 0.20.1, PyTorch 2.11.0+xpu, Intel Compute Runtime 26.14.37833.4
Model: `vrfai/Qwen3.6-27B-FP8`, static compressed-tensors W8A8 FP8 safetensors

## Root Cause

The static FP8 checkpoint uses singleton `(1,)` compressed-tensors attention scales for `q_proj`, `k_proj`, and `v_proj`. vLLM stored those singleton tensors directly as `_q_scale`, `_k_scale`, and `_v_scale`. Intel's XPU FlashAttention2 wrapper requires KV descales to be scalar-view tensors, checking `sum(k_descale.stride()) == 0`, so the length-1 stride `(1,)` tensor failed even though it contained a single scalar value.

Patch: `patches/vllm-xpu-fa2-compressed-tensors-scalar-scales.patch`

The patch reshapes singleton attention scales to scalar views with `tensor.reshape(())`. It should not change model quality because the exact scale values are preserved.

## Stable Results

All runs used power limits unchanged.

### 512 prompt / 256 output

Command shape:

```bash
ONEAPI_DEVICE_SELECTOR=level_zero:0,1,2,3 \
CCL_ATL_TRANSPORT=ofi \
CCL_ZE_IPC_EXCHANGE=sockets \
vllm bench latency \
  --model /home/steve/models/qwen3.6-27b-fp8-vrfai \
  --tensor-parallel-size 4 \
  --quantization compressed-tensors \
  --max-model-len 1024 \
  --input-len 512 \
  --output-len 256 \
  --num-iters-warmup 1 \
  --num-iters 5 \
  --gpu-memory-utilization 0.90 \
  --disable-log-stats \
  --no-enable-prefix-caching \
  --language-model-only
```

Result:

- Avg latency: `6.5199336920 s`
- Output throughput: `39.264 tok/s`
- Total throughput: `117.793 tok/s`
- LocalMaxxing: `cmorjwgi8000el7041jdd8faa`

### 512 prompt / 512 output

Same command shape, with `--output-len 512`.

Result:

- Avg latency: `12.3364390012 s`
- Output throughput: `41.503 tok/s`
- Total throughput: `83.006 tok/s`
- LocalMaxxing: `cmork3n3k000ujo04y73lbr1j`

This is effectively tied with the current Q4_0 TP3 sustained validation while using a higher-fidelity FP8 checkpoint.

## Negative / Neutral Screens

- TP2 static FP8 OOMs around LM-head allocation after about 15.4 GiB PyTorch allocations per GPU.
- Triton attention fallback is stable but slower: `33.081 tok/s` on 512 prompt / 256 output. LocalMaxxing: `cmorijmiu000kjr04cour40px`.
- `CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0` did not help and should stay disabled unless an allreduce microbenchmark proves otherwise.
- `VLLM_XPU_ENABLE_XPU_GRAPH=1` does not help TP4 because vLLM disables cudagraph mode when communication ops are present.

## Next Steps

- Consider upstreaming the scalar-view fix or carrying it as a local patch until vLLM/XPU handles singleton compressed-tensors attention scales directly.
- Compare against Q4_0 GGUF using the same prompt/output lengths when deciding which path is best for quality-preserving local serving.
- Revisit speculative/MTP only after base FP8 FA2 serving is stable, since the core path now reaches the Q4_0 validation range.