{ "date": "2026-05-09", "model": { "hfId": "Lasimeri/MiniMax-M2.7-int4-AutoRound", "base": "MiniMaxAI/MiniMax-M2.7", "localPath": "/mnt/corsair-external/llm-models/minimax-m2.7-int4-autoround", "quantization": "INT4 AutoRound W4A16" }, "hardware": { "gpu": "Intel Arc Pro B70", "gpuCount": 4, "vramGbPerGpu": 32, "cpu": "AMD EPYC 9015", "ramGb": 16, "os": "Ubuntu 24.04.4" }, "code": { "vllmSource": "/home/steve/src/vllm", "vllmPatch": "patches/vllm-minimax-llm-scaler-u4-decode-20260509.patch", "llmScalerSource": "/home/steve/src/llm-scaler/vllm/custom-esimd-kernels-vllm", "llmScalerPatch": "patches/llm-scaler-moe-int4-u4-decode-20260509.patch", "installedVenv": "/home/steve/.venvs/vllm-xpu", "kernelBuild": "oneAPI 2025.3.2, MoE-only llm-scaler extension, TORCH_XPU_ARCH_LIST=bmg" }, "change": { "summary": "Add an unsigned uint4 llm-scaler tiny MoE path and call it from vLLM only for decode-size MiniMax MoE batches.", "vllmGate": "VLLM_XPU_USE_LLM_SCALER_MOE=1, XPU, W4A16 symmetric, no expert parallel map, group_size=128, FP16 scales, x.dtype=float16, x.shape[0] <= 4", "qualityBoundary": "No sampling, routing, expert dropping, or power change. The path preserves the AutoRound W4A16 model and changes only MoE kernel execution/dequant. Standalone exact-shape checks versus vLLM fused experts were within about 3.052e-05 max absolute difference; the raw-u4 path matched the signed-compact compatibility path exactly in the nibble conversion microcheck." }, "results": [ { "label": "baseline_fp16_no_llm_scaler", "promptTokens": 512, "outputTokens": 128, "contextLength": 2048, "tokSOut": 20.17, "tokSTotal": 100.832219, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T011343Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T011343Z.json", "localmaxxingId": "cmoxnvmna00gmml01eqdyl428" }, { "label": "signed_llm_scaler_all_m_negative", "promptTokens": 512, "outputTokens": 128, "contextLength": 2048, "tokSOut": 12.27, "tokSTotal": 61.374, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T010605Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T010605Z.json", "submitted": false }, { "label": "u4_decode_only_p1_decode_isolation", "promptTokens": 1, "outputTokens": 128, "contextLength": 512, "tokSOut": 32.711775, "tokSTotal": 32.967336, "enabledMoeLayers": 62, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p1n128-20260509T014935Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p1n128-20260509T014935Z.json" }, { "label": "u4_decode_only_p512n128", "promptTokens": 512, "outputTokens": 128, "contextLength": 2048, "tokSOut": 29.74843, "tokSTotal": 148.742151, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 4.30274803499924, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T015634Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T015634Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-p512n128-20260509.payload.json", "localmaxxingId": "cmoxptkfd00hsml01hf2ajhhp", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-u4-decode-p512n128-20260509.response.json" }, { "label": "u4_decode_only_p512n256", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 33.033788, "tokSTotal": 99.101363, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 7.7496411530300975, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T022204Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T022204Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-p512n256-20260509.payload.json", "localmaxxingId": "cmoxq7cww00i8ml019ihbeqc9", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-u4-decode-p512n256-20260509.response.json" }, { "label": "u4_decode_only_ngram_gpu_p512n128_negative", "promptTokens": 512, "outputTokens": 128, "contextLength": 2048, "speculative": { "method": "ngram_gpu", "numSpeculativeTokens": 4, "promptLookupMin": 2, "promptLookupMax": 5 }, "outcome": "failed/stalled", "details": "Reached request processing at 0% and then worker processes terminated; vLLM reported RuntimeError: cancelled. No JSON throughput output.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n128-20260509T020356Z.log", "submitted": false }, { "label": "u4_decode_only_p512n256_ccl_fabric_vertex_override_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 32.726761, "tokSTotal": 98.180284, "env": { "CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK": "0" }, "outcome": "neutral/slower", "details": "The oneCCL topology override was previously near-neutral and remains slightly slower with the faster u4 decode path. Keep default topology recognition.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T023136Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T023136Z.json", "submitted": false }, { "label": "u4_decode_only_fp32_route_weights_p512n256", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 34.157842, "tokSTotal": 102.473527, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 7.4946185780572705, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T105353Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T105353Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-fp32-route-p512n256-20260509.payload.json", "localmaxxingId": "cmoy8hs3n002smk01ksgcpavr", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-u4-decode-fp32-route-p512n256-20260509.response.json", "change": "llm-scaler tiny u4 down kernel accepts FP32 topk route weights directly; vLLM bridge stops casting topk_weights to FP16 for every decode-layer call", "outcome": "improved" }, { "label": "u4_decode_only_xpu_graph_tp4_not_applicable", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "env": { "VLLM_XPU_ENABLE_XPU_GRAPH": "1" }, "outcome": "not_applicable", "details": "vLLM/XPU disables graph capture for this TP4 run before benchmarking: XPU Graph does not support capture communication ops, so cudagraph_mode is set to NONE. Run was stopped during shard loading; no JSON throughput output.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T110313Z.log", "submitted": false }, { "label": "u4_decode_only_p512n256_ccl_p2p_access_0_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "env": { "CCL_TOPO_P2P_ACCESS": "0" }, "outcome": "failed/stalled", "details": "USM mode reached model load and initial profiling/warmup, then repeatedly logged no available shared memory broadcast block for 60s intervals. Run was stopped; no JSON throughput output. Keep CCL_TOPO_P2P_ACCESS=1 for this MiniMax vLLM TP4 path.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T110741Z.log", "submitted": false }, { "label": "u4_decode_only_pp2_tp2_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 17.550271, "tokSTotal": 52.650812, "enabledMoeLayersLoggedOnPP0": 31, "peakVramGb": 28.0, "elapsedSeconds": 14.58666957897367, "parallelism": { "pipelineParallel": 2, "tensorParallel": 2 }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp2-p512n256-20260509T111942Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp2-p512n256-20260509T111942Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-pp2tp2-p512n256-20260509.payload.json", "localmaxxingId": "cmoy9exmf003lmk01d3it9cz2", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-u4-decode-pp2tp2-p512n256-20260509.response.json", "outcome": "negative/slower", "details": "PP2 x TP2 uses all four B70s and fits, but is much slower than TP4 at batch 1. Reduced TP collective degree does not offset pipeline bubbles and larger per-rank compute." }, { "label": "u4_decode_only_default_ipc_p512n256", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 34.578045, "tokSTotal": 103.734136, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 7.4035416769911535, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T112853Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T112853Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-default-ipc-p512n256-20260509.payload.json", "localmaxxingId": "cmoy9qat60040mk01l5y8n3al", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-u4-decode-default-ipc-p512n256-20260509.response.json", "change": "benchmark wrapper now supports CCL_IPC=default, which leaves CCL_ZE_IPC_EXCHANGE unset instead of forcing pidfd", "outcome": "improved/current best" }, { "label": "u4_decode_only_default_ipc_max_model_len_1024_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 1024, "tokSOut": 24.269656, "tokSTotal": 72.808968, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 10.548151146038435, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T113840Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T113840Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-default-ipc-maxlen1024-p512n256-20260509.payload.json", "submitted": false, "change": "MAX_MODEL_LEN reduced from 2048 to 1024 while keeping the p512/n256 workload and default oneCCL IPC behavior", "outcome": "negative/slower", "details": "The smaller compile/KV profile was slower even though the request fits inside 1024 tokens. vLLM compiled a (1, 1024) range and reported only 0.56 GiB available KV cache memory after loading. Keep MAX_MODEL_LEN=2048 for this path." }, { "label": "u4_decode_only_default_ipc_p512n512", "promptTokens": 512, "outputTokens": 512, "contextLength": 2048, "tokSOut": 37.136187, "tokSTotal": 74.272373, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 13.787091423990205, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T114811Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T114811Z.json", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-u4-decode-default-ipc-p512n512-20260509.payload.json", "localmaxxingId": "cmoyagit0004dmk014gk25e2k", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-u4-decode-default-ipc-p512n512-20260509.response.json", "outcome": "improved/best steady-state decode validation", "details": "Same TP4/default-IPC/u4 decode path as the p512/n256 current best, but with a longer generation window. Output throughput rose to 37.14 tok/s, showing the fixed prompt/setup portion is still diluting shorter-output measurements." }, { "label": "u4_decode_only_default_ipc_dcp2_p512n512_blocked", "promptTokens": 512, "outputTokens": 512, "contextLength": 2048, "parallelism": { "tensorParallel": 4, "decodeContextParallel": 2 }, "outcome": "blocked", "details": "vLLM rejects decode-context parallelism before model load for this MiniMax GQA/MQA config: tensor parallel size 4 must be greater than total num kv heads 8.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T115732Z.log", "submitted": false }, { "label": "u4_decode_only_default_ipc_max_model_len_4096_p512n512_negative", "promptTokens": 512, "outputTokens": 512, "contextLength": 4096, "tokSOut": 29.787984, "tokSTotal": 59.575969, "enabledMoeLayers": 62, "peakVramGb": 28.11, "elapsedSeconds": 17.188138523022644, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T115815Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T115815Z.json", "submitted": false, "outcome": "negative/slower", "details": "Increasing MAX_MODEL_LEN from 2048 to 4096 reduced p512/n512 output throughput. vLLM again reported only 0.56 GiB available KV cache memory, while the best MAX_MODEL_LEN=2048 run reported 1.02 GiB." }, { "label": "u4_decode_only_default_ipc_async_engine_p512n512_neutral", "promptTokens": 512, "outputTokens": 512, "contextLength": 2048, "tokSOut": 36.807084, "tokSTotal": 73.614167, "elapsedSeconds": 13.910365891002584, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "extraArgs": "--async-engine", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T120721Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T120721Z.json", "submitted": false, "outcome": "neutral/slightly slower", "details": "The async-engine benchmark path did not improve single-session p512/n512 throughput versus the default LLM benchmark path." }, { "label": "u4_decode_only_default_ipc_disable_detokenize_p512n512_neutral", "promptTokens": 512, "outputTokens": 512, "contextLength": 2048, "tokSOut": 37.124066, "tokSTotal": 74.248133, "elapsedSeconds": 13.791592606983613, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "extraArgs": "--disable-detokenize", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T121428Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T121428Z.json", "submitted": false, "outcome": "neutral", "details": "Disabling detokenization was effectively identical to the best p512/n512 result, so tokenizer/detokenizer overhead is not the bottleneck in this harness." }, { "label": "u4_decode_only_default_ipc_kv_fp8_inc_p512n512_blocked", "promptTokens": 512, "outputTokens": 512, "contextLength": 2048, "kvCacheDtype": "fp8_inc", "outcome": "blocked", "details": "vLLM accepted --kv-cache-dtype fp8_inc and logged the accuracy-warning path, but XPU worker initialization failed with Unsupported data type of kv cache: fp8_inc.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n512-20260509T122140Z.log", "submitted": false }, { "label": "torch_profiler_p512n32_blocked", "promptTokens": 512, "outputTokens": 32, "contextLength": 2048, "outcome": "blocked/stalled", "details": "vLLM torch profiler with XPU activities reached generation but repeatedly logged no available shared memory broadcast block found in 60 seconds. No profile trace files were emitted; run was stopped.", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n32-20260509T123226Z.log", "profileDir": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/profile-p512n32-20260509T123226Z", "submitted": false }, { "label": "standalone_xccl_decode_size_allreduce_pidfd", "outcome": "diagnostic", "env": { "CCL_ZE_IPC_EXCHANGE": "pidfd", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/xccl-decode-size-allreduce-4xb70-pidfd-20260509T124512Z.log", "results": [ { "label": "minimax_hidden_fp16", "bytes": 6144, "avgMs": 0.015159 }, { "label": "minimax_hidden_fp32", "bytes": 12288, "avgMs": 0.015141 }, { "label": "two_hidden_fp16", "bytes": 12288, "avgMs": 0.014726 }, { "label": "two_hidden_fp32", "bytes": 24576, "avgMs": 0.014906 }, { "label": "small_20kb_fp32", "bytes": 20480, "avgMs": 0.014614 }, { "label": "small_64kb_fp16", "bytes": 65536, "avgMs": 0.014835 } ], "details": "Standalone XCCL allreduce latency for MiniMax hidden-state sized payloads is about 15 us with pidfd IPC, so raw communication latency alone is too small to explain the remaining p512/n512 decode time.", "submitted": false }, { "label": "standalone_xccl_decode_size_allreduce_default_ipc_blocked", "outcome": "blocked/stalled", "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/xccl-allreduce-4xb70-20260509T124218Z.log", "details": "Standalone torch.distributed XCCL allreduce hung before the first 4 KiB result when CCL_ZE_IPC_EXCHANGE was unset/default. vLLM default IPC still benchmarks correctly, but standalone communication probes should force pidfd.", "submitted": false }, { "label": "llm_scaler_moe_kernel_trace_p1n4_diagnostic", "promptTokens": 1, "outputTokens": 4, "contextLength": 512, "outcome": "diagnostic", "env": { "LLM_SCALER_MOE_TRACE_KERNELS": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p1n4-20260509T124828Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p1n4-20260509T124828Z.json", "trace": { "kernelWaitSamples": 1753, "avgWaitMs": 0.057533, "medianWaitMs": 0.04465, "p95WaitMs": 0.088551, "minWaitMs": 0.02976, "maxWaitMs": 2.05121 }, "submitted": false, "details": "Diagnostic only: the timing hook waits on each submitted llm-scaler MoE kernel, so throughput is intentionally distorted. The median tiny up/down launch wait is around 0.045 ms, implying roughly 5.5 ms/token for 62 MoE layers at one up+down pair per layer, which is significant but still far below the best p512/n512 decode step of about 26.9 ms/token. Remaining work should profile the vLLM bridge, router/top-k, attention/KV, and graph/scheduler boundaries." }, { "label": "u4_decode_only_default_ipc_triton_attention_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 13.074278, "tokSTotal": 39.222834, "elapsedSeconds": 19.580430924950633, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "extraArgs": "--attention-backend TRITON_ATTN", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T130109Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T130109Z.json", "submitted": false, "outcome": "negative/slower", "details": "Triton attention was accepted and ran, but it regressed p512/n256 heavily compared with the default FlashAttention path. Keep default XPU FlashAttention." }, { "label": "u4_decode_only_default_ipc_block_size_128_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 24.004933, "tokSTotal": 72.0148, "elapsedSeconds": 10.66447458701441, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "extraArgs": "--block-size 128", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T130937Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T130937Z.json", "submitted": false, "outcome": "negative/slower", "details": "Forcing KV block size 128 under FlashAttention regressed p512/n256 versus the default XPU FlashAttention block size of 64." }, { "label": "u4_decode_only_default_ipc_forced_xpu_graph_p512n256_blocked", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "outcome": "blocked", "env": { "VLLM_XPU_ENABLE_XPU_GRAPH": "1", "VLLM_XPU_FORCE_GRAPH_WITH_COMM": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T131754Z.log", "submitted": false, "details": "The local force-with-comm graph experiment reached PIECEWISE graph mode, then failed during graph memory profiling with an AssertionError because tensor-parallel graph_capture requires a CudaCommunicator. XPU/XCCL graph capture for TP needs a vLLM communicator integration change before this can be benchmarked." }, { "label": "u4_decode_only_default_ipc_xpu_graph_fixed_kv_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 32.723015, "tokSTotal": 98.169045, "elapsedSeconds": 7.823240025027189, "peakVramGb": 28.11, "env": { "VLLM_XPU_ENABLE_XPU_GRAPH": "1", "VLLM_XPU_FORCE_GRAPH_WITH_COMM": "1", "VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "extraArgs": "--kv-cache-memory-bytes 256M", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T141049Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T141049Z.json", "patch": "patches/vllm-xpu-graph-noop-communicator-capture-20260509.patch", "localmaxxingPayload": "data/localmaxxing-minimax-m27-autoround-xpu-graph-fixedkv-p512n256-20260509.payload.json", "localmaxxingId": "cmoyfl7cm0057mk01suxo0glp", "localmaxxingResponse": "/home/steve/bench-results/localmaxxing-minimax-m27-autoround-xpu-graph-fixedkv-p512n256-20260509.curl-response.txt", "outcome": "negative/slower", "details": "Two local vLLM graph experiments let XPU TP4 enter PIECEWISE graph mode: graph_capture skips CUDA-only communicator capture for XpuCommunicator under VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1, and XPU follows the documented skip path for CUDA graph memory profiling. With KV pinned to 256 MiB, graph capture finished and used 1.15 GiB, but p512/n256 throughput fell below the non-graph default-IPC result of 34.578045 output tok/s. Do not promote XPU graph mode for this MiniMax path yet." }, { "label": "u4_decode_only_default_ipc_no_prefix_cache_p512n256_neutral", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 34.013802, "tokSTotal": 102.041406, "elapsedSeconds": 7.526356474030763, "env": { "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "extraArgs": "--no-enable-prefix-caching", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T132708Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T132708Z.json", "submitted": false, "outcome": "neutral/slightly slower", "details": "Disabling prefix caching did not improve the single random request benchmark. Keep the default prefix-caching behavior." }, { "label": "u4_decode_only_default_ipc_no_prefix_no_chunked_prefill_p512n256_blocked", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "outcome": "blocked", "extraArgs": "--no-enable-prefix-caching --no-enable-chunked-prefill", "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T132643Z.log", "submitted": false, "details": "vLLM rejected the scheduler config because max_num_batched_tokens=1024 is smaller than max_model_len=2048 when chunked prefill is disabled. It also warns that this model does not officially support disabling chunked prefill, so this branch was not pursued further yet." }, { "label": "u4_decode_only_default_ipc_fp16_router_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 24.525612, "tokSTotal": 73.576836, "elapsedSeconds": 10.438067801005673, "env": { "VLLM_MINIMAX_M2_FP16_ROUTER": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T133722Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T133722Z.json", "patch": "patches/vllm-minimax-m2-fp16-router-experiment-20260509.patch", "submitted": false, "outcome": "negative/slower", "qualityRisk": "FP16 router math can alter expert selection, so this is not a quality-preserving optimization unless separately validated.", "details": "The experiment precomputes FP16 copies of MiniMax gate weights after model load, computes router logits with FP16 linear, then casts logits back to FP32 for the existing top-k path. It regressed throughput substantially, so keep the default FP32 router." }, { "label": "u4_decode_only_default_ipc_xpu_decode_timing_p512n8_diagnostic", "promptTokens": 512, "outputTokens": 8, "contextLength": 2048, "env": { "VLLM_XPU_DECODE_TIMING": "1", "VLLM_XPU_DECODE_TIMING_RANK": "0", "VLLM_XPU_DECODE_TIMING_PRINT_EVERY": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "tokSTotal": 122.993979, "elapsedSeconds": 4.227849227027036, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n8-20260509T143536Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n8-20260509T143536Z.json", "patch": "patches/vllm-xpu-decode-timing-and-qk-contig-20260509.patch", "submitted": false, "outcome": "diagnostic_only", "details": "Short run with per-step logging; do not treat as a throughput benchmark. Steady decode on rank 0 settled at roughly 26.5-26.9 ms for runner.forward, while runner.preprocess and runner.postprocess were sub-ms. The llm-scaler bridge prints around 0.018-0.022 ms for steady calls, so the remaining ceiling is inside compiled model forward rather than vLLM request scheduling." }, { "label": "u4_decode_only_default_ipc_enforce_eager_timing_p512n4_diagnostic", "promptTokens": 512, "outputTokens": 4, "contextLength": 2048, "extraArgs": "--enforce-eager", "env": { "VLLM_XPU_DECODE_TIMING": "1", "VLLM_XPU_DECODE_TIMING_RANK": "0", "VLLM_XPU_DECODE_TIMING_PRINT_EVERY": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "tokSTotal": 491.072892, "elapsedSeconds": 1.0507605040329508, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n4-20260509T144519Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n4-20260509T144519Z.json", "patch": "patches/vllm-xpu-decode-timing-and-qk-contig-20260509.patch", "submitted": false, "outcome": "diagnostic_only", "details": "Eager mode is much slower and changes execution, but it exposes hooks hidden by torch.compile. Per-layer steady p50 timings on rank 0 were roughly qkv 0.034 ms, qk_norm 0.247 ms, rope 0.018 ms, kv_attention 0.098 ms, o_proj 0.092 ms, router_linear 0.073 ms, experts_total 0.145 ms, with TP allreduce.direct p50 about 0.047 ms and about three allreduce calls per layer. This points toward Q/K norm plus collectives as an XPU fusion target." }, { "label": "u4_decode_only_default_ipc_qk_contiguous_p512n256_negative", "promptTokens": 512, "outputTokens": 256, "contextLength": 2048, "tokSOut": 33.92, "tokSTotal": 101.765371, "elapsedSeconds": 7.546771497000009, "env": { "VLLM_MINIMAX_QK_CONTIG": "1", "CCL_ZE_IPC_EXCHANGE": "unset/default", "CCL_TOPO_P2P_ACCESS": "1" }, "log": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T145504Z.log", "json": "/home/steve/bench-results/minimax-m2.7-autoround-vllm/vllm-minimax-m27-autoround-tp4-p512n256-20260509T145504Z.json", "patch": "patches/vllm-xpu-decode-timing-and-qk-contig-20260509.patch", "submitted": false, "outcome": "negative/slower", "details": "Exact-math opt-in experiment that makes packed-QKV q/k views contiguous before MiniMax Q/K allreduce+RMSNorm. It regressed versus the p512/n256 default-IPC baseline of 34.578045 output tok/s, so non-contiguous Q/K views are not the main current bottleneck." } ], "conclusion": "The useful MiniMax path is reducing decode MoE and communication overhead rather than n-gram speculation. The unsigned llm-scaler tiny MoE decode path improves p512/n128 from 20.17 to 29.75 output tok/s, p512/n256 initially reached 33.03 output tok/s, removing the per-layer FP32-to-FP16 router-weight cast improved p512/n256 to 34.16 output tok/s, and leaving CCL_ZE_IPC_EXCHANGE at the oneCCL default improved it again to 34.58 output tok/s. A longer p512/n512 run with the same best path reached 37.14 output tok/s, making it the best current steady-state MiniMax AutoRound decode result. The latest diagnostics show steady compiled decode is about 26.9 ms inside runner.forward; scheduling/postprocess are sub-ms, eager timing points at Q/K norm plus repeated TP collectives as a real fusion target, and making Q/K slices contiguous is negative. Speculative ngram_gpu remains blocked/negative on this TP4 XPU harness; PP2/TP2, CCL USM, MAX_MODEL_LEN=1024, MAX_MODEL_LEN=4096, async-engine, XPU graph mode with fixed KV, Triton attention, block-size 128, FP16 router, and Q/K contiguous copies are negative/neutral. DCP is blocked by TP4 versus 8 KV heads, detokenization is not the bottleneck, XPU KV fp8_inc is unsupported, and built-in torch profiling stalls. Standalone pidfd XCCL allreduce is about 15 us at MiniMax hidden-state payload sizes, so the remaining bottleneck is not raw small allreduce latency alone." }