moonshotai/Kimi-K2.6

Overview

Kimi K2.6 is an open-source, native multimodal agentic model built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes, as well as conversational and agentic paradigms.

Prerequisites

• vLLM version: >= 0.25.0 nightly for the optimized B300 EAGLE3 and native CPU KV offload path documented below
• Hardware (INT4): 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB)
• Hardware (NVFP4): 4x Blackwell GPUs; the optimized B300 path below was verified on vllm/vllm-openai:nightly-09663abde0f50944a8d5ea30120666024b503faa
• AMD support: 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12

NVIDIA B300: NVFP4 with Eagle3

The following text-only TP4 command mirrors the B300 configuration validated by InferenceX PR #2158. It uses the Kimi K2.6 Eagle3 MLA draft, TOKENSPEED_MLA attention, TRT-LLM ragged MLA prefill, FP8 KV cache, and full-and-piecewise CUDA graphs.

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve nvidia/Kimi-K2.6-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --block-size 64 \
  --gpu-memory-utilization 0.90 \
  --attention-backend TOKENSPEED_MLA \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 16384 \
  --stream-interval 10 \
  --enable-prefix-caching \
  --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4}'

Native CPU KV offload

Select Simple in the command builder's KV Offload row to extend the prefix cache into host DRAM with SimpleCPUOffloadConnector. The shared option uses 220 GiB per rank by default. The verified B300 TP4 run used 1,199 GiB total (299.75 GiB per rank); the equivalent explicit command is:

CPU_OFFLOAD_BYTES_PER_RANK=321854111744

vllm serve nvidia/Kimi-K2.6-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --block-size 64 \
  --gpu-memory-utilization 0.90 \
  --attention-backend TOKENSPEED_MLA \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 16384 \
  --stream-interval 10 \
  --enable-prefix-caching \
  --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4}' \
  --disable-hybrid-kv-cache-manager \
  --kv-transfer-config "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":${CPU_OFFLOAD_BYTES_PER_RANK},\"lazy_offload\":false}}"

Decode context parallelism

For higher concurrency, TP4/DCP4 was validated both with and without native CPU KV offload. DCP is intentionally guide-only rather than exposed as a command-builder option. Do not combine DCP with the Eagle3/TOKENSPEED_MLA flags above until vLLM PR #48180 lands. For the current pinned image, remove --attention-backend TOKENSPEED_MLA and --speculative-config, then add:

--decode-context-parallel-size 4

The successful agentic sweep covered these B300 points:

Serving path	Parallelism	Native CPU KV offload	Tested concurrency
Eagle3	TP8	No	1
Eagle3	TP4	No	2, 4, 8
Eagle3	TP4	Yes	8, 16, 32
DCP	TP4/DCP4	No	32, 64, 128
DCP	TP4/DCP4	Yes	64, 128, 256

AMD MI300X/MI325X

On 8x MI300X or MI325X (gfx942), use the standard W4A16 MoE path with AITER and INT4 QuickReduce.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --mm-encoder-tp-mode data

AMD MI350X/MI355X

On 8x MI350X or MI355X (gfx950), add --moe-backend flydsl to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --moe-backend flydsl \
  --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

Notes:

• The FlyDSL INT4 MoE path does not support expert parallelism; do not add --enable-expert-parallel.
• Keep --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'; it is required for this FlyDSL path on MI350X / MI355X.
• vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
• Keep vLLM's default block size unless you are tuning long-context throughput; --block-size 64 is safe to try.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Troubleshooting

• OOM errors: Lower --gpu-memory-utilization or adjust TP/EP to match your GPU count.
• Vision encoder performance: Use --mm-encoder-tp-mode data to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain.
• Unique multimodal inputs: Pass --mm-processor-cache-gb 0 to avoid caching overhead. For repeated inputs, --mm-processor-cache-type shm uses host shared memory for better performance at high TP settings.
• MoE kernel tuning: Use the benchmark_moe script from vLLM to tune Triton kernels for your specific hardware.
• Async scheduling: Enabled by default for better throughput. Disable if you encounter issues, and file a bug report to vLLM.
• Eagle3 with DCP: The current pinned image does not support the combination. Disable Eagle3/TOKENSPEED_MLA for DCP until vLLM PR #48180 is merged and available in the image.

References

• Kimi-K2.6 on Hugging Face
• NVIDIA Kimi-K2.6-NVFP4 on Hugging Face
• InferenceX Kimi-K2.6 B300 agentic sweep
• vLLM SimpleCPUOffloadConnector
• vLLM DCP + Eagle support PR
• vLLM multimodal inputs guide
• vLLM Expert Parallelism docs
• vLLM NixlConnector usage guide