News
Optimization and Tuning
7+ hour, 34+ min ago (504+ words) This guide aims to help users improve vLLM Ascend performance at the system level. It includes OS configuration, library optimization, deployment guide, and so on. Any feedback is welcome. Install vLLM and vLLM Ascend: Please follow the Installation Guide to…...
inkling
1+ day, 12+ hour ago (92+ words) vLLM Inkling model configs for the text backbone and audio/vision towers. NVFP4 (ModelOpt) support for the Inkling mixture-of-experts. Text-only entry point ( inkling_model checkpoints). Top-level (multimodal) entry point. Bases: _TmlForCausalLMBase, SupportsMultiModal Builds the vision + audio towers on top of the shared text backbone....
bf16x3_router_gemm_cutedsl
1+ day, 10+ hour ago (64+ words) vLLM CuteDSL BF16x3 router GEMM. Computes X @ W.T for BF16 X with shape [N, K] and FP32 router weights W with shape [M, K] by decomposing each FP32 weight value into three BF16 residual terms inside the kernel, then accumulating the three BF16 MMA results into…...
attention
1+ day, 12+ hour ago (13+ words) vLLM Project the per-head relative branch r to per-distance logits....
fa4_rel_attention
1+ day, 12+ hour ago (82+ words) vLLM Round the FA4 scheduling bound up to a power of two. Return the split-KV cap for FA4 with sheared relative bias. Paged varlen FA4 over the bound K/V cache with the Inkling relative bias. q is (num_tokens, num_heads, head_dim); key_cache / value_cache are the paged caches (num_blocks, block_size, num_kv_heads, head_dim); block_table is…...
config
1+ day, 8+ hour ago (30+ words) vLLM Translate vLLM KV cache metadata for native offloading backends. Translate vLLM configuration into the native offloading boundary. Return whether a KV cache tensor uses a packed block stride....
mm_towers
1+ day, 12+ hour ago (77+ words) vLLM Fused CUDA kernels for the Inkling vision/audio towers. Both kernels keep the reference paths' fp32 accumulation and per-op bf16 rounding points (native rms_norm / F.gelu ); outputs are frequently bit-identical and otherwise differ by 1-2 bf16 ulps from reduction-order (real-checkpoint-weight cosine vs reference > 0.9999998). Fused gelu…...
norm
1+ day, 11+ hour ago (42+ words) vLLM Fused res = residual + delta; y = rmsnorm(res). Fused rmsnorm(embed_table[input_ids], weight) row gather + norm. add_rmsnorm(residual, delta, weight, eps) ¶ Returns (y, res); both are fresh tensors (cudagraph-friendly, no in-place update of the inputs)....
session
1+ week, 1+ day ago (232+ words) P2PSession — bidirectional session combining client + server roles. Wire protocol is unchanged. Both sides advertise their NIXL metadata via ConnectMsg when their session is connected; the peer's ConnectMsg triggers transport.add_remote_peer; ConnectAckMsg confirms the peer received our ConnectMsg, after which queued outgoing messages…...
kimi_k2
1+ week, 1+ day ago (42+ words) vLLM Kimi K2 parser for reasoning and tool calls. Kimi K2 tool call format:: The header before is Kimi's native tool call id. The function name is the final component before:N. Kimi K2 parser backed by the declarative parser engine....