vLLM

AMD × vLLM Semantic Router — 𝘄𝗲’𝗿𝗲 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝗦𝘆𝘀𝘁𝗲𝗺 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝘁𝗼𝗴𝗲𝘁𝗵𝗲𝗿. With AMD, vLLM Semantic Router is evolving into an Intelligence Control Plane: • 𝗪𝗼𝗿𝗹𝗱 𝗜𝗡: secure what comes IN (inputs) • 𝗪𝗼𝗿𝗹𝗱 𝗢𝗨𝗧: govern what goes OUT (actions) • 𝗟𝗼𝗻𝗴-𝗧𝗲𝗿𝗺 𝗦𝘁𝗮𝘁𝗲: protect what persists (memory/state) When these three lifelines are secured, vLLM-SR stops being just a model selector. It becomes the answer to a fundamental question: 𝗛𝗼𝘄 𝗱𝗼 𝘄𝗲 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 𝗳𝗿𝗼𝗺 𝗮 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴-𝘁𝗶𝗺𝗲 𝗮𝘀𝗽𝗶𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻𝘁𝗼 𝗮 𝗿𝘂𝗻𝘁𝗶𝗺𝗲 𝗶𝗻𝘀𝘁𝗶𝘁𝘂𝘁𝗶𝗼𝗻? Read the blog: https://lnkd.in/gkwfCxHW Shout out to teams who made this possible: Andy Luo, Haichen Zhang, Huamin Chen, Chen Wang, Yue Zhu and all the team from AMD and vLLM. #AMD #vLLM #SemanticRouter #SystemIntelligence #ROCm #MixtureOfModels #AIInfrastructure #Governance #AIAlignment #OpenSource

🚀🚀🚀 We’re excited to support NVIDIA and their new open family of models: NVIDIA Nemotron 3! Open in weights, data, tools, and training, Nemotron 3 is built for multi-agent apps and features: ⚡️ An efficient hybrid Mamba‑Transformer MoE architecture 📚 1M token context for long-term memory and improved reasoning 🧠 Multi‑environment reinforcement learning via NeMo Gym for advanced skill adaptation Plus NVFP4 pre-training, latent MoE, 1T tokens of data, and more! Read more: https://lnkd.in/gsz9qSTp

Multimodal serving pain: vision encoder work can stall text prefill/decode and make tail latency jittery. We built Encoder Disaggregation (EPD) in vLLM: run the encoder as a separate scalable service, pipeline it with prefill/decode, and reuse image embeddings via caching. This provides an efficient and flexible pattern for multimodal serving. Results: consistently higher throughput (5–20% across stable regions) and significant reductions in P99 TTFT and P99 TPOT. Read more: https://lnkd.in/gFr7ZyqC

New vLLM blog drop: 𝗧𝗼𝗸𝗲𝗻-𝗟𝗲𝘃𝗲𝗹 𝗧𝗿𝘂𝘁𝗵 — introducing 𝗛𝗮𝗹𝘂𝗚𝗮𝘁𝗲, a production-grade hallucination gate that catches 𝗲𝘅𝘁𝗿𝗶𝗻𝘀𝗶𝗰 𝗵𝗮𝗹𝗹𝘂𝗰𝗶𝗻𝗮𝘁𝗶𝗼𝗻𝘀 (the “tool returned correct data, model still made stuff up” problem) before they hit users. What’s different: 1. 𝗖𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹: only fact-checks when the prompt needs it. 2. 𝗧𝗼𝗸𝗲𝗻-𝗹𝗲𝘃𝗲𝗹: flags the exact unsupported spans + NLI explains whether it’s contradiction vs not supported. 3. 𝗙𝗮𝘀𝘁 + 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹: CPU-only, ~50ms overhead, block / warn in header or body before routing to human review. Guardrails as 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲, not “please be truthful” vibes. 𝗦𝗵𝗶𝗽 𝘁𝗿𝘂𝘀𝘁. 𝗡𝗼𝘁 𝘃𝗶𝗯𝗲𝘀. https://lnkd.in/g7AUJ3XU #vLLM #HaluGate #SemanticRouter #SemanticRouting #LLMRouting #ModelRouting #MixtureOfModels #MoM #InferenceRouting #AgenticAI #AIInfra #LLMOps #MLOps #ProductionAI #ToolCalling #RAG #Hallucinations #TrustAndSafety #Observability #ReliabilityEngineering

vLLM was mentioned in about half of the PyTorch Conference 2025 talks (≈53/117)! Several months ago, when the PyTorch conference agenda was out, we noticed that there would be 5 dedicated talks about vLLM. After the PyTorch conference, we find that actually about half of the talks mentioned vLLM! Check out https://lnkd.in/geknb3CA for more details. These mentions can be roughly categorized as: - vLLM itself as a hosted project in PyTorch foundation, including vLLM general introduction, vLLM V1 on AMD GPUs, vLLM-triton-backend - Usage in the open-source ecosystem, including Ray, PyTorch Core, PyTorch Symmetric Memory, PyTorch and WheelNext, TorchForge, TorchTitan, Flex Attention, TRL, torch.accelerator, Transformers, llm-d, verl, LMCache, torch.compile, Kubernetes Gateway, LibTorch ABI - Diverse hardware support, including Arm CPU, IBM Spyre, AMD ROCm, Google TPU, Rebellions, AWS Neuron, Ascend, Intel XPU - Usage in various companies/organizations, including AI2, PyTorch Lightning, Crusoe AI, Oracle, Spotify, Simplismart, BAAI, D.E. Shaw Research, Lemurian Labs, OpenTeams, Luma AI, Union AI, Amazon Rufus, Moonshot AI, Snowflake - General inference related discussions, including NVIDIA's Dynamo and TensorRT-LLM We are truly honored that LLM inference becomes a topic that is so important, and vLLM plays a critical role in the inference ecosystem. The honor belongs to all the ecosystem partners and contributors. Looking forward to a more fruitful year in 2026! 🚀

Low-bit LLM quantization doesn’t have to mean painful accuracy trade-offs or massive tuning runs. Intel's AutoRound PTQ algorithm is now integrated into LLM Compressor, producing W4A16 compressed-tensor checkpoints you can serve directly with vLLM across Intel Xeon, Gaudi, Arc GPUs, and more. Huge thanks to the Intel Corporation Neural Compressor, Red Hat Developer optimization, LLM Compressor, and vLLM community for making this integration happen. Repo: https://lnkd.in/gvsCKgzr Blog: https://lnkd.in/gUKDKeuM

🎉 Congrats to the Mistral AI team on the launch of Devstral 2! vLLM now delivers Day-0 support for the Devstral 2 Instruct models — optimized for agentic coding, deep codebase exploration, and multi-file editing at scale. Feel free to reach out 👇 uv pip install vllm --torch-backend=auto --extra-index-url https://lnkd.in/gxdk2_4M vllm serve mistralai/Devstral-2-123B-Instruct-2512 \ --tool-call-parser mistral \      --enable-auto-tool-choice \      --tensor-parallel-size 8

📢 vLLM v0.12.0 is now available. For inference teams running vLLM at the center of their stack, this release refreshes the engine, extends long-context and speculative decoding capabilities, and moves us to a PyTorch 2.9.0 / CUDA 12.9 baseline for future work. Two engine paths are now available for early adopters: GPU Model Runner V2 (refactored GPU execution with GPU-persistent block tables + a Triton-native sampler) and prefill context parallel (PCP) groundwork for long-context prefill. Both are experimental, disabled by default, and intended for test/staging rather than new production defaults. Beyond the engine, v0.12.0 ships EAGLE speculative decoding improvements, new model families, NVFP4 / W4A8 / AWQ quantization options, and tuned kernels across NVIDIA, AMD ROCm, and CPU. We recommend building new images with PyTorch 2.9.0 + CUDA 12.9, validating on staging workloads, and only then rolling out more broadly. Release notes: https://t.co/9Xx5CqREhi

🚀 vLLM now offers an optimized inference recipe for DeepSeek-V3.2. ⚙️ Startup details Run vLLM with DeepSeek-specific components: --tokenizer-mode deepseek_v32 \ --tool-call-parser deepseek_v32 🔖 Usage tips Enable thinking mode in vLLM: – extra_body={"chat_template_kwargs":{"thinking": True}} – Use reasoning instead of reasoning_content 🙏 Special thanks to Tencent Cloud for compute and engineering support. 🔗 Full recipe (including how to properly use the thinking with tool calls feature): https://lnkd.in/eedav4bp #vLLM #DeepSeek #Inference #ToolCalling #OpenSource