ZenDNN 5.2.1 moved quantization from experimental to production-grade, with asymmetric WOQ, INT8 dynamic quantization, and standardized accuracy validation. That release proved that AMD EPYC™ CPUs could deliver serious quantized LLM inference at scale. ZenDNN 6.0 takes the next step: FP16 functional support for the upcoming 6th Gen AMD EPYC™ Server processors, Mixture-of-Experts (MoE) model optimization, and the broadest vLLM compatibility window for ZenDNN yet.
Why This Matters: From Current Gen to Next Gen
ZenDNN 5.2 introduced the re-engineered ZenDNN runtime with Low Overhead API and vLLM V1 integration. ZenDNN 5.2.1 brought quantization to production grade. ZenDNN 6.0 builds on this with a dual focus: expanding model support and improving inference performance on the current generation 5th Gen AMD EPYC™ processors, while also laying the groundwork for next-generation hardware.
On the current-gen side, fused MoE operators, optimized attention kernels, and broader quantization options enable better throughput and support for newer model architectures like Mixture-of-Experts. Looking ahead, FP16 functional support prepares the software stack for the upcoming 6th Gen EPYC™ processors and their hardware FP16 capabilities. vLLM compatibility through v0.23.0 keeps both current and future deployments aligned with the fast-moving open-source ecosystem.
What’s Under the Hood
ZenDNN 6.0 builds on the modular multi-backend architecture introduced in 5.2, extending it with FP16 operator support, MoE-specific fusions, optimized attention kernels, and deeper quantization integration across the zentorch plugin and ZenDNN runtime.
Key Highlights at a Glance
FP16 Functional Support: Native FP16 data path for the upcoming 6th Gen EPYC™ processors across MatMul (with DLP), BatchMatMul, and Embedding operators in the ZenDNN core library and zentorch plugin.
Mixture-of-Experts (MoE) Optimization: BF16 Fused MoE operator, quantized MoE support (DA8W8, INT4 WOQ) in the vLLM pipeline, and Group MatMul optimized for expert parallelism.
Expanded vLLM Plugin Compatibility: Version parity from vLLM 0.20.0 through 0.23.0, maintaining the zero-code-change acceleration philosophy.
Quantization Pipeline Enhancements: LLM-Compressor integration, upstream quantized linear dispatch to vLLM, new TorchAO configuration presets, and DA8W8 support for MoE architectures.
zentorch Features and Optimizations: Optimized attention implementation in the ZenDNN core library, AOTI (CPP_Wrapper) support for Ahead-of-Time Inductor compilation, and targeted BF16 LLM throughput optimizations including matmul overhead reduction and runtime API improvements.
Modernized Framework Stack: PyTorch 2.12.0, TensorFlow 2.21.0 (backward build compatibility TF 2.16–2.21), Python 3.10–3.13, and vLLM 0.20.0–0.23.0.
Performance
Through ZenDNN 5.2.1, performance was measured using offline benchmarking, where a fixed batch of prompts is processed without simulating concurrent users. With ZenDNN 6.0, we move to online serving benchmarks, where requests arrive continuously at varying rates, reflecting real-world deployment conditions.
We also adopt multi-instance serving as the primary benchmarking mode for LLM inference on AMD EPYC™ CPUs. Rather than running a single vLLM process across an entire socket, we run N smaller vLLM instances, each pinned to a disjoint block of CPU cores. This keeps each instance’s working set NUMA-local, avoids cross-instance interference, and lets aggregate throughput scale closer to the available core count.
Load is generated using GuideLLM, which sends chat workload requests at configured concurrency levels to an NGINX load balancer that round-robins traffic across the instances. Each run captures standard serving metrics (median and p95): request latency, time to first token (TTFT), inter-token latency (ITL), time per output token (TPOT), and output throughput (tokens/s). Page cache is dropped between runs for cold-start parity, and the load generator is NUMA-pinned so the client never contends with the instances under test.
Single-Socket Analysis
Dual-Socket Analysis
Accuracy
vLLM-zentorch quantization preserves model accuracy within tight margins of the BF16 baseline. Quantized models are validated using the LM Evaluation Harness with 5-shot prompting across standard benchmarks including GSM8K and ChartQA.
Conclusion
ZenDNN 6.0 features across AMD 5th Gen and upcoming 6th Gen EPYC™ processors:
5th Gen AMD EPYC™ processors: Fused MoE operators, optimized attention kernels, and expanded quantization options improve throughput and bring support for newer model architectures.
6th Gen AMD EPYC™ processors: FP16 functional support lays the software foundation for half-precision inference on upcoming hardware.
Our Commitment to the Ecosystem Our upstream-first philosophy continues. The quantized linear dispatch in ZenDNN 6.0 has been upstreamed to vLLM (v0.22.1+). The expanded vLLM compatibility (now through 0.23.0) helps ensure that AMD EPYC™ CPU users can adopt the latest open-source inference releases without losing ZenDNN acceleration.
Run More with Less FP16 functional support will establish the foundation for half-precision inference on the upcoming 6th Gen EPYC™ processors. Combined with DA8W8 and INT4(WOQ) quantization, ZenDNN 6.0 offers the widest range of precision options for balancing throughput, memory, and accuracy. For organizations running AMD EPYC™ infrastructure, ZenDNN 6.0 makes it possible to run more AI workloads on existing hardware
𝐒𝐭𝐚𝐲 𝐢𝐧𝐟𝐨𝐫𝐦𝐞𝐝 𝐰𝐢𝐭𝐡 𝐨𝐮𝐫 𝐥𝐚𝐭𝐞𝐬𝐭 𝐮𝐩𝐝𝐚𝐭𝐞𝐬 𝐛𝐲 𝐣𝐨𝐢𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 WhatsApp Channel now! 👈📲
𝑭𝒐𝒍𝒍𝒐𝒘 𝑶𝒖𝒓 𝑺𝒐𝒄𝒊𝒂𝒍 𝑴𝒆𝒅𝒊𝒂 𝑷𝒂𝒈𝒆𝐬 👉 Facebook, LinkedIn, Twitter, Instagram