Infrastructure August 26, 2026

The Inference Chip Nvidia Paid $20 Billion For Is Now Shipping. Meet the Groq 3 LPX.

Nvidia's Groq 3 LPX is now in full production, turning the token-by-token decode bottleneck of agentic AI into a dedicated rack-scale hardware problem.

In December 2025, Nvidia reportedly paid roughly $20 billion to license inference technology from Groq Inc. and hired its founders. It was not an acquisition in the traditional sense — Groq the company still exists — but the technology, team, and architecture behind one of the fastest inference accelerators moved into Nvidia's AI factory platform.

At Hot Chips 2026 last week, Nvidia revealed what that deal was actually for.

The Groq 3 LPX — the third-generation version of the chip originally developed by Groq founder Jonathan Ross and his team — is now in full production. Production shipments begin this fall.

The headline number is 3,400 output tokens per second running Gemma 4 31B with a 100,000-token context window. Artificial Analysis benchmarked the configuration; Nvidia says that is four times faster in responsiveness than the nearest alternative platform for latency-sensitive workloads.

The Agentic Bottleneck

To understand why this matters, it helps to understand where AI systems actually slow down. Training a large model is slow by design, and prefill — the step where a model ingests a long document or conversation history — can tolerate some latency. The decode step, where the model generates tokens one at a time to build a response, is where users feel the difference between a system that feels alive and one that feels like it is buffering.

For agentic AI workflows — systems that reason through problems, plan actions, call external tools, and loop back to revise their work — decode runs constantly. An agent that writes code, checks its output, runs tests, and iterates may execute dozens of decode cycles per task. Slow token generation compounds at every step in that chain.

That was Groq's original insight: build silicon optimized for the sequential, memory-bandwidth-intensive nature of token-by-token generation. Not for training, and not primarily for prefill — for making tokens come out fast and predictably.

Nvidia's Vera Rubin architecture reflects the split directly. Rubin GPUs handle context ingestion and the portions of inference best suited to high-throughput compute, while Groq 3 LPX accelerates latency-sensitive decode work. Together, Nvidia says, the pair expands the usual throughput-versus-response-time tradeoff. That remains a vendor claim, but it is a specific architectural bet rather than just a faster-chip slogan.

Platform Architecture: Five Racks, Seven Chips

The Vera Rubin platform combines Groq 3 LPX with the Vera CPU, Rubin GPU, BlueField-4 storage, and Spectrum-6 networking in five purpose-built rack types that operate as a unified AI supercomputer. A single LPX rack contains 256 Groq 3 LPU accelerators connected by high-bandwidth chip interconnects.

The LPX system itself is designed around deterministic, compiler-scheduled execution and SRAM-first memory. Nvidia lists 128 GB of total SRAM, 40 PB/s of on-chip SRAM bandwidth, and 640 TB/s of scale-up bandwidth for the rack. The purpose is to minimize the jitter that becomes visible in small-batch, interactive inference.

The broader NVL72 platform's capacity, power, cooling, and networking choices are consequential because this is not a component intended to live in isolation. Nvidia is pitching a full-stack operating model for running AI at factory scale, including liquid-cooled, cableless compute trays and a software layer that can route work between GPUs and LPUs.

First Customer: Nebius Token Factory

Nebius is the first publicly confirmed customer planning to deploy Groq 3 LPX, integrating it into its Token Factory production inference platform. The name is worth noticing.

Token Factory reflects a shift in how enterprise AI infrastructure is being conceptualized: not simply as a cloud API routing requests to shared models, but as a production process with defined throughput, quality, and reliability targets. Buying dedicated inference silicon rather than renting capacity from a hyperscaler is a bet on predictable demand at scale.

The economics of that bet depend on volumes most organizations do not yet have. For cloud inference providers such as Nebius, however, the hardware math can favor ownership once utilization is consistently high.

What Comes Next

Production shipments begin in fall 2026, so most buyers will not have hardware before year end. Nebius is the only confirmed public commitment so far, and the 3,400-tokens-per-second figure is specific to one model and context length. Real-world performance across workloads will vary.

Still, the signal from Hot Chips is clear: Nvidia is treating agentic AI as a primary design target for its next platform generation, not an accommodation for an existing GPU architecture. The explicit split between context processing and token generation — handled by silicon tuned for each task — is Nvidia's architectural bet on how AI workloads will be structured for the next several years.

That bet is tied directly to the Groq deal. Nvidia did not need to spend $20 billion to build a general-purpose inference chip. It needed to address the bottleneck that makes agentic AI feel slow, and the fastest route was to license technology from the team that had already concentrated on that problem.

Sources

NVIDIA Newsroom — NVIDIA Groq 3 LPX now in full production: https://nvidianews.nvidia.com/news/nvidia-groq-3-lpx-now-in-full-production-with-world-class-speed-for-agentic-ai

NVIDIA Technical Blog — Inside NVIDIA Groq 3 LPX: https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-the-low-latency-inference-accelerator-for-the-nvidia-vera-rubin-platform/