Low-Latency Inference on AMD MI355X: 2,100+ Tokens per Second per User at Full Model Quality

Today we are announcing the fastest inference engine on AMD GPUs. Compared with the other major open-source engines on the same eight AMD Instinct MI355X GPUs, the Lithos engine runs DeepSeek V4.1 Flash up to 3.4× faster and Kimi K3 up to 2.5× faster, and it is the only one of them that runs each model at full model quality.

Results: stacking the deck against ourselves

We benchmarked the Lithos engine against four other major open-source engines, OSS-1 to OSS-4, each in its low-latency configuration, including development builds ahead of stable releases. Some configurations use reduced precision or more GPUs. The Lithos engine preserves each model’s full quality and official precision formats on AMD MI355X.

Speed

We measure decode speed as tokens per second per user, as shown in the plots below; higher is better. Every engine is credited the same speculative acceptance, 75% of speculated tokens, so the main difference between them is step time, the time to run one decode step; lower is better.

On DeepSeek V4.1 Flash, the other engines take 2.8–3.4× as long per step on eight MI355Xs. Even with half the GPUs, the Lithos engine is still the fastest: on four MI355Xs it takes 3.75 ms per step, less than any other engine on four or eight GPUs.

On Kimi K3, every other engine cuts precision and still takes 1.7–2.5× as long, even OSS-3, which checks only half as many tokens per step.

Cost per token

Fast decode is usually the expensive end of serving, so we priced it. We estimate cost per token using SemiAnalysis’s hyperscaler GPU ownership rates: $1.50 per MI355X-hour. For each configuration, we divide the total hourly GPU cost by the number of tokens generated per hour.

At the shared acceptance rate, Kimi K3 on our engine costs $5.11 per million tokens at 652 toks/s/user; the fastest other engine, OSS-3, costs $8.93 at 373 toks/s/user. For DeepSeek V4.1 Flash, ours costs $1.04 per million tokens on four MI355Xs at 1,598 toks/s/user and $1.57 on eight at 2,116 toks/s/user; the fastest other engine, OSS-2’s low-latency mode on four GPUs, costs $1.80 at 928 toks/s/user.

Long context

Low latency is especially challenging for long-context agents: as conversation history, documents, and tool outputs accumulate, each decode step must process a larger context. To evaluate how latency scales, we tested each engine on the same LongBench v2 documents, using eight documents at each prompt length from 1K to 1M tokens. When a published configuration could not fit a 1M-token prompt, we adjusted its documented memory settings to allow it.

On eight MI355X GPUs, the Lithos engine’s DeepSeek V4.1 Flash step time rises from 2.8 ms at 1K tokens to 3.3 ms at 1M. At that longest context, the fastest other engine we tested, OSS-2’s low-latency mode on four GPUs, takes 6.7 ms. For Kimi K3, our step time increases by just 15% from 1K to 128K tokens, reaching 10.4 ms, and rises to 16.5 ms at 1M.

Different hardware architectures require different specialization

Both MI355X and B300 have the same memory on paper: 288 GB of HBM at 8 TB/s of peak bandwidth, so reading the weights once per step costs about the same on both. However, compute, caches and the links between GPUs are organized differently. The result is a vastly different optimization space.

At low latency, a decode step is a long chain of small dependent stages: dozens per layer, across Kimi K3’s 93 layers, each waiting for the one before. Large batches hide these waits behind computation; low-latency serving exposes them. We call this waiting the coordination tax. The two architectures differ in three places that determine the overall cost.

Where software has to do more

1. Starting the next kernel

On NVIDIA GPUs starting with Hopper, Programmatic Dependent Launch (PDL) lets a dependent kernel begin before its predecessor in the same CUDA stream finishes. The first kernel signals when the next may launch; the next kernel can then perform independent setup, such as loading constants, while the first continues. Before consuming the first kernel’s outputs, it synchronizes to ensure that kernel has finished and its writes are visible. This can hide launch latency and setup work behind useful computation, though overlap is not guaranteed. AMD’s HIP has no equivalent early-launch mechanism within a single stream: dependent kernels normally wait for the previous kernel to finish, leaving launch latency exposed.

2. Summing across GPUs

Eight B300s connect through NVLink switches that can add values inside the switch (NVLS), so in an all-reduce the GPUs do none of the additions and each sends less data. Eight MI355Xs are wired to each other directly over Infinity Fabric, with no switch in between. The GPUs do all of the adding themselves, typically in two phases (reduce-scatter, then all-gather), or in one phase in which every GPU sends its full vector to all seven others, four times the data of the two-phase method.

3. Matrix cores

On B300, Tensor Memory Accelerator (TMA) copies tiles in asynchronously, a single thread can start a matrix multiply, and the tensor cores write their results into a dedicated Tensor Memory of 256 KB per SM rather than into registers, so loading, matrix math and the softmax of attention can run on different warps at the same time; a pair of SMs can also work on one large matrix tile together. On MI355X, the wave that does the math reads its operands into its own registers and receives the results there, so in practice it also runs the softmax on them: handing the results to another wave costs a round trip through Local Data Share (LDS). Overlap has to come from interleaving waves and instructions in software, within each wave’s fixed register budget.

Where MI355X gives software more room

Two other differences of the MI355X architecture are a larger register file and a documented instruction set that kernels can use directly.

1. A large register file

Each MI355X compute unit has 512 KB of registers, as much as a B300 SM’s registers and Tensor Memory combined; across the GPU that is 128 MiB, against 74 MiB on B300. More registers let a kernel keep more work staged: while one task runs, the next can already load its inputs into registers, so it starts without waiting on memory.

2. Control down to the instruction

On B300, the lowest level a kernel author can write is PTX, a virtual instruction set that NVIDIA’s closed assembler turns into machine code. On MI355X, the CDNA 4 instruction set is documented by AMD and compiled by the open-source LLVM AMDGPU backend, and kernels can use it directly from HIP through inline assembly. That gives us direct control over how each step’s work is laid out and scheduled.

Specialization unlocks engine performance

To cut the coordination tax, we rewrote how a whole layer executes instead of tuning kernels one at a time. The work has four parts.

1. Megakernels

A stock engine launches each operator as its own kernel, and every launch is a boundary where the GPU waits for one kernel to finish and the next to start. A decode step has many such boundaries, and with one request at a time they add up to a large part of the step. We turn a layer’s operators into tasks inside a few persistent megakernels, which stay resident and move from task to task as their inputs become ready, so most of those boundaries disappear.

2. Overlapping one megakernel with the next

Some boundaries remain. On NVIDIA, PDL lets the next kernel start its setup early; on MI355X the next kernel normally waits until the previous one completes. Where a boundary remains, we either fold its two sides into one megakernel or let the next one do its setup while the current one completes, doing in software what PDL does in hardware.

3. Communication inside the computation

Communication across GPUs usually runs as its own step between kernels, and every GPU waits for it to finish. At decode’s small message sizes, that wait costs more than moving the data. We run the communication inside the megakernels instead, sending each piece of the result as soon as it is computed, so it overlaps the computation around it.

4. A recursive, self-improving kernel harness

Making a kernel fast takes both theory and practice: theory says which ideas are worth trying, and measurement on the hardware identifies which ones work. We built a recursive, self-improving kernel optimization harness around this loop. It is fast and effective. It gives our engineers and researchers an exponential gain in efficiency, and it extends naturally to new models, new configurations and heterogeneous hardware architectures.

What comes next

Specialization goes a long way. Building the engine around how MI355X actually works made it the fastest engine on AMD GPUs, at full model quality. And the hardware still has more to give: a roofline analysis of the decode step, built from our own measurements on MI355X, shows there is still room before it reaches the GPU’s memory and compute limits. This is just the beginning of the journey.

We will keep working with AMD on fast inference on Instinct GPUs. Our engine is available today to a small group of partners. If you’d like to try it on your workload, don’t hesitate to reach out!