The Need for Speed in Agentic Inference

AI workloads are changing.

Over the last year, the center of gravity has shifted from single-turn text generation to agentic workflows. Agents no longer just answer a prompt. They reason, call tools, observe results, update plans, write code, run commands, inspect failures, and continue until the work is done.

That changes the meaning of inference performance.

For agents, inference happens inside a loop. Every model call affects how quickly the agent can take its next step and eventually complete the task. A delay that looks small on one model call compounds across the workflow, increasing the time it takes to produce a working patch, resolve an incident, or complete a long-running task. That's the real-world cost of inference latency.

A model that is slower at each step can turn a workflow that should feel alive into one that feels stuck. A system that is fast enough can keep the agent moving and complete more useful work in the same amount of time.

This is why we built LithosAI.

At LithosAI, we built an inference engine purpose-built for agents, delivering ultra-high tokens per second per user with low latency across the agent loop. On a standard 8×B200 GPU node, our engine runs Kimi K2.7 Code, a one-trillion-parameter model, at more than 1,000 peak tokens per second per user, sustaining 850 tokens per second on coding tasks, while preserving native precision and full model quality. Typical GPU serving for frontier models delivers 80 to 200 tokens per second per user; LithosAI runs 4 to 10× faster, at the same model quality.

This brings custom-silicon-class inference speed to standard GPUs, helping agents complete useful work faster without requiring specialized hardware.

Speed is the next battleground for AI differentiation

For the last few years, model quality has been the primary axis of AI progress. That focus was necessary. Better models unlocked new tasks, expanded what products could do, and determined whether an application worked at all.

But agentic workloads change the equation.

For agents, intelligence alone is not enough. That intelligence must be delivered quickly and consistently across every step of the agent loop. A better model determines what an agent can do; inference speed increasingly determines how quickly it can turn that capability into useful work.

You can already see this shift across the frontier. OpenAI's GPT-5.6 Sol announcement highlights serving GPT-5.6 Sol on Cerebras at up to 750 tokens per second, bringing frontier intelligence to customers at "unprecedented speed." Reaching that speed required custom silicon. LithosAI clears that performance on standard GPUs.

The frontier is no longer only about making the model smarter. It is increasingly about making frontier intelligence fast enough to use in real agent workflows.

For agents, what matters is how fast the loop is

Agent performance is increasingly shaped by how quickly each user can receive model output over time. A single user request can turn into hundreds or thousands of generated tokens across planning, tool use, debugging, and recovery. When those tokens arrive slowly, the entire agent slows down.

Your end-users don't care about benchmark scores. They want to know:

• How long does it take from the moment I ask for something to the moment the agent returns the result?
• Can I stay in the flow, or does every request become something I have to stop and come back to later?
• Can I give an agent ambitious, long-horizon tasks and compress hours or days of work into minutes?

In agentic coding and debugging workflows, inference speed directly shapes the user experience. A coding agent has to read the codebase, edit files, run tests, inspect failures, and try again. Users experience the entire loop, from the first prompt to the time it takes to produce a useful patch. Faster loops mean more attempts, faster recovery from failures, and shorter time to a working pull request.

For example, in site reliability engineering (SRE) and service management, the value of inference speed shows up in incident-response metrics. During an incident, an agent has to inspect alerts, query logs, compare traces, check recent deployments, and recommend actions while the system is still failing. A faster loop can surface the right signal sooner, shorten the path from evidence to mitigation, and help reduce mean time to detection and mean time to resolution. A slow agent summarizes the outage after the fact; a fast agent helps address the incident while it is still unfolding.

For long-horizon tasks, speed compounds. Agents working on complex goals may execute thousands of model calls across workflows that would otherwise take days or weeks. Delays at each step accumulate into hours of waiting. The math is simple: a task that generates one million tokens spends nearly three hours waiting on token generation at 100 tokens per second; at 1,000 tokens per second, it takes under 17 minutes. Faster inference compresses those workflows into hours or minutes, enabling agents to complete substantially more work between human check-ins.

Why you need to start measuring time to useful work

Production agents need a new metric: time to useful work, measuring how long it takes an agent to turn a user request into meaningful progress.

Aggregate throughput and cost per token still matter, but they do not capture what users actually experience. What matters is how quickly each agent can complete useful work and how quickly the infrastructure can close the loop for each active user.

Time to useful work captures what benchmark scores, cost per token, and aggregate throughput miss. It asks how much useful work one user can get per hour, per day, or per week, and how much of a long-running task is actual progress rather than accumulated waiting across thousands of model calls.

Coding agents expose this through time to a working pull request. SRE agents expose it through mean time to detection and resolution. Long-horizon agents expose it through how much progress they can make over a workday, or between human check-ins.

Tokens per second per user is the infrastructure metric that drives time to useful work. Higher per-user speed reduces waiting across dependent model calls, keeps the loop moving, and enables each agent to complete more useful work in less time.

Inference performance is agent performance.

Build ultra-fast agents with LithosAI

As model quality continues to move quickly, speed will be one of the clearest ways to differentiate. The winners will be the teams that can deliver model intelligence as useful work the fastest.

At LithosAI, we built agent infrastructure for that future: ultra-fast agentic inference on standard GPUs, delivering over 1,000 peak tokens per second per user and sustaining 850 on coding tasks, with low latency across the loop and faster time to useful work.

We are opening this up to teams building production agents. Try LithosAI and see what your agents can do when the loop moves faster.