It may be moving the KV cache.
There is an asymmetry buried inside transformer inference. Every token generates Q, K and V — Query, Key and Value. Q is ephemeral. It is used for the current attention calculation and then discarded.
K and V persist. They accumulate into the KV cache, allowing the model to reuse the state of everything it has already processed rather than recomputing it for every new token.
That works extremely well when inference remains on the same GPU or within a tightly connected cluster. But AI infrastructure is becoming increasingly distributed.
Inference workloads will move across GPUs, clusters, data centers, clouds and geographies based on availability, latency, cost, power and accelerator type.
And that creates a problem: The compute can move. But what happens to the KV cache?
Imagine an inference session starts in San Francisco. Capacity tightens, and the optimal compute for the next stage is in Dallas. Later, another part of the workflow would ideally execute in Virginia.
Moving the workload is relatively straightforward. Moving potentially large amounts of accumulated KV state across hundreds or thousands of miles is not.
So the orchestration problem becomes much more complicated.
- Do you move the KV cache with the workload?
- Replicate it proactively to another location?
- Maintain regional copies?
- Compress or transfer only the most valuable portions?
- Recompute the KV cache when that is cheaper than transmitting it?
- Or leave the KV cache where it is and route the compute toward the state?
Suddenly, “find the best available GPU” is no longer enough. The optimization equation becomes something closer to:
Compute + KV locality + network bandwidth + latency + congestion + transfer cost + replication cost + recomputation cost
And this could become increasingly important as context windows grow, reasoning becomes longer, and agentic applications maintain increasingly persistent state. It also means networking becomes part of the inference runtime.
The network can’t simply connect distributed GPU clusters. The control plane needs to understand the cost of moving inference state across that network—and potentially anticipate where that state will be needed next.
This is an important part of how we think about the Atlaris Network.
Atlaris is being designed to orchestrate inference across distributed infrastructure. Over time, that means optimizing not simply where compute runs, but the relationship between compute, network and state.
The optimal GPU may not be the cheapest or least congested GPU. It may be the GPU that produces the best overall economics after accounting for where the KV cache already lives and what it costs to move it.
The Bottom Line:
The ability to intelligently place, move and replicate that state across a distributed network could become one of the defining infrastructure problems of the inference economy.
