The particular combination of VMware Cloud Foundation (VCF), VMware vSphere Kubernetes Service (VKS), NVIDIA Run:ai, and NVIDIA Dynamo is not written down anywhere. So we wrote it down, ran a 120-billion-parameter Nemotron on it both ways, and found something we were not looking for.
Every enterprise looking seriously at LLM infrastructure eventually reaches the same fork. Serve your model the simple way, one replica doing everything, or pull it apart: a pool of GPUs that reads prompts, a separate pool that generates tokens, a transfer fabric shuttling the KV cache between them. Disaggregation promises a better fit between the hardware and the work. It also promises more moving parts.
We wanted to know whether that trade is worth making. Not in the abstract, and not on someone else’s benchmark: on eight H100s, on a governed on-premises platform, with a model large enough to be worth the trouble.
The first thing we found had nothing to do with the answer. Every component here is well-documented on its own. Nobody has published the combination. So before measuring anything, we wrote the recipe, and that recipe is now the first thing the paper hands you.
The stack composes cleanly. Someone just had to write it down.
Layer by layer: VC 9.1 with vSAN and NSX, the Supervisor and a vSphere Kubernetes Service cluster, the NVIDIA GPU Operator, NVIDIA Run:ai, then Dynamo’s control plane and its prefill and decode worker pools. What stands out in retrospect is how little of that needed special handling. VKS is CNCF-conformant and it shows: the GPU Operator, NVIDIA Run:ai’s scheduler, and Dynamo’s Kubernetes deployment path all ran exactly as their upstream documentation describes, with no platform-specific workarounds anywhere in our harness. Across days of standing the cluster back up in a different serving shape, vSphere and vSAN never lost a volume or dropped a path, and NVIDIA Run:ai’s project-scoped GPU quota is why every redeploy got its GPUs back cleanly.
The real decisions are about how you model a topology. The most useful one: a topology is not a single job. It is a coupled set (frontend router, prefill workers, decode workers, etcd, and NATS) that comes up together and switches atomically when you change shape. NVIDIA Run:ai schedules both CLI-submitted workloads and standard Kubernetes workloads submitted directly, and for a coupled set the second path is clearly the right one. We render each worker as a Deployment and apply the set into the NVIDIA Run:ai project namespace, so the shape lives in version-controlled YAML rather than shell history. That is what makes an unattended, multi-shape campaign reproducible.
Also in the build: why these workers get whole GPUs rather than partitions, and one load-path detail worth knowing. A port-forward is a debugging convenience, not a load path, and pushing real concurrency through one will distort your latency numbers.
NVIDIA Nemotron-3-Super is genuinely new ground
Nemotron-3-Super-120B-A12B is the model we wanted: roughly 120 billion parameters, genuinely capable, not chosen for being easy to serve. It is also a Mamba/Attention hybrid, and that is where the novel work was. Disaggregation guides assume a pure-attention transformer. NVIDIA publishes a disaggregated recipe for the larger Nemotron-3-Ultra; for this one we found nothing to check against.
Running it split took five specifics that are not in any guide we could locate, all of them in the serving runtime and the KV transfer rather than the platform underneath, each root-caused out of the runtime’s own error text before we applied it. The paper gives all five, so the next person does not have to derive them.
Two of them are worth more than a list entry. Resolving those meant switching off, permanently, capabilities the monolithic baseline keeps, and they accelerate precisely the phase where disaggregation later turns out to be weakest. Which left us a choice: hold the baseline back so the comparison would be symmetric, or let the disaggregated shape run at a disadvantage and say so loudly. We chose the disadvantage, because holding back a feature the monolith can legitimately use would have measured a configuration nobody would deploy. It does mean every number is tilted toward the monolith before a single request is sent, and we flag that wherever the two appear side by side.
Traffic that strives to be realistic (and the completion length trap)
By now we had a working cluster and two serving topologies, and it was time to test them using realistic workloads. This is the part that many benchmarks skip, and it is largely what our two previous papers, SPOC and the Envoy AI Gateway study, were about. We developed our own custom load generator based on traffic-modeling research and our internal user characterization to measure realistic, SLO-bounded capacity.
When we compared our approach to the benchmarking tools provided in NVIDIA’s open-source Dynamo repository (which uses AIPerf to replay static datasets like the Mooncake trace), we found a stark difference in what gets exposed. AIPerf is effective at measuring raw engine throughput, but throughput on a static trace measures a different reality than the one enterprise operators face.
Our approach differs in two major ways. First, we don’t rely on a single average traffic pattern. Instead, we blend four distinct, stochastic workload profiles simultaneously: autonomous agent bursts, heavy CI/CD log dumps, rapid UI navigation, and long-context human chat. Real users do not arrive on a convenient Poisson schedule. Agent swarms burst. CI pipelines dump. Humans pause to read. So we drive four profiles with four different arrival processes, the bursty ones grounded in Paxson and Floyd’s foundational result on wide-area traffic.
Second, and crucially, we push the decoders much harder on completion length. In standard benchmark traces, requests often average fewer than 1,000 output tokens. At that length, a disaggregated setup shines because the decode GPUs free up quickly. But in our harness, we cap generation at 8,192 tokens to reflect the heavy reality of long-form agentic coding fixes and knowledge-worker reports—up to 10x the generation length of a standard trace.
Token-wise completion length changes the physics of the cluster. When a request needs 8,000 tokens, it locks up decode GPUs for tens of seconds. In a fixed 4-prefill / 4-decode disaggregated split, the decode pool saturates rapidly while the prefill GPUs sit mostly idle. A monolithic deployment, by contrast, dynamically shares all 8 GPUs across both tasks, absorbing the long-generation penalty much more gracefully.
What surprised us was how little those profiles have in common once they reach the GPU: two of them differ by more than two orders of magnitude in the context they carry, so sizing on their average puts you wrong for both at once. And one thing worth knowing before trusting any published serving number, ours included. Drive a benchmark from a small prompt set and vLLM’s prefix cache hands you a beautiful result that means nothing, because you have cached away the exact cost a disaggregation comparison exists to measure.
What we actually found
We went in expecting to learn which architecture was faster. That is not what we learned.
Neither shape cleared the strict per-profile latency ceilings we had set. We could have loosened them until one passed. We would rather tell you what happened, because the useful result was never how much load each one carried. It was how each behaves once pushed past what it can sustain.
The monolith carries far more. It is also the one that goes without warning. Time-to-first-token sits at roughly two seconds, and one ramp step later it is 43.6 seconds. No gradual decline, no band of “getting worse” in which an operator might notice and react. Just a wall.
The disaggregated shape never reaches a wall at all. It gets stopped much earlier, while still perfectly healthy, by a gate that fires before anything has actually gone wrong. Why it behaves that way turns out to have far less to do with the transfer fabric everyone worries about than with which pool owns which GPUs. That mechanism, rather than the headline numbers, is what an operator should walk away with.
And about those headline numbers. Put the two shapes’ governing concurrencies next to each other and they look like a twentyfold capacity gap. They are nothing of the kind, and that misreading is the easiest way to get this result wrong. We spend a section on how to read them properly, then compress the comparison into a one-page table: given your traffic and your tolerance for how a system fails, here is which shape to choose.
Read the Whitepaper
Two limits, stated there and here. Each shape was measured once, so these are point estimates with no error bars. And we compared each architecture’s best realistic deployment rather than a flag-identical control, for the reason above. We would rather report an honest asymmetry than round it into a winner.
If you are standing at the same fork, read Disaggregated vs. Monolithic LLM Serving on VMware Cloud Foundation, with NVIDIA Run:ai and NVIDIA Dynamo. And if you take the method to your own models and traffic, we would genuinely like to hear how it goes.
Want to know more?
- Complete this form to contact us!
- Visit VMware.com/AIML for more information.
- Connect with us on Twitter at @VMwareVCF and on LinkedIn at VMware VCF.
Discover more from VMware Cloud Foundation (VCF) Blog
Subscribe to get the latest posts sent to your email.
