Sharing GPUs Without Flying Blind: Kubernetes Patterns for AI Inference

[Video transcript]

Thank you for joining us today for the “Sharing GPUs Without Flying Blind” webinar. My name is Daniela, and I’ll be the moderator for today’s session. I’m excited to introduce Julien, who is the Head of Kubernetes Engineering here at Kubex, who will lead us through today’s session.

Julien: Hey, everyone, thanks for today’s session. Hey, Daniela. Julien will take us through a few slides and then get right into a demo. And with that, I’m going to pass the mic over to Julien to get us started.

Julien: Thanks, Daniela. All right, so welcome everyone, and thanks for joining us today. Today we’ll be talking about GPU sharing, and the title says “Without Flying Blind” because that is the real challenge here. Once multiple workloads land on the same physical GPU, the old mental model of “one pod, one GPU” starts to break apart. And all the assumptions we have around how CPU and memory are shared on Linux also break down.

So we’ll start by understanding some of the GPU sharing technologies, such as time-slicing, MPS, and MIG. Then we’ll look into where open source Kubernetes-native projects fit in. One important distinction that we’ll keep coming back to is this: scheduling a fraction is not the same thing as enforcing a fraction of a GPU. By the end, the goal isn’t for everyone here to pick the same technology; the goal is for everyone here to understand the GPU sharing patterns and pick what is right for each use case.

Quick intro: I’m Julien, Head of Kubernetes Engineering here at Kubex. I lead the organization that builds automation for container- and GPU-backed infrastructure. I’m also a technical lead within the Developer Experience Technical Advisory Group at the CNCF. For anyone less familiar, the CNCF is the open source foundation behind many of the cloud-native projects we use every day within the Kubernetes ecosystem, including Kubernetes itself. In this session, we’ll touch on a few CNCF projects that are becoming relevant to GPU sharing in Kubernetes.

This slide here is our agenda for the session. We’ll stay practical and won’t get into a vendor feature comparison. Our biggest focus will be on what GPU sharing means on Kubernetes and how to make it work well. The key theme is that the workload should drive the decision. Latency, memory behavior, isolation requirements, and workload behavior while sharing a GPU matter more than a generic target utilization number. And once sharing is enabled, our job isn’t done: we need to know what the fraction really buys us, what it does and doesn’t guarantee, and what to watch so that sharing doesn’t become guesswork.

First, let’s understand why GPU sharing is important. Running LLM/AI workloads on Kubernetes focuses a lot on training jobs as well as running inference. Training is used to create the LLM and uses batch jobs: you size once and run to completion. They go as fast as they can, and giving them whole GPUs makes a ton of sense there. But inference, which is serving the LLM, isn’t like that. It’s more like a service: it behaves like a web server, where load follows how people actually use the product, so demand moves throughout the day.

We see it here: usage peaks in the middle of the day, but the reserve line at the top is flat, because we hand out whole GPUs. The space between those two lines is capacity that we pay for and don’t use. So one might ask, “If it’s so obvious, why are we handing out whole GPUs for those use cases?” The answer is that, unlike CPUs, the Linux kernel and Kubernetes have no native way to give a pod a slice of a GPU and hold it to that boundary. Sharing GPUs isn’t a core primitive of Kubernetes, and the mechanisms that do exist for sharing are vendor-specific: each vendor has several sharing options, and each has its own pros and cons.

Sharing has also been hard to configure and even harder to observe in Kubernetes, so a full GPU card is usually the safer choice. That was fine when capacity was cheap and easy to get. But with GPU inventory limited among cloud providers, and considering today’s prices, finding a way to be more efficient with the GPUs we have has become even more critical than it was. So fixing the gap in this chart is the problem the rest of the session is about.

In Kubernetes, a GPU fraction is not a supported unit. Many solutions exist to create those fractions, and they carry very different guarantees depending on the mechanism you choose. So “a quarter GPU” might mean one of several things:

• Access to the GPU’s compute one quarter of the time.
• The scheduler in Kubernetes accounting for a value that says this workload should fit beside others because it uses 25% of the GPU, and there is enough remaining shared capacity on the GPU for it.
• 25% of the GPU memory reserved to the pod, enforced by some software.
• A hardware partition with strong isolation that is roughly 25% of the GPU.

All of those are very different operational promises. So when someone asks, “Can I give this workload 25% of a GPU?” the better question is, “What do you need 25% to mean?” Is it throughput? Is it memory? Does it need isolation? Do you need best-effort or guaranteed performance? And what matters most: cost or performance?

This slide is the mental model for the rest of the talk. The important distinction is that there are two separate jobs to accomplish. First, orchestration: the Kubernetes-side decision. Where does the workload land? Which queue or team does it belong to? Does it need gang scheduling? What priority does it have? And how much GPU capacity should we account for? Second, fractioning: the runtime or hardware-side decision. Once the workload is on the card, what actually prevents it from using more memory, more compute, or more isolation than intended?

A scheduler can make a placement decision: it can say a pod should get a fraction and should share the GPU with other pods, but that doesn’t mean it automatically enforces that fraction. This is where the bottom line on this slide matters: a scheduled fraction is not an enforced fraction, and a decision must be made on how to enforce it. There are more options than the ones we’ll see here, and the same principles still apply if you use a different scheduler, device plugin, or vendor-specific mechanism. Today we’ll focus on the NVIDIA mechanisms that allow for fractioning.

First, keep in mind that from an orchestration and Kubernetes scheduling point of view, the GPU is modeled as a resource that’s allocated as a whole. It’s not a fully managed resource like CPU or memory, and that’s why various technologies exist to perform the sharing.

Let’s compare the three primary ways of sharing an NVIDIA GPU. A useful way to think about them is as three ways of sharing a kitchen among several cooks:

• Time-slicing: several cooks share one single stove. They take turns using it, and nothing prevents a cook from using the stove longer than they should, or from taking over all the burners for themselves; nothing controls that. Translating that to GPUs: a workload gets access to the GPU, but that doesn’t guarantee a fixed percentage of compute or memory. It may use more, and nothing prevents it from doing so.
• MIG goes to the opposite end and divides the kitchen into completely separate stations, like a cooking competition where each participant gets their own dedicated station in the big kitchen. Translating that to GPUs: each workload receives dedicated compute and memory resources that others cannot consume. The big trade-off is that station sizes are fixed before the workloads arrive, and changing the size of those stations isn’t trivial. Specifically for MIG, we need to pre-decide how to partition the device; if the partitions need to change in size, the device must be reconfigured and then repartitioned.
• MPS lives somewhere in the middle. The cooks are coordinated through a kitchen manager: they all work within the same kitchen at the same time, each on their own burner, but the kitchen manager keeps everyone in line. Everything becomes much more organized, but every action has to go through the kitchen manager; if the manager isn’t there anymore, everything stops working. Translating that to GPUs: multiple workloads talk to an MPS control daemon that coordinates who is using the GPU and how much memory and compute each workload can use.

Overall, the key distinction is what is being shared and where the boundary is enforced. Time-slicing shares access with little to no controls. MPS shares the GPU concurrently with software-level controls via the MPS control daemon. MIG creates hardware-isolated partitions.

Let’s look at an overview of what each sharing mechanism gives us. Time-slicing gives multiple workloads access to a GPU with no compute or memory reservation. MPS adds software control to ensure fairness. MIG makes it a hardware boundary. The key trade-off is flexibility versus predictability: you choose based on the guarantee the workload can’t afford to lose.

Time-slicing is the easiest baseline and doesn’t require specialized GPU hardware, but it has many limitations. Let’s look at how we can add workload-aware sizing and enforcement around it to make it better.

The problem is where the number lives. The most common approach for NVIDIA time-slicing in Kubernetes is to configure the split on the node through the NVIDIA device plugin. The workload still asks Kubernetes for one GPU; it can’t say “I need 0.30 or 0.45” of a GPU. It only lands on a node whose GPUs have already been divided into equal replicas. So the key change we want here is in where the fraction is defined. With basic time-slicing, the cluster administrator chooses a fixed number of replicas for every GPU. Workloads request one or more of these slices, but they’re still choosing from a cluster-wide configuration. Changing the size means the plugin has to be completely reconfigured and rolled out across the nodes.

The KAI Scheduler, a CNCF project, moves that decision into the workload. One pod can ask for 0.22 of a GPU, another for 0.38 of a GPU, and KAI can place them together as long as the total fits. That gives much finer-grained packing without pre-partitioning every GPU. Under the hood, Kubernetes still understands whole-GPU resources: KAI uses a reservation pod to claim the physical GPU from the default scheduler and then keeps its own accounting for the workloads sharing it. One important caveat: this fraction is an allocation and scheduling unit, and it doesn’t automatically guarantee the same percentage of compute. Flexible allocation is useful, but we still need something to enforce the boundary at runtime, which we’ll look at next.

This is one reference architecture, not the only correct one. Here, KAI handles the orchestration side: it decides placement, applies queues and quotas, respects priority, and tracks how much of the GPU each workload has requested. At admission time, KAI integrates with HAMi-core to inject a C library that ensures the container stays within its allocated memory fraction. This combination uses HAMi-core, not the full HAMi scheduling platform (also a CNCF project), so you could replace parts of this architecture depending on the workload. MPS could provide per-client memory and compute control. MIG is the better fit when the boundary must exist in hardware. HAMi can also provide its own scheduler instead of using KAI.

But the combination here goes on top of time-slicing and provides memory limits safely, so pods don’t go beyond their own memory fraction. It doesn’t limit the compute fraction of the GPU, which is something we’ll look at solving outside of KAI and HAMi-core. Overall, the idea is separation of responsibility: one layer places and accounts for workloads, that’s KAI. Another layer enforces, that’s HAMi-core.

Device-level utilization was often enough when one pod owned the entire GPU. But once several workloads start sharing the card, it stops telling the full story. A GPU being used at 90% can be a good thing or a bad thing when it’s shared. If two pods were scheduled with fractions that sum to 50%, but utilization is at 90%, that means the scheduler risks adding more pods onto that GPU and creating contention.

That’s where our Kubex exporter comes in. It looks at GPU utilization at the process level and correlates it to the container and the pod, so we can continue to track resource utilization at the pod level accurately. You can currently use it for free, and it will also be open source soon. It plugs into your existing Prometheus deployment to give you this added visibility. Another important layer is ensuring you have the appropriate application metrics; these complement the resource utilization metrics to help you understand any performance impacts when you start sharing GPUs.

The fraction we choose at admission time is really just a prediction about future demand. Even a good initial estimate will eventually become wrong as traffic patterns, batch sizes, and other factors change. Fraction management needs to happen in a control loop rather than as a one-off decision: we observe the workload, establish the baseline, choose a safe target, add headroom, and continue watching both GPU demand and application health. When the evidence is strong enough, we can start upsizing or downsizing to rebalance resources for the workloads so it stays representative of real usage. The loop itself is straightforward; the guardrails are what make it safe. In practice, we usually want to grow allocations quickly when an application is at risk, and shrink them more cautiously.

Now let’s look at how this works in our demo. Here we’re in the Kubex UI. We see our overall GPU utilization, and then we’ll dive into one specific workload.

Let’s look at one of our vLLM containers: first, one we haven’t yet optimized. Here we have a vLLM pod serving inference, and we can see the current average compute utilization of the GPU for that container sits at roughly 6%, peaking at around 27%. GPU memory is even lower, averaging about 3.5% and peaking at 17%. So there’s a clear underuse of the GPU here, and a clear opportunity for optimization using GPU sharing.

What we’re doing here is recommending a fraction of 0.375. Using the exact same GPU, just fractioning it, we can bring average utilization up to 15.7%, peaking at 72%. We’re still staying safe and keeping headroom, but we’re getting much better GPU utilization. Memory utilization becomes more efficient too, though we have to round up to whichever resource (compute, in this case) we use the most.

Looking into the details, we see we’re already using the KAI Scheduler, which is good. The only change is going from one whole GPU to a fraction of a GPU, which also results in less memory being used. That means we can go from a monthly cost of about $1,100 to $439.

Now let’s look at a container that has already been optimized. We see the same container here, but this one has been using GPU sharing. If we look at its utilization pattern, the orange line at the top represents the GPU allocation. It was previously using one whole GPU. We enabled fractioning; resource utilization was even lower than it is now, so we were able to size it down to 0.25 of a GPU. That’s where the control loop becomes very important: we continuously monitor GPU resource usage to make sure the fraction stays representative of actual use. The loop detects when GPU usage starts to rise and upsizes the fraction so there’s enough headroom, avoiding contention from going beyond the allocated fraction. We see it jump here to 0.375. By using this control loop, we ensure we’re always getting an accurate GPU fraction, which is important for operationalizing this safely.

Here’s our final capability blueprint, showing what vanilla NVIDIA time-slicing gives us in Kubernetes versus what we get by adding KAI, HAMi-core, and the Kubex solution into the mix. We haven’t reached a perfect solution that fits every use case, but we’ve overcome multiple time-slicing limitations, including: flexible, Kubernetes-native scheduling using KAI; a hard memory runtime cap for workloads using HAMi-core; per-workload resource usage visibility using the Kubex GPU exporter; and active adaptation for sizing GPU fractions and ensuring fairness using the Kubex platform.

In the end, you choose a mechanism based on your workload’s requirements and the sharing technologies available in your GPU and Kubernetes environment. If you’re interested, Kubex offers free trials, so you can try this for yourself and see how it behaves in your environment. We’ll stop here and take any questions you may have. Thank you very much.

Daniela: Thanks, Julien. I do see a couple of questions. The first one: how do you stop an automated right-sizing loop from reacting to every short spike?

Julien: That’s important. The continuous loop is monitoring resource usage, and it’s important to react to spikes that risk creating contention on the GPU. Memory is the critical point there: with HAMi-core, we limit it so we’re not creating OOMs. If compute goes a bit beyond its allocated fraction, that can be fine. Generally, the goal is to upsize fast and downsize slowly. But for compute, we can actually afford to upsize slowly too; in Kubex, we have policies that govern how fast or slowly you upsize and downsize, so you stay in control of your workload.

Daniela: Perfect, thank you. I’ve got two more questions. The next one: does it work with all NVIDIA series?

Julien: That’s one of the key points of time-slicing with KAI and HAMi-core: with MIG, you’re constrained to newer, more expensive hardware. Time-slicing is supported on most commodity hardware from NVIDIA, and almost all recent NVIDIA chips. Even Tesla T4s, which are fairly cheap and available, support time-slicing, so it’s a good way to reuse existing hardware without having to buy new hardware for MIG partitioning. You don’t have to roll out an MPS control daemon either. Time-slicing gives you a solution for your existing hardware, but you have to add these layers to make it safe, versus vanilla time-slicing, which has all the limitations we talked about.

Daniela: Perfect, thank you. One more: what happens when a workload needs more GPU capacity, but there’s no free space on its current card?

Julien: What we do in our solution is, whenever we’re upsizing, we try to do it in place first, if it fits. If it doesn’t fit, we evict the pod and it gets rescheduled onto another node with available GPU capacity. If there’s no available GPU, it can auto-scale up. So we do it in place if it fits, so the scheduler can take that fraction into account, but otherwise evict so it’s placed on another GPU.

Daniela: Thank you, those are all the questions I’m seeing. Thank you, Julien, for taking us through all this content, and thank you everyone for joining us today. Have a great rest of your day.

Julien: Thank you, everyone.