Introduction
The default Kubernetes scheduler was built for stateless services: web servers, APIs, databases. It schedules a pod, checks that a node has enough of whatever resources were requested, and binds it. For CPU and memory, that model works fine. For GPUs, it falls apart in three specific ways.
First, GPUs are treated as an opaque integer resource. A pod can request nvidia.com/gpu: 1, but the scheduler has no concept of a fraction of a GPU, no awareness of which GPU model it is, and no way to express “give me a device with at least 40GB of memory” or “give me two GPUs on the same NVLink domain.”
Second, there is no concept of a group of pods that must be scheduled together. A distributed training job needing eight GPU workers can end up with seven pods running and one stuck pending, with the other seven sitting idle burning expensive GPU-hours while waiting for a peer that may never get scheduled.
Third, there is no organizational fairness model. Nothing stops one team from monopolizing a shared GPU pool, and nothing lets an administrator define quotas, priorities, or borrowing rules between teams competing for the same scarce hardware.
An entire ecosystem of schedulers, queueing layers, and GPU-sharing middleware has grown up specifically to close these three gaps. This article walks through the major options, explains what problem each one actually solves, looks at how a modern MLOps platform assembles them into a coherent stack, covers a recent and significant integration between two of the most important projects in this space, and closes with where Kubernetes itself is heading as some of this functionality moves into the core scheduler.
What a Modern MLOps Platform Looks Like
Before comparing schedulers, it helps to see where scheduling actually sits in the stack, because none of these tools operate in isolation. A reasonably complete MLOps platform on Kubernetes tends to have five distinct layers.
Hardware enablement is the bottom layer: the NVIDIA GPU Operator or an equivalent, installing drivers, the container toolkit, and either the legacy device plugin or the newer Dynamic Resource Allocation (DRA) driver, so Kubernetes can see GPUs as schedulable resources at all.
Scheduling and queueing sits above that: this is where Volcano, Kueue, Apache YuniKorn, and the NVIDIA KAI Scheduler live. Their job is deciding which pods get admitted, in what order, with what fairness guarantees, and whether a group of pods is treated as an atomic unit.
GPU sharing and isolation is a layer that can sit either inside or alongside the scheduler: NVIDIA’s MIG (hardware partitioning), NVIDIA GPU Operator time-slicing, and HAMi (software-enforced fractional sharing) all live here, deciding how many workloads can actually share one physical device and how strictly they’re kept from interfering with each other.
Workload orchestration is the layer data scientists and ML engineers interact with directly: training-job operators like Kubeflow’s Training Operator, workflow engines like Argo, or higher-level frameworks that submit PyTorchJob or TFJob custom resources, all of which ultimately produce pods that the scheduling layer places.
Serving and inference sits on top for the production-facing side: platforms like KServe or dedicated inference-serving stacks, which have their own autoscaling and routing concerns but still depend on the scheduling layer underneath to actually place their pods on suitable, fairly allocated hardware.
The schedulers this article focuses on live in that middle layer, and the honest summary of why so many exist is that no single one covers all three original gaps well. Some are built for gang scheduling and fairness. Some are built for fine-grained GPU sharing. A production MLOps platform typically combines at least two of them.
Gang Scheduling and Queueing: Volcano, Kueue, and Apache YuniKorn
These three solve overlapping but distinct problems: getting groups of pods scheduled atomically, and enforcing organizational fairness over who gets access to limited hardware.
Volcano is a CNCF-graduated batch scheduling system, in production use since 2019, that replaces the default scheduler for the workloads it targets. Its core strength is mature gang scheduling: pods belonging to a job are grouped into a PodGroup with a minimum member count, and none of them are bound to a node until enough resources exist to satisfy the whole group at once. It also supports queue-based fair-share scheduling with priority classes, and has broad native integration with distributed training frameworks such as PyTorch, TensorFlow, MPI, and Horovod. The tradeoff is real operational complexity — it introduces its own CRDs and a scheduler that fully replaces kube-scheduler for the workloads routed through it.
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
name: distributed-training-pg
spec:
minMember: 8
minResources:
cpu: "64"
memory: 256Gi
nvidia.com/gpu: "8"
queue: ml-team-queue
Kueue takes a lighter-weight, more Kubernetes-idiomatic approach. Rather than replacing the scheduler, it sits above it as a job-queueing layer, using standard admission webhooks and native Kubernetes APIs to manage quotas, admission order, and preemption. It supports quota borrowing across teams, so a team under its allocated share can temporarily use another team’s idle capacity. What it does not do natively is gang scheduling — Kueue decides when a job is admitted, but the actual atomic placement of that job’s pods still needs something else underneath it. This is why Kueue and Volcano are frequently deployed together: Kueue governs the organizational quota question, Volcano (or another gang-aware scheduler) governs the placement guarantee.
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: ml-team-cluster-queue
spec:
namespaceSelector: {}
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: gpu-flavor
resources:
- name: "nvidia.com/gpu"
nominalQuota: 32
Apache YuniKorn takes the third approach: a full scheduler replacement, like Volcano, but built around a hierarchical resource queue model that maps naturally onto organizational structures — departments, teams, individual projects — with fairness policies enforced at each level of that hierarchy. It supports both batch and long-running service workloads under one scheduler, and offers web-based visibility into queue state that administrators coming from HPC or Hadoop-style schedulers tend to find familiar. Some organizations run YuniKorn in place of Volcano specifically for that hierarchical queue model and its dashboard tooling, even though the underlying gang-scheduling guarantee it provides is functionally similar.
None of the three, on its own, addresses fine-grained GPU sharing within a single device. That is a separate problem, solved by a different set of tools.
GPU Sharing and Isolation: MIG, Time-Slicing, HAMi, and KAI Scheduler
Even with gang scheduling and fair queueing solved, a GPU is still, by default, an indivisible unit — one pod gets the whole device, or none of it. For small models, notebooks, or lightly-loaded inference workloads, that wastes enormous amounts of expensive hardware. Several mechanisms exist to divide a physical GPU across more than one workload, and they differ enormously in how strong a boundary they actually enforce.
Multi-Instance GPU (MIG) is a hardware-level partitioning feature available on newer NVIDIA datacenter GPUs. It physically divides a GPU into isolated instances, each with dedicated memory, compute cores, and cache — genuinely separate mini-GPUs from the perspective of workloads running on them. This gives hard isolation with predictable performance, at the cost of fixed partition shapes and availability only on specific GPU models.
Time slicing (GPUs), configured through the NVIDIA GPU Operator, lets the device plugin advertise multiple logical replicas of one physical GPU, and the driver context-switches between the processes sharing it. This works on nearly any NVIDIA GPU and requires no special hardware, but provides no memory or fault isolation whatsoever — one workload can still consume all the memory and starve its neighbours, and GPU kernels generally run to completion before yielding, so a long-running kernel can block everything else sharing that slice for its full duration.
Project HAMi is a CNCF Incubating project that takes a software-virtualization approach distinct from both of the above. It combines a custom device plugin, a scheduler extender for topology-aware placement, and a runtime component called HAMi-core that intercepts CUDA and NVML API calls inside the container. That interception is what makes HAMi’s isolation genuinely enforced rather than cooperative: a container’s nvidia-smi output is virtualized to show only its allocated slice, and memory-allocation calls that would exceed the declared cap are actually rejected at the driver-interception layer, not just tracked in a scheduler’s bookkeeping. HAMi supports slicing by memory, by compute core percentage, or by device count, and extends beyond NVIDIA GPUs to other accelerator families including AMD, and various domestic Chinese AI chips.
apiVersion: v1
kind: Pod
metadata:
name: shared-gpu-inference
spec:
schedulerName: hami-scheduler
containers:
- name: inference
image: some-inference-image:latest
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: "4000" # MiB, enforced by HAMi-core
nvidia.com/gpucores: "30" # percent of SM utilisation
NVIDIA KAI scheduler occupies a different position again. It grew out of Run:ai’s commercial scheduling engine — NVIDIA acquired Run:ai in late 2024 and open-sourced the scheduler under Apache 2.0 in April 2025, and it is now a CNCF Sandbox project. Its primary strength is AI-workload-aware scheduling behaviour: gang scheduling via PodGroups, hierarchical fair-share queues with borrowing and reclaim between teams, topology-aware placement that understands NVLink domains, and fractional GPU allocation. That last capability, however, was originally cooperative rather than enforced in the same way HAMi’s is — which is the subject of the next section.
The HAMi and KAI Scheduler Integration
In June 2026, two core pull requests merged HAMi’s isolation technology directly into the NVIDIA KAI Scheduler codebase, shipping as a built-in feature starting with KAI Scheduler v0.16.4. It’s worth understanding both why this happened and what it actually changed, because the two projects did not merge as organizations — this is a targeted technical integration between two projects that continue to exist independently.
The gap it closed was specific. KAI Scheduler’s original fractional GPU allocation was cooperative: the scheduler tracked the sum of requested memory shares against a GPU’s total capacity and would not overcommit that ledger, but nothing prevented an individual workload from actually using more memory than it declared. A container that requested a 2GB slice could still see the entire physical GPU through the CUDA API and nvidia-smi, and could, deliberately or accidentally, consume far more than its share — a real problem in multi-tenant production clusters, where the only historical workaround was asking every application to voluntarily self-limit its own memory usage in code.
The fix reuses HAMi’s existing enforcement mechanism rather than building a new one. NVIDIA’s own engineers chose to adopt HAMi-core directly: KAI Scheduler’s admission component now injects a memory-limit environment variable into containers that request shared GPU memory, a companion component ships the HAMi-core library to every GPU node as a DaemonSet, and a mutating webhook injects that library into the relevant pods at creation time. At runtime, HAMi-core intercepts CUDA memory-allocation calls and enforces the declared cap directly at the driver-interception layer — turning what was previously a scheduling-level accounting exercise into an actual, physically enforced boundary.
helm install kai-scheduler kai-scheduler/kai-scheduler `
--version 0.16.4 `
--set global.gpuSharing=true `
--set binder.plugins.hamicore.enabled=true
What is important, and what several project maintainers have been explicit about, is that the integration is deliberately loosely coupled. KAI Scheduler keeps its own scheduling logic entirely — gang scheduling, queue fairness, topology awareness — and only borrows HAMi-core for the isolation piece specifically. It does not adopt HAMi’s own scheduler or device plugin, which would conflict with KAI’s own. The two projects continue to have separate governance, separate release cadences, and separate primary missions.
That separation is also visible in how differently the two projects describe their own roadmaps. KAI Scheduler’s stated focus remains scheduling behaviour for AI workloads specifically — the batch and fairness problems described in the previous section. HAMi’s own public roadmap, by contrast, explicitly names deeper integration with other scheduling projects as an ongoing goal, listing KAI Scheduler, Koordinator, Kueue, and Volcano together as ecosystem partners it intends to keep working with, alongside expanding hardware support to more accelerator vendors. That framing — HAMi positioning itself as a shared isolation substrate multiple different schedulers can sit on top of, rather than a scheduler competing to win market share against them — is a meaningfully different long-term posture than a head-to-head rivalry.
The practical, durable takeaway for anyone building an MLOps platform: the two tools are converging on complementary roles rather than converging into one product. Expect KAI Scheduler to keep owning scheduling policy (queues, fairness, gang scheduling, priority) and HAMi to keep owning the enforcement mechanism underneath fractional allocation, whether that allocation was requested through KAI, Volcano, Koordinator, or any other scheduler HAMi chooses to integrate with next. Competition in the sense of “which one wins” is unlikely; what is more likely is HAMi-core becoming a de facto standard isolation layer that most GPU-sharing schedulers eventually plug into, the way most Linux distributions eventually converged on the same handful of underlying init systems regardless of which one shipped first.
What’s Converging into Native Kubernetes
Two of the capabilities described above — expressive, non-integer device requests, and gang scheduling — are in the process of moving from third-party add-ons into Kubernetes’ own core APIs. Neither is finished, and it’s worth being precise about how far each has actually progressed, because both have been the subject of some premature “problem solved” commentary.
Dynamic Resource Allocation
Dynamic Resource Allocation (DRA) replaces the old device-plugin model, where a GPU was an opaque integer resource, with a structured API: hardware vendors publish device inventories with real attributes (memory size, product family, topology position), administrators define DeviceClass objects describing categories of allowable hardware, and workloads express requirements as ResourceClaim objects — “a device from this class with at least this much memory” — rather than a bare count. The scheduler matches claims to actual devices at allocation time instead of relying on an opaque number.
DRA’s path through Kubernetes has been gradual: introduced as alpha in Kubernetes 1.26, substantially redesigned in 1.31, promoted to beta with the v1beta1 API in 1.32, updated to v1beta2 in 1.33, and the core API group graduated to General Availability in Kubernetes 1.34, released in August 2025. That GA milestone was described by the Kubernetes project itself as the headline feature of that release. Momentum has continued since: at KubeCon Europe in March 2026, both NVIDIA and Google donated their respective DRA drivers for GPUs and TPUs to the open-source community in the same week, a signal that major hardware vendors are converging on this as the standard integration point rather than maintaining proprietary scheduling extensions. Subsequent releases have layered on refinements — device health reporting surfaced directly in pod status, and share-identifier support that lets a driver distinguish between multiple consumers of the same shared or partitioned device.
It’s important to be precise about what DRA does and doesn’t solve on its own. DRA gives Kubernetes a much richer way to describe and request hardware, and it gives device drivers a standard integration point. It does not, by itself, create fractional GPU sharing, and it does not guarantee isolation between workloads sharing a device — that still depends on what the underlying driver actually does with a claim once it’s allocated. In practice, DRA is best understood as the plumbing layer that tools like MIG, time-slicing, and HAMi can now plug into more cleanly, rather than a replacement for any of them.
Gang Scheduling
Native gang scheduling has followed a similar, still-unfinished trajectory. Kubernetes 1.35, released in December 2025, introduced the foundational Workload API in alpha, alongside an initial PodGroup-based implementation of gang scheduling: pods belonging to a group are identified, and the scheduler holds all of them at the same stage of the scheduling and binding cycle until every pod in the group can proceed together, or releases all of them if a timeout is hit. This tracks the KEP-4671 proposal, which is explicit that its initial goal is framework support and the necessary building blocks, not a fully-featured gang-scheduling algorithm competing with what Volcano or YuniKorn already offer.
Kubernetes 1.36, released in April 2026, advanced this further with a real architectural change: separating the static policy (the Workload object, describing how a group of pods should be scheduled) from the runtime state (a new, first-class PodGroup object), and adding a dedicated PodGroup scheduling cycle to kube-scheduler to process a workload’s pods atomically. A related feature gate lets the built-in Job controller automatically generate the Workload and PodGroup objects for suitable jobs, rather than requiring an operator to hand-author them. Work continues into the following release cycle on workload-aware preemption — a parallel preemption path specifically for gang-scheduled groups, distinct from Kubernetes’ existing per-pod preemption logic — and on topology-aware scheduling constraints for pod groups.
apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata:
name: training-job-workers-pg
spec:
schedulingPolicy:
gang:
minCount: 8
---
apiVersion: v1
kind: Pod
metadata:
name: worker-0
spec:
schedulingGroup:
podGroupName: training-job-workers-pg
As of the most recent release at the time of writing, every one of these features — the Workload API, native gang scheduling, workload-aware preemption, and topology-aware group scheduling — remains alpha, disabled by default, and gated behind explicit feature flags on both kube-apiserver and kube-scheduler. Kubernetes’ own documentation on the subject is direct about this: if gang scheduling is needed in production today, established tools like Volcano and Kueue carry a genuine maturity advantage, and the native primitives represent an important long-term direction rather than a present-day replacement.
The realistic timeline
Kubernetes ships three releases a year on a roughly four-month cadence. Given that native gang scheduling only reached alpha in the December 2025 release and underwent a significant architectural revision in the very next release four months later, a conservative expectation is: continued alpha refinement through at least one more release cycle, a beta graduation sometime after the API has been judged stable — which the project has explicitly declined to commit to a specific version for — and GA realistically a year or more beyond that, following the same multi-release beta soak that DRA itself went through between 2025 and 2026. Organizational fairness, hierarchical queues, and quota borrowing across teams — the layer Kueue and YuniKorn currently own — are not part of any of these KEPs at all, and there is no indication the Kubernetes core scheduler intends to absorb that layer; it appears likely to remain ecosystem territory indefinitely, on the reasoning that Kubernetes aims to provide scheduling building blocks rather than a full organizational policy engine.
The practical implication for anyone building a platform today is straightforward: treat DRA as safe to adopt for expressing GPU requirements, since it’s GA and has real vendor driver support behind it, but continue relying on Volcano, Kueue, YuniKorn, or KAI Scheduler for gang scheduling and fairness for the foreseeable future, and revisit that decision only once native gang scheduling has spent a release or two in beta with real production usage behind it.
Choosing a Stack
There’s no single right answer, because the right combination depends on which of the three original gaps — GPU expressiveness, gang scheduling, and organizational fairness — matters most for a given cluster, and how strict the isolation requirement is.
For a cluster mainly running distributed training with a small number of trusted teams, Volcano alone, paired with the NVIDIA GPU Operator for basic device enablement, covers most of the need: gang scheduling and reasonable queue fairness without introducing a second scheduling layer.
For a larger, multi-tenant organization where quota governance across departments matters more than gang-scheduling sophistication, Kueue as the admission and quota layer, with Volcano or YuniKorn underneath for actual placement, is a common and well-documented pairing.
For a cluster running many small inference workloads that don’t need a full GPU each, fine-grained sharing is the priority, and the choice is really between MIG where hardware-enforced isolation is worth the fixed partition shapes, HAMi where flexible, enforced fractional sharing across ordinary GPUs is needed, or KAI Scheduler (now with the HAMi-core integration) where AI-specific scheduling policy and enforced sharing are both wanted from a single, NVIDIA-native tool.
For a platform being built with a multi-year horizon, adopting DRA now for how workloads describe their hardware requirements is a safe, forward-compatible choice, while treating native gang scheduling as something to watch rather than something to depend on until it has matured well past its current alpha state.
Most real production MLOps platforms end up running two or three of these tools together rather than one, precisely because each one was built to solve a different piece of a problem that the default Kubernetes scheduler was never designed to handle in the first place.
Connect with me on Medium @david.b.chase
