Introduction
Serving large language models at scale is fundamentally a distributed systems problem. A single GPU, or even a single node, is rarely enough once you need multiple models, multiple replicas, tensor-parallel sharding across GPUs, or high-availability rollouts. Kubernetes solves general container orchestration well, but it has no native concept of a GPU-aware, actor-based compute cluster. Ray fills that gap, and when combined with vLLM it has become one of the dominant patterns for serving LLMs in production.
This article explains what Ray is, how it schedules and manages work, how it handles GPUs and sharing, how a Ray-orchestrated deployment differs from a plain Kubernetes deployment, and exactly how vLLM plugs into Ray’s execution model. Code examples are included throughout so you can see the shape of a real deployment, not just the theory.
What Ray Is
Ray is an open-source distributed computing framework, originally developed at UC Berkeley’s RISELab, built to let Python code scale from a laptop to a cluster of thousands of nodes with minimal code changes. It was not designed specifically for LLMs — it started as a general-purpose framework for machine learning workloads such as reinforcement learning and hyperparameter tuning — but its execution model turns out to be a good fit for LLM serving as well.
Ray’s architecture has a few core pieces worth understanding before anything else makes sense:
- Driver: the process running your user program, the one that submits work.
- Worker: a process that actually executes tasks or hosts actors.
- Raylet: a per-node daemon that handles local scheduling and object management. Every Ray node, head or worker, runs one.
- GCS (Global Control Store): the cluster-wide metadata service, running on the head node, that tracks actor registries, node membership, and scheduling state.
- Object store: a distributed, shared-memory store that allows zero-copy data passing between tasks on the same node and efficient transfer across nodes.
On top of this core runtime, Ray provides two fundamental units of work:
Tasks are stateless remote functions. You call a normal Python function, decorate it with @ray.remote, and Ray runs it asynchronously somewhere in the cluster, returning a future.
Actors are stateful, long-running worker processes. Unlike a task, an actor keeps instance variables alive across calls, executes its methods sequentially by default, and lives for as long as you keep it around. This is the piece that matters most for LLM serving, because a loaded model with its weights resident in GPU memory is exactly the kind of long-lived state an actor is built to hold.
import ray
ray.init()
@ray.remote(num_gpus=1)
class ModelWorker:
def __init__(self, model_path):
self.model = load_model(model_path) # loaded once, stays resident
def generate(self, prompt):
return self.model.generate(prompt)
worker = ModelWorker.remote("some/model")
result = ray.get(worker.generate.remote("Hello, world"))
That pattern — an actor that loads a model once and serves many requests against it — is the foundation everything else in this article builds on.
How Ray Manages and Schedules Work
Ray treats CPU, GPU, and memory as logical resources. When a Ray node starts, it auto-detects the physical resources available (or you specify them explicitly) and advertises that quantity to the GCS. When you submit a task or create an actor with a resource requirement like num_gpus=1, Ray’s scheduler looks for a node with enough declared free capacity and places the work there.
An important and frequently misunderstood point: these logical resources are accounting constructs, not physical guarantees. Ray does not pin a task to a physical CPU core the way an operating system scheduler might, and — critically for GPU workloads — Ray does not enforce that a task or actor actually stays within its declared resource footprint. If you ask for num_gpus=0.5 and your process uses more memory than that implies, Ray will not stop you. It simply will not schedule more declared fractions onto that device than fit within 1.0 per GPU. Enforcement of actual usage is left to the application.
Ray does provide one concrete form of GPU isolation: it automatically sets the CUDA_VISIBLE_DEVICES environment variable for each task or actor, which most ML frameworks, including PyTorch and vLLM, respect. That prevents accidental cross-talk about which physical device an actor should use, even though it does not enforce memory limits.
Placement groups are the mechanism Ray uses for more deliberate control over where actors land relative to each other. Rather than letting individual actors get scheduled wherever there is room, a placement group reserves a bundle of resources (for example, “4 GPUs, ideally on the same node”) up front, and actors are then created inside specific bundles of that group. This is exactly the mechanism used to keep the workers of a single tensor-parallel model together, or to deliberately spread them across nodes when a model does not fit on one.
Ray on Kubernetes: KubeRay
Ray can run standalone on bare VMs, but running it inside Kubernetes is the common production pattern, and that is handled by KubeRay.
KubeRay is an open-source Kubernetes operator that translates Ray’s cluster concepts into native Kubernetes custom resources. It does not replace Ray — it is purely the deployment and lifecycle layer that lets Kubernetes stand up, scale, and tear down Ray clusters declaratively, the same way any other operator manages a stateful application.
KubeRay defines three primary custom resource definitions:
RayCluster represents a full Ray cluster: one head pod plus one or more groups of worker pods. KubeRay manages the entire lifecycle, including creation, deletion, autoscaling, and fault tolerance.
RayJob is for batch or one-off workloads. It creates a RayCluster, submits a job to it once the cluster is ready, and can optionally tear the cluster down automatically when the job finishes.
RayService combines a RayCluster with a Ray Serve application definition. This is the resource you use for long-running inference serving, and it supports zero-downtime upgrades and health-checked high availability of the Serve layer.
Installing KubeRay is a Helm operation:
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
# Installs both the CRDs and the operator
helm install kuberay-operator kuberay/kuberay-operator --version 1.4.0
Once the operator is running, you check it the same way as any other pod:
kubectl get pods
# NAME READY STATUS RESTARTS AGE
# kuberay-operator-6b68b5b49d-jppm7 1/1 Running 0 2m
A minimal RayCluster manifest looks like this:
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: llm-raycluster
spec:
rayVersion: "2.56.0"
headGroupSpec:
rayStartParams: {}
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.56.0-py311-gpu
resources:
limits:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
workerGroupSpecs:
- groupName: gpu-workers
replicas: 2
minReplicas: 1
maxReplicas: 4
rayStartParams: {}
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.56.0-py311-gpu
resources:
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
Apply it and watch KubeRay reconcile the actual pods:
kubectl apply -f raycluster.yaml
kubectl get pods
# NAME READY STATUS AGE
# kuberay-operator-6b68b5b49d-jppm7 1/1 Running 10m
# llm-raycluster-head-gw958 0/1 ContainerCreating 30s
# llm-raycluster-gpu-workers-worker-wl7k2 0/1 Init:0/1 30s
Note the naming pattern: pods are named after the RayCluster and their role (head or worker group), not after any application running inside them. That naming convention is a visible symptom of a deeper architectural difference, which is the next topic.
Standalone vLLM Pods vs Ray-Orchestrated Pods
If you deployed vLLM the plain Kubernetes way, you would typically write a Deployment where each pod runs a single vllm serve <model> process, fronted by a Service and probably an HPA for scaling replica count.
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-mistral-7b
spec:
replicas: 2
selector:
matchLabels:
app: vllm-mistral-7b
template:
metadata:
labels:
app: vllm-mistral-7b
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- "--model=mistralai/Mistral-7B-Instruct-v0.3"
- "--tensor-parallel-size=1"
resources:
limits:
nvidia.com/gpu: "1"
ports:
- containerPort: 8000
In this model, each pod is a self-contained vLLM process. Scaling means Kubernetes creating more identical pods. There is no coordination between them beyond what your load balancer provides, and if a model needs tensor parallelism across multiple GPUs, you either need multi-GPU pods with manual multi-process coordination, or a different tool entirely, because a bare Kubernetes Deployment has no concept of jointly-scheduled, communicating replicas.
Running the same model through Ray changes the picture substantially. A RayService spec does not describe a container command; it describes a Ray Serve application, with the vLLM engine configuration as data inside it:
apiVersion: ray.io/v1
kind: RayService
metadata:
name: vllm-mistral-7b
spec:
serveConfigV2: |
applications:
- name: mistral-7b
route_prefix: /
import_path: ray.serve.llm:build_openai_app
args:
llm_configs:
- model_loading_config:
model_id: mistral-7b
model_source: mistralai/Mistral-7B-Instruct-v0.3
engine_kwargs:
tensor_parallel_size: 1
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 4
rayClusterConfig:
rayVersion: "2.56.0"
headGroupSpec:
rayStartParams: {}
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.56.0-py311-gpu
resources:
limits:
nvidia.com/gpu: "1"
workerGroupSpecs:
- groupName: gpu-workers
replicas: 1
minReplicas: 1
maxReplicas: 4
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.56.0-py311-gpu
resources:
limits:
nvidia.com/gpu: "1"
Deploy it, and what shows up under kubectl get pods is head and worker pods named after the RayCluster, not the model:
kubectl apply -f rayservice.yaml
kubectl get pods
# NAME READY STATUS AGE
# vllm-mistral-7b-raycluster-c9wk4-head-gw958 1/1 Running 4m
# vllm-mistral-7b-raycluster-c9wk4-worker-gpu-l7k2 1/1 Running 4m
This is more than a naming convention. Each of those pods is a Ray node first: it runs a Raylet, it is part of the Ray cluster’s GCS-tracked membership, and it can host any number of Ray actors, of which the vLLM engine is only one kind. Scaling a model up in this world usually means more Ray actors joining an existing worker pod (if there is spare GPU capacity) or more worker pods joining the RayCluster — not a 1:1 relationship between “one more replica” and “one more independently-named pod.”
The practical differences worth internalizing:
Standalone Kubernetes: one process per pod, Kubernetes-native scaling and scheduling, simple mental model, but no built-in mechanism for coordinating multi-node tensor parallelism, no shared placement logic across models, and no cross-model resource sharing.
Ray-orchestrated: pods are generic compute units in a cluster that Ray subdivides among actors. This enables cross-node tensor and pipeline parallelism, multiple models or replicas sharing a pod’s GPUs, prefix-aware and session-aware routing, and prefill/decode disaggregation. The tradeoff is an extra layer of orchestration (Ray itself) sitting on top of Kubernetes, with its own head node, GCS, and failure modes to understand.
Ray Actors and the RayService CRD
It is worth being precise about how a Ray actor relates to the RayService custom resource, because the two live at completely different layers and are easy to conflate.
RayService is a Kubernetes object. It is declarative state stored in etcd, reconciled by the KubeRay operator’s control loop, and visible to every standard Kubernetes tool: kubectl get rayservice, kubectl describe, and so on. What RayService actually owns, in Kubernetes terms, is a RayCluster (the head and worker pods) plus a Serve application config to run on top of it.
A Ray actor is not a Kubernetes object at all. It is a runtime construct that exists only inside a live Ray cluster’s process model, tracked by the GCS on the head pod. There is no CRD for an actor, no etcd record, and kubectl has no way to list them, because as far as Kubernetes is concerned an actor is just ordinary memory and threads inside an already-running container.
The chain of ownership between the two looks like this:
- You apply a RayService manifest. KubeRay’s controller reconciles it into a RayCluster, which in turn produces head and worker pods, exactly as described earlier.
- Once the cluster’s pods are running, KubeRay submits the
serveConfigV2application definition to that cluster’s Ray Serve controller, over the head pod’s dashboard/agent endpoint. - The Serve controller — itself a long-running Ray actor — reads that config and is responsible for creating, scaling, and tearing down the deployment replica actors it describes. For a vLLM-backed deployment, that means the vLLM engine actors (and, for tensor-parallel models, the
RayWorkerWrapperactors covered later in this article) come into existence as a direct consequence of the Serve controller’s own decisions, not as a direct action by KubeRay or by Kubernetes. - KubeRay does not track individual actors. It periodically polls the Serve controller’s status endpoint to ask whether the application is healthy and how many replicas are running, and uses that answer for two things: reporting RayService status back into the CRD’s
statusfield, and driving zero-downtime upgrades, where a new RayCluster is stood up, its actors are confirmed healthy, and only then does traffic cut over from the old cluster to the new one.
So RayService describes what should exist and where it should run; the actors it results in are created, supervised, and destroyed dynamically by Ray’s own control plane running inside the pods that RayService provisioned. Kubernetes-native visibility stops at the pod boundary — to see the actors themselves you need Ray-native tooling, either the Ray dashboard or the CLI, run against the head pod:
kubectl exec -it qwen-2-5-7b-raycluster-mbcpn-head-qrznn -- ray list actors
Actor ID Class name State Resources
a1b2c3... ServeController LIVE {}
d4e5f6... ServeReplica:qwen-7b ALIVE {"GPU": 1.0}
g7h8i9... RayWorkerWrapper ALIVE {"GPU": 1.0}
That output — actors with names like ServeController and ServeReplica — is the layer RayService is ultimately responsible for bringing into existence, even though the CRD itself never mentions an actor by name.
GPU Scheduling and Sharing in Ray
Ray’s GPU handling has two levels: how Ray decides where to place GPU work, and what actually happens to the hardware once it does.
Whole-GPU allocation
By default, num_gpus=1 on a task or actor reserves one full logical GPU. Ray tracks this per node and will not oversubscribe: if a node has two physical GPUs, at most two num_gpus=1 actors will be scheduled there concurrently, and Ray sets CUDA_VISIBLE_DEVICES for each actor so it only sees its assigned device.
Fractional GPU allocation
Ray also accepts fractional values, such as num_gpus=0.5, allowing multiple actors to share a single physical GPU from Ray’s scheduling perspective.
import ray
ray.init(num_gpus=1)
@ray.remote(num_gpus=0.5)
class SmallModelActor:
def ping(self):
return ray.get_runtime_context().get_accelerator_ids()["GPU"]
actors = [SmallModelActor.remote() for _ in range(2)]
print(ray.get([a.ping.remote() for a in actors]))
# Both actors report GPU id: ['0'] — sharing the same physical device
This is genuinely useful for small models that don’t need an entire H100 or H200. But it is essential to understand what this mechanism actually is: bookkeeping, not isolation.
Ray’s own documentation is explicit that resource requirements do not impose limits on actual physical usage. Nothing stops a num_gpus=0.5 actor from allocating more than half the GPU’s memory; Ray simply has no visibility into or control over that. There is no time-slicing scheduler, no memory partitioning, and no equivalent of NVIDIA MPS or MIG happening under Ray’s control. If two fractional actors together try to use more memory than the GPU has, you get an out-of-memory error, not a graceful denial.
The consequence is that fractional GPU sharing in Ray only works if the application layer self-limits. For vLLM specifically, that means keeping gpu_memory_utilization aligned with the fraction you declared to Ray:
# Two 0.4-fraction vLLM deployments sharing one GPU
applications:
- name: model-a
args:
llm_configs:
- model_loading_config:
model_id: model-a
model_source: some-org/small-model-a
engine_kwargs:
gpu_memory_utilization: 0.4
deployment_config:
ray_actor_options:
num_gpus: 0.4
- name: model-b
args:
llm_configs:
- model_loading_config:
model_id: model-b
model_source: some-org/small-model-b
engine_kwargs:
gpu_memory_utilization: 0.4
deployment_config:
ray_actor_options:
num_gpus: 0.4
Good practice for fractional serving:
- Leave headroom: request slightly less than the theoretical fraction (0.35 instead of 0.4) to absorb framework overhead.
- Keep
gpu_memory_utilizationand the declarednum_gpusfraction in sync; a mismatch is the most common cause of OOM in shared configurations. - Account for everything: model weights, KV cache, and CUDA graph memory all count against the budget, not just weights.
- Start conservative and increase replica density gradually while watching for OOM errors.
If you need real hardware-enforced isolation between workloads sharing a GPU rather than a cooperative agreement, that requires a different layer entirely, such as NVIDIA MPS or MIG partitioning at the driver level. Ray’s fractional GPUs are a scheduling convenience, not a substitute for those.
Placement groups for multi-GPU models
When a model needs more than one GPU, Ray uses placement groups to reserve a bundle of resources together and keep tensor-parallel workers co-located when possible:
llm_config = dict(
model_loading_config=dict(
model_id="llama-3.1-70b",
model_source="meta-llama/Llama-3.1-70B-Instruct",
),
engine_kwargs=dict(
tensor_parallel_size=4,
pipeline_parallel_size=2,
),
)
By default, Ray Serve LLM uses a PACK placement strategy: fit as many of a model’s workers as possible onto as few nodes as possible, and only spill onto additional nodes once a single node’s resources are exhausted. This is what allows a model too large for one node’s GPUs to still be served coherently across several.
True GPU Time Slicing with KAI or Volcano
The fractional GPU mechanism described above has a real limitation: it is Ray’s own bookkeeping, invisible to Kubernetes, and enforced by nothing. Ray’s scheduler will happily admit two num_gpus=0.5 actors onto one device, but Kubernetes itself still sees a single, indivisible nvidia.com/gpu: 1 resource, and the actual hardware sharing between those two actors’ processes is left entirely to the application layer to self-limit. If you need real interleaved execution on the hardware, enforced below Ray rather than trusted to it, that has to be handled by a GPU-aware Kubernetes scheduler instead of, or alongside, Ray’s own accounting.
This is exactly the gap that schedulers like the NVIDIA KAI Scheduler and Volcano are built to close, and both now have native KubeRay integration.
NVIDIA KAI Scheduler
KAI Scheduler (open-sourced by NVIDIA, originating from Run:ai) became natively integrated with KubeRay as of KubeRay v1.5. The integration brings gang scheduling, workload autoscaling, and workload prioritization to Ray clusters — including letting high-priority inference jobs preempt lower-priority training jobs — plus GPU sharing across teams on the same physical device.
The GPU sharing mechanism works by pairing KAI’s scheduling logic with the NVIDIA GPU Operator’s time-slicing feature. The device plugin is configured to advertise multiple replicas of a single physical GPU to Kubernetes, and KAI performs admission control and placement scoring over fractional requests, tracking how much of each GPU has already been claimed so it can bin-pack more intelligently than naive round-robin sharing. Enabling it is a flag on the KubeRay operator install:
helm install kuberay-operator kuberay/kuberay-operator `
--version 1.5.0 `
--set batchScheduler.name=kai-scheduler
A worker group then requests a fraction of a GPU the same way it would request any other Kubernetes resource, via KAI’s annotations rather than a bare nvidia.com/gpu limit:
workerGroupSpecs:
- groupName: shared-gpu-workers
template:
metadata:
labels:
runai/queue: inference-team
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.56.0-py311-gpu
schedulerName: kai-scheduler
# GPU fraction requested via KAI's pod-level annotation, e.g. gpu-fraction: "0.5"
The important caveat carries over from the discussion of Ray’s own fractional GPUs, and it is worth stating plainly: this is genuinely truer time slicing than Ray’s logical accounting, because the GPU driver itself is now the thing context-switching between the two pods’ processes, rather than nothing enforcing anything at all. But it is still not memory isolation. Ray’s own documentation on the KAI integration is explicit that GPU sharing with time slicing happens at the Kubernetes layer, letting multiple pods share a device, but the scheduler does not enforce memory isolation — applications must still manage their own usage to avoid interfering with their neighbours. A workload that ignores its declared fraction can still starve or crash whatever else is sharing that GPU. For workloads that need hardware-level memory and fault isolation rather than cooperative sharing, that requires MIG partitioning instead, which is a different mechanism again and only available on newer datacenter GPUs.
Volcano
Volcano is a CNCF batch scheduling system for Kubernetes, and its integration with KubeRay predates KAI’s, going back to KubeRay v0.4. Its headline contribution to Ray workloads is gang scheduling: guaranteeing that all the pods belonging to a RayCluster, RayJob, or (as of KubeRay v1.5.1) a RayService come up together as an all-or-nothing unit, rather than Kubernetes scheduling some of them and leaving the rest pending indefinitely.
This solves a specific and genuinely painful failure mode. Without gang scheduling, a Kubernetes scheduler has no concept that a Ray worker group’s pods are jointly required — it might place the head pod and half the worker pods for a tensor-parallel deployment, consume cluster resources doing so, and then simply never be able to schedule the remaining workers if the rest of the cluster is busy. The Ray cluster sits there partially formed, holding resources hostage, unable to ever complete initialization. Volcano fixes this by grouping the pods into a PodGroup with a minMember count, and refusing to schedule any of them until all of them can be scheduled together:
kubectl get podgroup ray-test-cluster-0-pg -o yaml
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
spec:
minMember: 3
minResources:
cpu: "3"
memory: 4Gi
queue: kuberay-test-queue
status:
phase: Running
Turning it on is, again, a flag at operator install time:
helm install kuberay-operator kuberay/kuberay-operator `
--version 1.5.0 `
--set batchScheduler.name=volcano
Volcano also ships its own device-sharing scheduling plugins, capable of fractional and time-sliced GPU allocation in a similar spirit to KAI’s approach, though in practice gang scheduling and queue-based fair sharing across tenants are the more commonly cited reasons to pair Volcano with Ray.
Choosing between them, and what neither one buys you
KAI and Volcano are not mutually exclusive with Ray’s own fractional GPU support — they operate one layer below it. Ray still decides how to place actors within a cluster’s declared logical resources; KAI or Volcano decide how Kubernetes places the pods those actors will eventually run inside, and, if configured for GPU sharing, arrange for the driver to genuinely time-slice between pods rather than trusting Ray’s bookkeeping alone. For gang scheduling specifically, either scheduler removes a real source of stuck deployments for any multi-GPU, multi-node Ray Serve LLM configuration. For GPU sharing specifically, both improve on Ray’s fractional GPUs by making the sharing visible to and coordinated by Kubernetes itself, and by giving the driver an actual interleaving mechanism to use — but neither delivers hardware-enforced memory isolation. If your workloads cannot tolerate one tenant’s memory usage affecting another’s, only MIG achieves that, and it comes with its own constraints on which GPU models support it and how finely a device can be partitioned.
How vLLM Integrates with Ray
vLLM’s connection to Ray is not superficial glue code; it is a first-class execution backend built into vLLM’s own architecture.
The two layers inside vLLM
vLLM separates its model-execution logic into two layers: an Executor and a Worker. The Executor sits beside the engine core and acts as a proxy to one or more worker processes or actors, handling process lifecycle, RPC dispatch, and result collection. The Worker runs inside each process (or Ray actor) and does the actual heavy lifting: device initialization, model loading, KV cache allocation, and running forward passes.
vLLM ships more than one Executor implementation. UniprocExecutor runs everything in a single process for the simplest case. MultiprocExecutor spawns local subprocesses when everything fits on one node. RayDistributedExecutor is the one that matters here — it distributes those Worker instances as Ray actors.
Choosing the backend
vLLM decides automatically whether to use multiprocessing or Ray, and you can override it explicitly:
from vllm import LLM
# Explicit Ray backend, useful for multi-node
llm = LLM(
model="meta-llama/Llama-3.1-70B-Instruct",
tensor_parallel_size=8,
distributed_executor_backend="ray",
)
Or via the OpenAI-compatible server:
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 8 \
--distributed-executor-backend ray
The default behaviour is sensible: if the requested tensor-parallel size fits on the current node’s available GPUs and you are not already inside a Ray placement group, vLLM uses local multiprocessing. Multi-node inference, however, currently requires Ray, because only Ray provides the cross-node process placement and coordination that scenario needs.
What RayDistributedExecutor actually does
When Ray is selected as the backend, vLLM’s RayDistributedExecutor performs a specific sequence during initialization:
- Resolves bundle indices from the Ray placement group, so it knows which reserved GPU slots to use.
- Creates
RayWorkerWrapperremote actors, one per tensor-parallel/pipeline-parallel rank. - Collects IP addresses from all created actors and re-sorts workers so the driver’s own node comes first.
- Adjusts ranks via a collective RPC call to account for that re-sorting.
- Sets
CUDA_VISIBLE_DEVICESper node via another collective RPC call. - Calls
init_worker,init_device, andload_modelacross all actors to bring the distributed engine online. - Organizes workers into a
[pipeline_rank][tensor_rank]grid so subsequent forward passes know exactly which actor handles which shard.
The key point: RayWorkerWrapper is a genuine Ray actor class. vLLM’s model-execution code runs inside its methods, invoked the same way any other Ray actor method would be — via Ray’s RPC mechanism, with Ray responsible for keeping the actor alive, tracking its placement, and detecting failure.
Where Ray’s job ends and vLLM’s job begins
It’s worth being precise about the division of labour, because it clarifies a lot of otherwise-confusing behaviour. Ray’s contribution is cluster orchestration: process placement, fault detection, and resource bookkeeping across nodes. The actual high-throughput data path — tensor communication between GPUs during a forward pass — does not go through Ray at all. It uses the same mechanisms vLLM’s own multiprocessing backend uses: NCCL for cross-device tensor communication, and ZMQ or shared memory for internal control-plane messaging. Ray is not in the hot path of token generation; it placed the actors and is watching them, but it is not routing tensors between them.
This is also why the Ray community has been actively revising this backend. A recent proposal (tracked as an RFC against the Ray-team-authored executor) suggested simplifying the Ray executor to use Ray purely as a process launcher and placement manager, aligning its control and data plane with the plain multiprocessing executor rather than a more complex compiled-graph mechanism. The direction of travel is toward Ray doing less inside the hot path over time, while still owning orchestration.
Data-parallel engine cores as actors too
For data-parallel serving (running multiple full copies of a model’s engine core to increase throughput rather than to fit a large model), vLLM defines a dedicated Ray actor class for exactly this purpose, running the engine core’s busy loop inside a Ray actor and skipping the network handshake that a standalone process would otherwise need, because all addressing information is already known at actor-creation time.
Ray Serve LLM
Layered on top of the raw executor integration is Ray Serve LLM, Anyscale’s purpose-built serving framework for LLMs on Ray, which is what most production KubeRay deployments actually configure directly rather than wiring up RayDistributedExecutor by hand.
Ray Serve LLM provides:
- Advanced parallelism strategies, combining pipeline parallelism, tensor parallelism, expert parallelism, and data-parallel attention for models of any size.
- Prefill-decode disaggregation, separating and independently scaling the two phases of inference for better resource utilization.
- Custom request routing, including prefix-aware and session-aware strategies to maximize KV-cache hit rates.
- Automatic multi-node placement and coordination for models that span more than one machine.
- An OpenAI-compatible ingress, so existing client code that talks to the OpenAI API can point at your Ray Serve LLM endpoint with no changes.
A basic single-model deployment, defined entirely in Python:
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(min_replicas=1, max_replicas=4),
),
accelerator_type="L4",
engine_kwargs=dict(
tensor_parallel_size=1,
max_model_len=8192,
enable_chunked_prefill=True,
),
)
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
Serving multiple models from the same cluster is just multiple LLMConfig entries, each getting independent autoscaling:
llm_configs = [
LLMConfig(
model_loading_config=dict(
model_id="mistral-7b",
model_source="mistralai/Mistral-7B-Instruct-v0.3",
),
engine_kwargs=dict(max_model_len=8192),
deployment_config=dict(
autoscaling_config=dict(min_replicas=1, max_replicas=2),
),
accelerator_type="A10G",
),
LLMConfig(
model_loading_config=dict(
model_id="qwen-7b",
model_source="Qwen/Qwen2.5-7B-Instruct",
),
engine_kwargs=dict(max_model_len=4096),
deployment_config=dict(
autoscaling_config=dict(min_replicas=1, max_replicas=2),
),
accelerator_type="A10G",
),
]
app = build_openai_app({"llm_configs": llm_configs})
serve.run(app, blocking=True)
Cross-node parallelism for a model that doesn’t fit on a single node’s GPUs:
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-70b",
model_source="meta-llama/Llama-3.1-70B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(min_replicas=1, max_replicas=1),
),
engine_kwargs=dict(
tensor_parallel_size=4,
pipeline_parallel_size=2,
max_model_len=8192,
enable_chunked_prefill=True,
max_num_batched_tokens=4096,
),
)
That configuration requests eight total GPUs (four tensor-parallel times two pipeline-parallel), and Ray’s placement group logic decides how to lay those eight workers across your available nodes, packing them as tightly as it can before spilling across machines.
Deploying Ray Serve LLM Through KubeRay
Bringing this together, a RayService is how Ray Serve LLM configurations actually reach a Kubernetes cluster. The serveConfigV2 field carries the same application and LLMConfig structure shown above, expressed as YAML:
apiVersion: ray.io/v1
kind: RayService
metadata:
name: qwen-2-5-7b
spec:
serveConfigV2: |
applications:
- name: qwen-7b
route_prefix: /
import_path: ray.serve.llm:build_openai_app
args:
llm_configs:
- model_loading_config:
model_id: qwen-7b
model_source: Qwen/Qwen2.5-7B-Instruct
engine_kwargs:
tensor_parallel_size: 1
max_model_len: 8192
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 3
rayClusterConfig:
rayVersion: "2.56.0"
headGroupSpec:
rayStartParams: {}
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.56.0-py311-gpu
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
resources:
limits:
nvidia.com/gpu: "1"
workerGroupSpecs:
- groupName: gpu-workers
replicas: 1
minReplicas: 1
maxReplicas: 3
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.56.0-py311-gpu
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
resources:
limits:
nvidia.com/gpu: "1"
After applying it, three levels of autoscaling can be in play simultaneously: the Ray Serve application autoscaler adjusts replica count based on request load, the Ray autoscaler adds or removes worker pods based on logical resource demand, and the Kubernetes cluster autoscaler provisions new GPU nodes when the Ray autoscaler’s demand exceeds what the cluster currently has.
Checking the deployment uses ordinary kubectl commands, but the pod names reveal the Ray substrate underneath:
kubectl get pods -n default
# NAME READY STATUS AGE
# kuberay-operator-56fd8bff68-rwdrq 1/1 Running 3d
# qwen-2-5-7b-raycluster-mbcpn-head-qrznn 1/1 Running 10m
# qwen-2-5-7b-raycluster-mbcpn-worker-zwlqg 1/1 Running 10m
kubectl get svc
# NAME TYPE PORT(S)
# qwen-2-5-7b-raycluster-head-svc ClusterIP 8000/TCP,8265/TCP
# Access the Ray dashboard
kubectl port-forward svc/qwen-2-5-7b-raycluster-head-svc 8265:8265
# Send a request to the OpenAI-compatible endpoint
kubectl port-forward svc/qwen-2-5-7b-raycluster-head-svc 8000:8000
curl http://localhost:8000/v1/chat/completions `
-H "Content-Type: application/json" `
-d '{
"model": "qwen-7b",
"messages": [{"role": "user", "content": "Explain what a Raylet does."}]
}'
A Note on the Head Node
One operational detail that catches people out: the Ray head node in a KubeRay deployment is meant to act primarily as the control plane, running the GCS, the Ray dashboard, and the Serve controller. It is common practice to keep GPU-heavy workloads off the head node entirely, so it stays responsive as a scheduler even under load. In vLLM’s Ray executor, however, there is currently a wrinkle: because the vllm serve command itself runs on the Ray head node when Ray is the backend, vLLM requires at least one visible GPU on that pod, meaning the head node ends up participating in tensor and pipeline parallelism whether or not that’s architecturally desirable. Worth accounting for when sizing your head node’s resource requests, and worth watching for changes as the Ray executor backend continues to evolve.
Summary
Ray is a general-purpose distributed compute framework built around tasks and actors, with a per-node Raylet and a cluster-wide GCS coordinating scheduling and fault tolerance. KubeRay translates that model into Kubernetes-native custom resources — RayCluster, RayJob, and RayService — so Ray clusters can be declared, scaled, and upgraded the way any other Kubernetes-managed application is.
Running vLLM under Ray changes what your cluster looks like at the pod level: instead of one pod per model replica, you get Ray head and worker pods that host vLLM engines as actors, with Ray responsible for placement and fault detection while NCCL and shared memory handle the actual data path. GPU sharing in Ray, including its fractional GPU support, is a scheduling abstraction rather than an enforced hardware boundary, which puts the responsibility for staying within a memory budget on vLLM’s own configuration.
vLLM’s integration with Ray is not incidental. Its RayDistributedExecutor creates and manages RayWorkerWrapper actors directly, and Ray Serve LLM builds a full production serving layer — multi-model composition, autoscaling, prefix-aware routing, prefill-decode disaggregation — on top of that foundation. For anything beyond a single model on a single node, that combination is the standard way to run LLM inference at scale inside Kubernetes today.
Connect with me on Medium @david.b.chase
