KServe: Model Inference on Kubernetes Explained

While Kubernetes is the de facto answer for all modern scalable deployment requirements, deploying a model inference point comes with several challenges, including versioning, traffic management, GPU resource allocation, autoscaling signals, cold start behavior, and runtime compatibility that standard Kubernetes primitives were never designed to address. 

KServe is a purpose-built inference infrastructure built on Kubernetes to address these challenges. 

This article explains key concepts and best practices for deploying models with KServe.

Summary of key KServe concepts 

Understanding KServe

KServe is a standardized inference platform built on top of Kubernetes that abstracts away the complexity of deploying ML models as an HTTP or gRPC service through declarative custom resources: InferenceService for predictive AI models and, since v0.16, LLMInferenceService for LLM workloads. 

One can use InferenceService to define what needs to be served, such as an inference framework, a model URI, or a scaling policy. It integrates with Knative Serving for request-driven auto-scaling and Istio for traffic management. 

Key features of KServe include the following. 

Scale to zero

KServe relies on the Knative Pod Autoscaler (KPA) for request-based autoscaling, It’s a key aspect of serving ML/AI models. It scales resources to zero when there are no requests and wakes them up when requests start coming in. 

KPA scales on concurrent requests or requests per second, which suits bursty inference traffic. That differs from the default Kubernetes Horizontal Pod Autoscaler (HPA), which scales on CPU or memory and cannot scale to zero. 

While scaling to zero reduces GPU costs, it introduces cold-start latency issues, which can be mitigated through strategies like keeping a minimum warm replica pool, pre-loading models, or predictive autoscaling tuned to your traffic patterns.  

Canary deployments

Canary deployment is a risk-mitigation strategy used when releasing new versions of AI/ML systems. It involves routing a small percentage of traffic through a newer model or service while keeping the original model, helpful while executing A/B tests, shadow deployments, progressive deployment, etc.

KServe enables this by specifying a percentage directly in the InferenceService spec. 

Multi-model serving

KServe supports serving multiple models through integrating with the ModelMesh extension. ModelMesh works best with high-density, smaller models such as classical ML or small NLP models. For large LLMs that occupy an entire GPU, the standard single-model-per-pod setup is still the better choice.

Wide spectrum of inference workloads

KServe can serve classical ML models as well as large language models. It comes with prebuilt run times for Triton, TFServing, TorchServe, etc. It also supports

• Standardized V2 inference protocol endpoint.
• Dedicated LLM runtimes like vLLM, HuggingFace TGI, and OpenAI-compatible serving backends.
• Continuous batching as well as streaming responses without the user having to manage the runtime cycle. 

AI gateway integration

For generative AI workloads, KServe integrates with Envoy AI Gateway, provisioning and managing it automatically as part of the LLMInferenceService deployment. This matters because LLM traffic needs things classical model serving doesn’t, like prefix-cache aware routing to the replica most likely to already hold relevant context, token-aware rate limiting, and multi-tenant auth, all exposed through a single governed endpoint rather than a patchwork of separately sourced tools.

Observability

Kserve supports metrics, tracing, and logging. It can integrate with Prometheus to capture request latency, throughput, queue depth, and any model-specific metrics viewable on Grafana dashboards. 

KServe also supports payload logging by injecting a logging sidecar that has access to raw input and output without touching the actual model container. 

KServe vs. Kubeflow

KServe and Kubeflow address fundamentally different parts of the MLOps surface. 

Kubeflow is an open-source, Kubernetes-native MLOps platform that allows data scientists and engineers to manage the entire AI lifecycle, from data exploration and model training to deployment, without needing to master low-level Kubernetes infrastructure. KServe is integrated and abstracted by Kubeflow, making it a good choice if you want a single platform to manage all the components mentioned above. 

KServe is focused only on inference. If the use case is only inference, it is better to use KServe for operational simplicity and reduce overheads. Using it standalone requires you to bring your own training infrastructure. 

Features like ModelMesh are native to KServe, and, at this point, LLM inference support is better on KServe. 

KServe key components

KServe has three main components split across two planes. The KServe controller operates in the control plane, reconciling the desired state of every InferenceService. ModelMesh and the open inference protocol sit in the data plane, where they load models and serve live requests. 

Underneath all of this is the Predictor, the core building block of an InferenceService. The Predictor holds your model and its runtime, and the other components attach around it.

KServe controller

The KServe controller runs as a pod within the cluster and monitors for custom InferenceService resources. The controller translates the InferenceService definition YAML file into all the Kubernetes and Knative objects required to actually serve the model. This includes:

• Serving runtime container pod
• Storage initializer to pull the artifact 
• Knative service that handles autoscaling
• Istio service for traffic management. 

The controller runs a continuous loop to manage the cluster state, ensuring that the resources declared in the definitions are indeed there even after a crash or failure. 

ModelMesh

The default KServe deployment strategy allocates a dedicated set of pods per model. While this is fine for fewer models, it fails to scale.

ModelMesh replaces this with a shared set of inference pods, where models are loaded or evicted dynamically based on demand using the least-recently-used (LRU) policy. A dedicated sidecar agent on each server pod executes load and eviction operations, and a central etcd store tracks which model lives on which pod. Incoming requests are routed to the relevant pod that handles the specific model. 

Open inference protocol

Open Inference Protocol v2 is the standard protocol that KServe follows to expose all the model endpoints. Irrespective of the underlying model runtime (e.g., Triton, TFServe), the client always sees the same API structure and request format. This helps decouple the client application code from the serving backend runtime, enabling runtime swaps at will without affecting client applications. 

Optional components

In addition to the three main components mentioned above, KServe includes optional components. 

• The Transformer helps to deploy hooks for pre- and post-processing logic without affecting the actual prediction container. 
• The Explainer component provides model output explanations by integrating with explainability libraries.
• The Logger captures request and response payloads by streaming them to an external endpoint, without touching the model container.

Configuring and deploying with KServe

The components above come together in a single declarative manifest that you apply to the cluster.

Everything with KServe starts with the InferenceService custom resource definition. Its spec maps directly to the components described earlier:

• Predictor (required)
• Transformer (optional)
• Explainer (optional)
• Logger (optional)

Let’s consider an example InferenceService YAML to understand the concepts. Given below is a YAML file for deploying a Triton inference runtime that requests a single GPU. 

spec:

  predictor:

    minReplicas: 1

    maxReplicas: 10

    scaleTarget: 5          # target concurrent requests per pod

    scaleMetric: concurrency

    model:

      modelFormat:

        name: triton

      runtime: triton-2.x   # pin a specific ServingRuntime

      storageUri: s3://my-models/resnet50/

      resources:

        requests:

          cpu: "2"

          memory: 4Gi

          nvidia.com/gpu: "1"

        limits:

          nvidia.com/gpu: "1"

      env:

        - name: OMP_NUM_THREADS

          value: "4"

The above snippet uses the minReplicas configuration to specify that the scaling should be to a minimum of one instance. If one sets this to zero, KServe will scale down to zero instances at the cost of some cold-start latency. 

The scale target here is 5, which means up to 5 concurrent requests are expected. The model format and model runtime names are specified as Triton and its specific version. 

Canary deployments, introduced earlier as a key feature, are configured on this same predictor. With many Kubernetes tools you would deploy a second copy of the service for the new version. KServe works differently. You do not add a second predictor to run a canary. Instead, you update the existing predictor with the new model and add a canaryTrafficPercent field. KServe then routes that share of traffic to the new model and keeps the rest on the current one. The snippet below sends 10 percent of traffic to a new fraud-detector model.

spec:
  predictor:
    canaryTrafficPercent: 10
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://my-models/fraud-detector/v4   # canary (new)

A similar version can also be used to serve LLMs in a basic single-node setup. For advanced LLM serving, KServe v0.16 introduced a dedicated LLMInferenceService CRD that adds prefill-decode separation, intelligent routing, and multi-node orchestration. Given below is a configuration to serve a LLAMA model using vLLM as the runtime.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-8b
  namespace: production
spec:
  predictor:
    minReplicas: 1
    model:
      modelFormat:
        name: vllm
      storageUri: s3://my-models/llama3-8b/
      resources:
        requests:
          nvidia.com/gpu: "2"
        limits:
          nvidia.com/gpu: "2"
      args:
        - --tensor-parallel-size=2
        - --max-model-len=8192
        - --enable-chunked-prefill

KServe supports three deployment modes. 

• Serverless, in which each InferenceService has its own Knative service with full autoscaling, including scaling to zero. (Default deployment mode)
• RawDeployment, which has lower operational complexity but does not use Knative and loses Knative-specific capabilities, such as scaling to zero. 
• ModelMesh, which enables the use of a shared resource pool that dynamically loads and evicts models. 

One can configure the deployment modes using the snippet below.

metadata:

  annotations:

    serving.kserve.io/deploymentMode: ModelMesh

Best practices while working with KServe

Consider implementing the following best practices.

Pin server runtimes deliberately

While KServe can infer appropriate server runtime from your model format, it is always better to pin the runtime version deliberately to reduce risks. If Kserve auto-selects run time, upgrading the ClusterServingRuntime definitions can silently change the runtime version your model loads against. 

Right-size GPU instances

GPU instances are the largest cost driver in any inference platform. The typical instinct to provision a large instance and move on can lead to cost issues later. 

However, GPU right-sizing is not straightforward. Several parameters like GPU utilization, throughput capacity, etc., have complex relationships with the model’s memory footprint, batch size, traffic patterns, etc. Platforms like Kubex that analyse GPU memory utilization across your pods, NVIDIA GPU configs like MIG slicing, can help in flagging antipatterns and recommending optimized configurations. 

Adopt ModelMesh for multi-model serving

If your platform has more than a few dozen models, the default one model per pod is not an ideal configuration. It can quickly fill up your GPU budget through idle resource allocation. The shared pool approach of ModelMesh improves GPU utilization by dynamically loading and evicting models.

Use the DCGM exporter for hardware-level observability

While KServe’s built-in metrics cover the basics like request latency, throughput, queue depth, etc., they do not reveal what is happening at the GPU hardware level. NVIDIA’s DCGM exporter can run as a DaemonSet on the GPU node and provide information regarding SM utilization, memory bandwidth, NVLink throughput, temperature, power draw, etc. 

Audit frequently for stale models

KServe makes it very easy for anyone within the team to deploy models in your cluster, which can sometimes lead to model accumulation. Each InferenceService runs pods corresponding to its minReplica parameter, which consume GPU resources. A periodic audit coupled with an analysis of your observability metrics provides visibility into ways of reducing GPU costs. 

Enforce multi-tenancy and isolation in clusters

In shared clusters, it is important to ensure namespace-level isolation. One should enforce dedicated namespaces per team with ResourceQuota objects to cap GPU and CPU utilization. One can also use Kubernetes RBAC to restrict tenant access to the InferenceService descriptor. Cluster-scoped resources, such as ClusterServingRuntime, must remain within the platform team’s ownership. 

Monitor and optimize latency continuously

Inference latency is not a static metric. It often degrades as time passes because of traffic pattern shifts, model evolution, cluster resource pressures etc. Beyond the serving layer, there are also the infrastructure-level factors that affect latency. A platform like Kubex that provides infrastructure-level GPU analysis and historical utilization patterns can help in analyzing latency issues. Latency issues can stem from several factors, such as resource contention, suboptimal instance selection, or scheduling inefficiencies. Kubex’s GPU analysis capabilities can unearth these.

Automated AI / GPU Infrastructure Optimization

LEARN MORE

• 

  Predict optimal GPU / XPU needs using transparent machine learning

• 

  Automate w/ a purpose-built controller based on the Nvidia KAI scheduler

• 

  Use agentic AI for workload analysis and scenario modeling

Last Thoughts

KServe helps simplify inference deployments in Kubernetes. It wraps the complexity of runtimes, autoscaling, traffic management, and protocol standardization behind a single InferenceService declaration. It also provides precise operational controls for canary rollouts, scale-to-zero, multi-model density, and payload logging. 

Consistently using those controls is what separates a well-run inference platform from an expensive one. Platforms like Kubex surface exactly where GPU resources are being wasted across your KServe deployments and how to prevent it.