Background Mask Animation
GPU and Inference Optimization

Automated Resource Tuning for Inference and ML Workloads

More Inference – Fewer GPUs

Break the one model per GPU pattern with dynamic fractioning.
Put them to work on multiple models or reduce your GPU footprint.

What Kubex does for AI inference and ML

  • GPU fractioning

    Several models share one GPU, each with a sized slice of compute and memory.

    • Time-slicing, MIG or MPS, chosen per workload
    • GPU memory limits enforced per container
    • Slices resized as demand changes
  • KV cache tuning

    KV cache sized to real traffic, automatically, so reserved GPU memory gets used.

    • Tracks cache usage and preemptions
    • Frees memory vLLM reserved but never used
    • Keeps headroom for traffic spikes
  • Continuous vLLM optimization

    A closed loop that reads vLLM metrics, adjusts settings and checks the result.

    • Token throughput, time to first token and per-token latency
    • Batch size, concurrency and replicas
    • Changes only within your latency targets

Sound familiar?

Every new model comes with a request for another GPU.

We gave out a GPU for each model and never went back.

The dashboards say busy. Throughput says otherwise.

40-70%
2-3X
< 1Day
  • Works with what you run

    Kubernetes-native. No rip and replace.

    Built on CNCF projects

    • KAI Scheduler

      GPU-aware scheduling and fractional allocation.

    • HAMi-core

      GPU memory and compute isolation for safe sharing.

  • You stay in control

    Kubex acts inside the limits you set.

    • Set configuration by team, model or GPU pool
    • Recommend-only, HITL mode for sensitive workloads
    • Every change logged and reversible

Runs on and alongside

Agentic harness

The market leading agentic & deterministic engine accessed from where you work.

The Kubex Agent works through the deterministic engine and inside your policies, so every answer is grounded and every action is governed.

  • Agent

    One Kubex Agent. Ask it questions, approve its changes, or let it act on its own.

  • Skills

    What the agent knows how to do: GPU fractioning, KV cache sizing, vLLM tuning, GPU node selection. Pull the same skills into Claude, Cursor or your own tools.

  • Tasks

    Scheduled prompts that put skills to work on your clusters. Turn on the ones that ship, or write your own.

  • Proposals

    Every change the agent, skills and tasks find comes forward as a proposal. Review and approve them for automation.

Example tasks

Turn these on, or build your own.

  • Pick GPU types

    Matches each model to a GPU type and instance shape.
  • Pre-warm GPU nodes

    Warms nodes and loads models ahead of traffic.
  • Find idle GPUs

    Flags GPUs with little or no work on them.
  • Compare GPU cost across clouds

    Prices the same workload on each provider.
  • Report cost per token

    Sends a weekly summary by model.
  • Your own task

    Write it as a prompt, pick the skills, set the schedule.
Use Kubex skills in

Frequently asked questions

What is GPU fractioning?

What is the difference between MIG, time-slicing and MPS?

Can one model running out of memory affect others on a shared GPU?

What is KV cache tuning?

What does continuous vLLM optimization mean?

Will Kubex hurt inference latency?

Which serving stacks and platforms does Kubex support?

Does Kubex choose GPU types?

Can I review changes before they happen?