Automated Resource Tuning for Inference and ML Workloads
More Inference – Fewer GPUs
Break the one model per GPU pattern with dynamic fractioning.
Put them to work on multiple models or reduce your GPU footprint.
What Kubex does for AI inference and ML
-
GPU fractioning
Several models share one GPU, each with a sized slice of compute and memory.
- Time-slicing, MIG or MPS, chosen per workload
- GPU memory limits enforced per container
- Slices resized as demand changes
-
KV cache tuning
KV cache sized to real traffic, automatically, so reserved GPU memory gets used.
- Tracks cache usage and preemptions
- Frees memory vLLM reserved but never used
- Keeps headroom for traffic spikes
-
Continuous vLLM optimization
A closed loop that reads vLLM metrics, adjusts settings and checks the result.
- Token throughput, time to first token and per-token latency
- Batch size, concurrency and replicas
- Changes only within your latency targets
Sound familiar?
Every new model comes with a request for another GPU.
We gave out a GPU for each model and never went back.
The dashboards say busy. Throughput says otherwise.
Better GPU utilization
Lower cost per token
To a full GPU fleet baseline
-
Works with what you run
Kubernetes-native. No rip and replace.
Built on CNCF projects
-
KAI Scheduler
GPU-aware scheduling and fractional allocation.
-
HAMi-core
GPU memory and compute isolation for safe sharing.
-
-
You stay in control
Kubex acts inside the limits you set.
- Set configuration by team, model or GPU pool
- Recommend-only, HITL mode for sensitive workloads
- Every change logged and reversible
Runs on and alongside
Agentic harness
The market leading agentic & deterministic engine accessed from where you work.
The Kubex Agent works through the deterministic engine and inside your policies, so every answer is grounded and every action is governed.
-
Agent
One Kubex Agent. Ask it questions, approve its changes, or let it act on its own.
-
Skills
What the agent knows how to do: GPU fractioning, KV cache sizing, vLLM tuning, GPU node selection. Pull the same skills into Claude, Cursor or your own tools.
-
Tasks
Scheduled prompts that put skills to work on your clusters. Turn on the ones that ship, or write your own.
-
Proposals
Every change the agent, skills and tasks find comes forward as a proposal. Review and approve them for automation.
Example tasks
Turn these on, or build your own.
-
Pick GPU types
Matches each model to a GPU type and instance shape. -
Pre-warm GPU nodes
Warms nodes and loads models ahead of traffic. -
Find idle GPUs
Flags GPUs with little or no work on them. -
Compare GPU cost across clouds
Prices the same workload on each provider. -
Report cost per token
Sends a weekly summary by model. -
Your own task
Write it as a prompt, pick the skills, set the schedule.











