Coding agents changed implementation economics faster than they changed confidence. They let us produce more code, more quickly, but they did not make reviewers any better at understanding system-wide consequences. In our Kubernetes automation stack, that gap became impossible to ignore once AI started generating meaningful amounts of controller code.
For a while, our review habit stayed the same: inspect the diff, mentally simulate the side effects, and hope the meaning of the change stayed local. That broke down faster than we expected. The problem was not that the diffs were large. The problem was that even small diffs lived inside a system where selection logic, safety checks, retries, and apply behavior were too entangled to reason about locally.
What fixed that was not a better diff discipline. It was a better boundary. The unit of safety stopped being the diff and became the contract boundary between components that produce intent and the component that decides behavior.
Why small diffs stopped being meaningful
Our old rightsizing system could automate container resizing, but it did so through a broad polling and scanning flow with blurry ownership. Reviewers looking at a small code change still had to answer system-wide questions: where precedence was really decided, which safety checks applied on that path, and whether a component was merely producing a recommendation or quietly deciding runtime behavior.
Once AI accelerated implementation, that weakness became much more expensive. More code landed across the same wide control surface, and review burden grew faster than diff size. Smaller diffs still helped, but only when their meaning was actually local.
That forced a different review model. Instead of asking reviewers to inspect code more carefully across the whole system, we changed the system so that the consequences of a change were easier to localize.
The boundary that made review local again
The core architectural split was simple: producers emit intent, and a central arbiter decides behavior. That rule changed review because it gave each change a clearer owner and a narrower set of consequences.
A producer could focus on one question: what should happen for this workload? The arbiter could focus on a different one: given all incoming recommendations, what is the safe action? Producers no longer decided final behavior inline, and reviewers no longer had to infer those semantics from several controller paths at once.
At a high level, the flow looked like this:
That diagram mattered because it showed where meaning lived. Producers were allowed to propose intent. One place was responsible for turning that intent into behavior. Once that interface was explicit, review could move up a level without becoming vague.
The recommendation contract
Mechanically, the boundary was carried through a serialized recommendation payload attached to workload and pod annotations. The full payload contains more detail, but the minimal shape looked like this:
{
“policyKind“: “StaticPolicy“,
“policyName“: “baseline”,
“policyWeight“: 10,
“automationStrategyName“: “safe-resize”,
“workloadKind“: “Deployment”,
“containers”: {
“*”: {
“cpu“: “250m“,
“memory”: “256Mi”
}
}
}
What mattered was not the JSON itself, but what it made visible at one interface:
- Who produced the recommendation
- What workload it targeted
- What resources it requested
- What policy kind and weight it carried
- Which automation strategy should govern execution
That gave reviewers something concrete to validate. A producer change could be reviewed in terms of the intent it emitted, rather than by re-deriving precedence and execution semantics from the surrounding system.
The PolicyEvaluation reconciler then owned the behavior that used to leak across the rest of the codebase. It selected the winning recommendation, resolved effective automation settings, centralized safety checks, built the resize plan, and chose whether to resize in place, evict, retry, or wait. Producers could still do recommendation-specific filtering as part of what they produced, but final action gating lived in one place.
What real feature pressure showed
A clean boundary only matters if it survives real features. Three cases convinced us this one was doing real work.
ProactivePolicy: existing capability on a cleaner interface
ProactivePolicy proved that the rewrite was not only elegant on a greenfield path. We already had rightsizing automation that was driven by our SaaS machine-learning analytics, so the new architecture had to preserve that capability while moving it onto a more reviewable boundary.
It did. The producer still owned domain-specific work like reading recommendations from our API, filtering stale inputs, and packaging per-container resource intent. But, it stopped short of deciding whether a pod would be resized now, later, or not at all. That made ProactivePolicy the continuity test: existing capability survived the rewrite while moving onto a much cleaner review surface.
StaticPolicy: a small feature that stayed small
StaticPolicy was the simplest proof, and maybe the most useful one. The controller matched workloads by selector, built a rightsizing recommendation from declared resource within the object, attached an automation strategy reference, and wrote that recommendation.
What it did not need was just as important. There was no custom precedence path, no bespoke apply logic, and no duplicated safety code. StaticPolicy showed that adding a producer no longer meant threading behavior through the system; it mostly meant emitting a valid contract.
RollbackPolicy: the failure-path stress test
RollbackPolicy was the strongest test because failure paths expose fake boundaries quickly. Rollback had to deal with one of the messiest parts of the system: what happens after a resize when things go wrong.
In the old design, that kind of logic would have leaked across selection, safety, and apply paths. In the new design, rollback could observe failure, express recovery intent, and let the rest of the system handle that intent through the same bounded flow. That symmetry mattered because failure handling became explicit instead of being buried inside a wide control surface.
It also mattered because RollbackPolicy had state, timing, and unhappy paths. If the boundary had only been clean for happy-path features, rollback would have exposed that immediately. Instead, it fit the same contract and decision flow, which made it the strongest evidence that the architecture was doing real work.
Why this worked better with AI and tests
AI benefits more from narrow responsibilities than from broad cleverness. When a controller owns one thing, contract names are explicit, and downstream semantics are centralized, generated code has fewer ways to create hidden coupling. That does not remove review, but it makes review local again.
Tests benefited from the same boundary. In the old system, tests exercised a broad polling surface, so failures were harder to interpret. In the new architecture, better assertions became possible: a producer emitted the expected recommendation, competing recommendations resolved the expected way, PolicyEvaluation chose the expected action, and the runtime applied it, retried it, or safely waited.
Reviews and tests started validating the same interface. That alignment mattered more than any single test count because it made failures easier to understand and changes easier to trust.
In practice, release cadence improved from about 1.25 to 3.5 major releases per month during comparable active periods. That is not proof that this single change made all the difference, but it is a strong directional signal that AI throughput only became shippable after the boundaries improved.
The lesson we learned
The deeper lesson is broader than Kubernetes. If a system has multiple producers, competing inputs, safety constraints, or ugly failure paths, you need to separate intent production from behavior selection and defend that boundary aggressively.
For producers, the rules are simple:
- Emit explicit intent.
- Include the metadata the arbiter needs.
- Do not smuggle in precedence or apply semantics.
For the arbiter, the rules are just as important:
- Keep precedence explicit and deterministic.
- Centralize safety checks.
- Preserve the same guarantees on failure paths, not only happy paths.
That became our review rule: reject changes that blur the boundary.
Smaller diffs still help, but only when their meaning is local. In systems with competing inputs, centralized safety logic, and AI-accelerated implementation, the thing worth validating is not the diff alone. It is the contract boundary that contains the consequences of the change.
If you want the adjacent part of the same rearchitecture story, see the related post on moving from ConfigMaps to CRDs.
