Kubernetes Cost Creep: Where It Hides

A client called me in for a monthly cost review. Their AWS bill was up eleven percent from the month before. No new product launch, no new customer segment, no incident anyone could point to. Just up.

We went line by line. Nobody in the room could explain more than about half of it. "It's probably just growth," someone offered, in the tone people use when they've stopped looking for the real answer.

It wasn't growth. It was five separate small leaks, none of them dramatic enough on its own to trigger an alert, all of them compounding quietly for months. Here they are, in the order I usually find them.

Where Kubernetes cost creep actually goes: a breakdown of the five most common sources of quiet spend

Oversized resource requests

This is almost always the biggest single number, and almost always the least visible. Requests drive scheduling and autoscaling decisions - not actual usage. A pod requesting 2 CPU and using 200m still reserves 2 full CPU worth of node capacity for the life of that pod. Multiply that by a few hundred pods across a cluster that's been growing for a year, and you're paying for phantom headroom nobody remembers reserving.

Nobody sets out to overprovision. It happens one "let's be safe" request at a time, and none of those requests ever get revisited.

The fix isn't clever, it's just unglamorous: pull real usage over a representative window and right-size against it, not against a guess made at deploy time.

# What's actually requested vs what's actually used, per pod
kube_pod_container_resource_requests{resource="cpu"}
  - on(pod, namespace) group_left
  avg_over_time(container_cpu_usage_seconds_total[7d])

Anything with a large, sustained gap on that query is a pod that's been quietly overprovisioned since the day someone typed a round number into a YAML file and moved on.

Idle nodes that never scale down

Cluster autoscalers are confident about scaling up and cautious about scaling down. A single pod with a strict pod disruption budget, or one with local storage the autoscaler won't evict, can pin an entire node in place for weeks. I've seen a three-replica logging DaemonSet keep a whole node group alive through two full sprints because nobody noticed the disruption budget was tighter than it needed to be.

One stuck pod, one stuck node: a single pod disruption budget keeping an entire node alive for weeks

Check your autoscaler's logs for the nodes it wanted to remove and couldn't - not just the ones it did remove. That list, not the removal log, is where the money usually is.

Orphaned persistent volumes

Deleting a workload doesn't always delete its storage. Depending on the reclaim policy, a persistent volume can outlive the pod and the PVC that created it, quietly billing block storage for data nobody has read in months. This one is the easiest to find and the one teams check least often, because it doesn't show up in any dashboard that assumes storage tracks workloads one-to-one.

The audit is simple: list every PV, cross-reference against active PVCs, and flag anything unattached for more than a sprint. It takes twenty minutes and it's rarely zero.

Cross-AZ and cross-region traffic

Data transfer between availability zones is cheap per gigabyte and expensive in aggregate. A chatty service mesh with no topology-aware routing or zone affinity can rack up transfer costs that never appear as one line item anyone questions - because it's spread across dozens of small charges that each look too small to investigate.

This is the leak that survives audits, because auditing for it means noticing something that isn't there: a single large number to be suspicious of.

Autoscaling with no ceiling

A horizontal pod autoscaler with no sensible max, paired with a cluster autoscaler with no sensible max, will scale exactly as far as a bug tells it to. I've watched a single retry loop - a downstream dependency timing out, retried without backoff - trigger a scale-up that ran for six hours before anyone caught it. By the time someone did, the cluster had briefly rented enough capacity to run a service ten times its normal size, to serve traffic that never grew at all.

The autoscaler with no ceiling: a retry loop triggering an unbounded six-hour scale-up with no real traffic behind it

Set real ceilings, everywhere. An autoscaler with no limit isn't protecting you from underprovisioning. It's removing the one guardrail that would have caught the incident automatically, while it was still small.

What I told that client

None of these five are individually hard to fix. The eleven percent wasn't one big mistake - it was five small, unmonitored ones, each easy to justify in isolation and none of them ever revisited. The fix wasn't a new tool. It was putting a recurring audit against each of these five questions on a calendar, instead of waiting for a bill to be the thing that made us look.

If your bill went up and nobody can point to why, it's worth going down this list before assuming it's growth. It usually isn't.