🚨 Why One Kubernetes Node Hit 96% Utilization While Another Sat at 48%

Why Kubernetes cluster can have available capacity and still struggle under production traffic.

We recently investigated an environment where application performance was degrading even though the cluster wasn't completely out of resources.

When we checked the worker nodes, the difference was immediately visible.

🔴 Node 1 — 96% utilization

🟡 Node 2 — 78% utilization

🟢 Node 3 — 48% utilization

One worker was approaching saturation while another still had significant capacity available.

The cluster was already configured to autoscale between 2 and 10 nodes.

So instead of immediately adding more infrastructure, we needed to answer a different question:

Why was one worker reaching 96% while another was sitting at 48%?

🔍 We Started With Pod Placement

The first thing we checked was where the application pods were actually running.

kubectl get pods -n <namespace> -o wide

Node-level CPU metrics tell us how busy a worker is.

They don't tell us which workloads are creating that pressure.

Looking at pod placement gave us the missing part of the picture.

A common assumption is that three application replicas running on a three-node cluster will automatically result in one replica per node.

Kubernetes doesn't guarantee that.

For example:

replicas: 3

Without additional scheduling rules, Kubernetes may place multiple replicas of the same application on one worker if that placement satisfies the scheduler.

That can create a hotspot when those replicas become busy at the same time.

It can also reduce resilience because losing one worker could affect multiple replicas of the same application.

📊 Requests vs. Actual Usage

Next, we compared actual resource consumption.

kubectl top nodes
kubectl top pods -n <namespace>

We also reviewed the CPU and memory requests configured for the application.

resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"

These values are examples. Production requests and limits should be based on actual workload behavior.

This check matters because Kubernetes doesn't simply look at the current CPU percentage and send the next workload to the least-utilized node.

Resource requests are an important input used by the scheduler when deciding where a new pod can run.

And Kubernetes doesn't normally move a healthy running pod just because another worker currently has lower CPU utilization.

So seeing:

Node 1 → 96%

and

Node 3 → 48%

doesn't mean Kubernetes will automatically rebalance the two.

💡 Current utilization and Kubernetes scheduling capacity are related, but they are not the same thing.

🎯 What We Found

The important finding was that this wasn't simply a question of total cluster capacity.

It was also a workload-placement problem.

Three different Kubernetes mechanisms were involved:

📈 HPA
Controls how many application replicas should be running.

⚙️ Kubernetes Scheduler
Decides which worker node a new pod should run on.

🖥️ Cluster Autoscaler
Adds or removes worker capacity when required.

These components work together, but they don't perform the same job.

HPA can create additional replicas when application load increases.

But HPA doesn't decide where those replicas run.

Cluster Autoscaler can provide additional nodes when the cluster needs capacity.

But adding another worker doesn't automatically redistribute every healthy pod already running on a busy node.

Pod placement remains a scheduling decision.

🛠️ Improving the Scheduling Strategy

For replicated workloads that should be distributed across workers, one useful Kubernetes mechanism is Topology Spread Constraints.

For example:

topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: backend

The important setting here is:

topologyKey: kubernetes.io/hostname

This tells Kubernetes that individual worker nodes are the topology domains we care about when distributing matching replicas.

Without an explicit distribution strategy, placement could look like:

Node 1 Node 2 Node 3

● ● ● ● -

With an appropriate topology spread policy, the scheduler can work toward:

Node 1 Node 2 Node 3

● ● ● ●

The objective isn't perfect CPU balance.

The objective is to avoid unnecessary concentration of matching replicas on the same worker.

⚖️ What Does maxSkew: 1 Do?

maxSkew controls how uneven the distribution of matching pods can become across eligible topology domains.

maxSkew: 1

In practical terms, we're telling the scheduler to keep the replica distribution reasonably close where possible.

🔄 Why ScheduleAnyway?

We also use:

whenUnsatisfiable: ScheduleAnyway

This makes the spread policy a preference rather than an absolute scheduling requirement.

Kubernetes should try to achieve a better distribution, but a pod doesn't necessarily need to remain Pending when the ideal distribution cannot be achieved.

For workloads where strict separation is required, DoNotSchedule may be more appropriate.

The choice depends on the application's availability requirements and cluster topology.

💾 Scheduling Rules Aren't Enough

Topology spread is only part of the solution.

Resource requests still need to represent realistic workload requirements.

If a container normally consumes around 800m CPU but only requests 100m, Kubernetes sees a much smaller workload when making scheduling decisions.

That can contribute to workloads being packed more aggressively onto nodes.

The opposite is also possible.

Requests that are unnecessarily high can make pods difficult to schedule even when runtime utilization appears to show available capacity.

A healthy scheduling strategy therefore needs several pieces working together:

Realistic resource requests

↓

Appropriate replica distribution

↓

Application autoscaling

↓

Cluster capacity autoscaling

💡 Good Kubernetes scheduling starts with giving the scheduler accurate information.

🧪 Validating the Change

After changing workload scheduling, we don't stop when Kubernetes reports:

✅ Running

We check where the new pods actually landed.

kubectl get pods -n <namespace> -o wide

Then we continue monitoring the nodes and application workloads.

kubectl top nodes
kubectl top pods -n <namespace>

We also observe what happens when traffic increases and HPA creates additional replicas.

The complete flow is:

Traffic ↑ → HPA → New Replicas → Scheduler → Topology Spread → Worker Nodes

If additional cluster capacity is required, Cluster Autoscaler handles that separately.

⚖️ Balanced Doesn't Mean Equal

There is one important point when evaluating the result.

We're not trying to turn:

🔴 96% / 🟡 78% / 🟢 48%

into:

🟢 70% / 🟢 70% / 🟢 70%

Kubernetes workers don't need identical utilization.

Different workloads receive different traffic and consume different amounts of resources.

Some variation is completely normal.

What we want to avoid is a persistent hotspot where one worker repeatedly approaches resource exhaustion while other suitable workers still have significant available capacity.

💡 The Engineering Takeaway

A Kubernetes node reaching 96% utilization is a signal — not automatically the root cause.

Before adding nodes or increasing instance sizes, check:

🔍 Where are the pods running?

📊 What are they actually consuming?

💾 What resources did they request?

⚙️ How are replicas being scheduled?

📈 How is HPA behaving?

🖥️ Does the cluster genuinely need more capacity?

The answer will tell you whether you're dealing with a capacity problem, scheduling problem, application problem — or a combination of them.

🚀 Observe → Investigate → Understand → Fix → Validate

More capacity can reduce pressure. Better workload placement addresses why that pressure may be concentrated in the first place.