It is 02:10. You are paged: three of your twenty worker nodes have flipped to NotReady, pods are being evicted onto the survivors, and now new pods are stuck Pending. Latency on the main API is climbing. Take me through your first thirty minutes.

Question

Accepted Answer

The first thing I say in the channel is that this is a cascade, not three broken nodes, because pods evicted from those three are landing on the seventeen survivors and pushing them toward the same edge. So I stabilise capacity before I ask why. I raise the node group desired count, say from 20 to 28, which on EKS gives me new nodes joining in roughly 3 to 4 minutes, and I cordon any node whose memory pressure or disk pressure condition is already true so nothing new schedules onto a node that is about to flip. If Pending pods are the API itself, I check that the API deployment has a priority class above the batch workloads so the scheduler evicts the right things. That is the first 5 minutes and it is what stops the bleeding. Then blast radius. I check whether the three nodes share an availability zone, an instance type, or an AMI, because that one query usually names the cause. Three nodes in one AZ points at the network or the control plane path. Three nodes on the same new AMI points at the last node group upgrade. I keep exactly one NotReady node cordoned and alive as evidence, and I let the others terminate. On that one I check kubelet status, disk usage on /var/lib, and whether the kubelet lost its connection to the API server. Once Ready count is back to 20 or more and Pending is zero and API p99 is back under 300ms, I write it up.

Answer

All three nodes are in the same availability zone. What does that tell you?

Stop looking at the nodes and start looking at the zone. Check the control plane path, the NAT gateway or the CNI in that AZ, and consider cordoning the whole zone so the scheduler stops placing work there.

Answer

You scale the node group up but the new nodes never join. Now what?

That is a bigger signal than the original page. Check the node group's quota, the AMI, the bootstrap script and the kubelet certificate. It means your capacity lever is broken, so switch to shedding load, scale batch to zero and protect the API.

Answer

How do you keep evictions from making it worse next time?

PodDisruptionBudgets on the critical services, priority and preemption classes so batch loses first, topology spread constraints across zones, and resource requests that actually reflect usage so the scheduler does not overpack.

It is 02:10. You are paged: three of your twenty worker nodes have flipped to NotReady, pods are being evicted onto the survivors, and now new pods are stuck Pending. Latency on the main API is climbing. Take me through your first thirty minutes.

It is 02:10. You are paged: three of your twenty worker nodes have flipped to NotReady, pods are being evicted onto the survivors, and now new pods are stuck Pending. Latency on the main API is climbing. Take me through your first thirty minutes.

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with troubleshooting

Knowing the answer is not the same as recalling it under pressure