When a Kubernetes rolling update fails, the fix follows a consistent sequence: check rollout status, inspect pod events and logs for the root cause, roll back to restore service, then patch the issue and redeploy. This post walks through each step and shows how AI tools can speed up the diagnosis in 2026.
What Actually Happens During a Rolling Update?
A Kubernetes rolling update replaces pods gradually. Old pods are terminated only as new pods pass their readiness probes and start serving traffic. As the Semaphore blog explains, the rolling update strategy ensures there are always pods available to handle requests during the transition.
Two settings control the pace. maxSurge defines how many extra pods Kubernetes can spin up beyond the desired count. maxUnavailable defines how many pods can be down at the same time. Together, they determine whether your update is cautious (zero downtime, slower) or aggressive (faster, brief capacity dip).
When everything works, you see pods cycling smoothly from old to new. When something breaks, new pods fail to become ready, old pods stay running, and the rollout stalls. Kubernetes gives a Deployment 10 minutes by default before marking it as failed. That is the progressDeadlineSeconds timer.
How Do You Diagnose a Failed Rolling Update?
The diagnostic sequence is the same every time. Start broad, then narrow down.
Step 1: Check rollout status. Run kubectl rollout status deployment/your-app. If it is stuck, you will see a message like "Waiting for deployment to finish: 1 old replicas are pending termination."
Step 2: Describe the deployment. Run kubectl describe deployment/your-app. Look at the Events section at the bottom. It tells you whether new ReplicaSets were created and whether pods failed to schedule or start.
Step 3: Inspect individual pods. Run kubectl get pods to find pods in CrashLoopBackOff, ImagePullBackOff, or Pending states. Then run kubectl describe pod/pod-name to see the specific error. Common causes include wrong image tags, missing secrets or ConfigMaps, failing health probes, and OOMKilled containers.
Step 4: Read container logs. Run kubectl logs pod-name --previous to see logs from the crashed container. The application-level error (a missing database connection, a config parse failure, a port conflict) usually shows up here.
This is where most guides stop. But in 2026, there is a faster path through this diagnostic loop.
How Can AI Tools Speed Up Kubernetes Debugging?
Here is the practical bridge between DevOps troubleshooting and the AI workflows we cover on this site. Instead of manually parsing dense kubectl output and searching Stack Overflow, paste the output directly into an AI coding agent.
Copy the full output of kubectl describe pod and kubectl logs --previous into ChatGPT or Claude. The model parses the error messages, identifies the misconfiguration, and suggests a specific fix. I have seen this cut diagnosis time from 30+ minutes of reading documentation to under 5 minutes of targeted conversation.
This is one of the ChatGPT workflows that saves real hours every week, and it works because Kubernetes errors are structured and well-documented. AI models have absorbed the entire Kubernetes docs, thousands of GitHub issues, and years of Stack Overflow answers.
For teams building and deploying apps fast (like the approach in how to build a working app in 7 days using only AI), Kubernetes failures are inevitable. Having an AI assistant that can parse a wall of YAML and event logs is not a luxury. It is basic workflow hygiene.
If you want to go further, some of the best AI coding agents in 2026 can directly edit your Kubernetes manifests, fix probe configurations, and generate corrected deployment files. The feedback loop shrinks from "fail, search, read, guess, redeploy" to "fail, paste, fix, redeploy."
What Should You Do After Rolling Back?
Rolling back is first aid, not a cure. Run kubectl rollout undo deployment/your-app to revert to the last working revision. The Kubernetes official docs note that each rollback updates the revision of the Deployment, so your history stays clean.
After service is restored, fix the actual problem. The most common root causes fall into a few buckets:
- Image issues. Wrong tag, private registry without pull secrets, or a broken build that produced a non-functional image.
- Configuration drift. A new environment variable or secret that the updated code expects but the cluster does not have yet.
- Probe failures. The new version changed its startup time or health endpoint, but the liveness and readiness probes still point to the old path or use the old timing.
- Resource limits. The new version uses more memory and gets OOMKilled because the resource limits were not updated.
Fix the root cause in your manifests or CI pipeline, test locally or in staging, then redeploy. Google's GKE documentation recommends treating rolling updates as incremental replacements scheduled on nodes with available resources, which means resource planning matters as much as the code itself.
Which Pitfalls Cause Repeated Rolling Update Failures?
Some teams fix the immediate error but keep hitting rolling update failures because of structural problems in their deployment workflow.
No staging environment. If you only test in production, every deployment is a gamble. Even a basic staging namespace in the same cluster catches most configuration issues before they affect users.
Probes copied from a template without tuning. Default liveness probe settings (check every 10 seconds, fail after 3 attempts) may be too aggressive for apps with slow startup. Set initialDelaySeconds high enough for your app to actually start.
No resource requests or limits. Without resource requests, Kubernetes cannot schedule pods effectively. Without limits, a memory leak in one pod can starve others. Both cause rolling updates to stall.
Revision history too short. The default revisionHistoryLimit is 10, which is usually fine. But if your team deploys many times per day, you might burn through that history fast and lose the ability to roll back to a known-good state.
For newcomers to AI and cloud concepts, our plain-English explainer on generative AI covers the foundations. Understanding how these models work helps you use them more effectively for debugging tasks like the ones described here. The full AI for beginners pillar on this site walks through more of these foundational concepts at the same level of approachability.
What Does a Complete Debugging Workflow Look Like?
Kubernetes rolling update failures follow predictable patterns, and so does fixing them: status, describe, logs, fix. The 2026 difference is that AI tools compress the "understand what went wrong" step from minutes or hours to seconds. Paste your kubectl output, get a targeted diagnosis, fix, redeploy.
We talk about practical AI workflows like this inside AI Masterminds, where builders share what actually works in production. If you are shipping code and want to get better at using AI tools across your stack, join AI Masterminds.
FAQ
How long does Kubernetes wait before marking a rolling update as failed?
By default, Kubernetes uses a progressDeadlineSeconds value of 600 seconds (10 minutes). If no new pods become ready within that window, the Deployment condition is marked as failed. You can adjust this value in your Deployment spec, but setting it too low causes false failures on slow-starting apps. Setting it too high delays your feedback loop. For most production workloads, 5 to 15 minutes is a reasonable range depending on image size and startup time.
What is the difference between maxSurge and maxUnavailable in Kubernetes?
maxSurge controls how many extra pods Kubernetes can create above the desired replica count during an update. maxUnavailable controls how many pods can be offline at once. For example, with 4 replicas, maxSurge of 1, and maxUnavailable of 1, Kubernetes will have at most 5 pods running and at least 3 available at any time. Setting maxUnavailable to 0 ensures zero downtime but slows the rollout. These two settings together define how aggressively Kubernetes replaces old pods with new ones.
Can I use ChatGPT or Claude to debug a failed Kubernetes deployment?
Yes, and it works surprisingly well. Copy the output of kubectl describe pod, kubectl get events, or kubectl logs and paste it into ChatGPT, Claude, or any AI coding agent. The model can parse error messages, identify misconfigurations like wrong image tags or failing health probes, and suggest specific fixes. This is one of the most practical AI workflows for DevOps in 2026, cutting diagnosis from hours of Stack Overflow searching to minutes of targeted conversation.
How do I roll back a failed Kubernetes deployment?
Run kubectl rollout undo deployment/your-deployment-name to revert to the previous working revision. Kubernetes keeps a revision history (default 10 revisions) so you can also roll back to a specific version with the --to-revision flag. Always verify the rollback succeeded by running kubectl rollout status afterward. Rolling back restores service immediately but does not fix the root cause, so treat it as a first-aid step, not a solution.
Why do Kubernetes pods keep restarting after a rolling update?
The most common causes are failing liveness or readiness probes, application crashes on startup (often due to missing environment variables or secrets), image pull errors from a wrong tag or missing registry credentials, and resource limits that are too tight causing OOMKilled events. Check kubectl describe pod for the exact reason under the Events section. The container's last state will show whether it was OOMKilled, failed a probe, or hit an error exit code. Each cause has a different fix.
Sources
- Deployments | Kubernetes · Kubernetes Official Documentation
- How to Rollback a Deployment in Kubernetes · LearnKube
- Updating Applications, GKE Docs · Google Cloud Documentation
- Kubernetes Deployments: A Guide to the Rolling Update Deployment Strategy · Semaphore Blog