Kubernetes ResourceQuota: Fix Pod Scheduling Failure Blocking CI/CD Pipeline Deployments
Quick Fix Summary
TL;DRCheck and increase ResourceQuota limits for the affected namespace.
A Pod cannot be scheduled because the namespace's ResourceQuota has exhausted its CPU or memory allocation, preventing new workloads from starting.
Diagnosis & Causes
Recovery Steps
Step 1: Verify the Pod Scheduling Error
Confirm the Pod is failing due to ResourceQuota by checking its events.
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'Step 2: Check Active ResourceQuotas in the Namespace
List all ResourceQuotas and their current usage to identify the constrained resource.
kubectl get resourcequotas -n <namespace> -o wide
kubectl describe resourcequota -n <namespace>Step 3: Analyze Total Resource Usage in the Namespace
Sum the resource requests of all running pods to understand quota consumption.
kubectl get pods -n <namespace> -o jsonpath='{range .items[*]}{.spec.containers[*].resources.requests.cpu}{"\n"}{end}' | awk '{s+=$1} END {print "Total CPU Requests:", s}'
kubectl get pods -n <namespace> -o jsonpath='{range .items[*]}{.spec.containers[*].resources.requests.memory}{"\n"}{end}' | awk '{s+=$1} END {print "Total Memory Requests:", s}'Step 4: Identify and Clean Up Unnecessary Pods (Immediate Relief)
Delete failed, completed, or non-essential pods to free up quota.
kubectl get pods -n <namespace> --field-selector=status.phase!=Running
kubectl delete pod <pod-name> -n <namespace>Step 5: Update the ResourceQuota Limits
Increase the hard limits of the constraining ResourceQuota.
kubectl edit resourcequota <quota-name> -n <namespace>
# In the editor, increase values for 'limits.cpu', 'requests.cpu', etc.Step 6: Scale Down Existing Deployments (Alternative)
Temporarily reduce replica counts of non-critical deployments to free quota for the new pod.
kubectl scale deployment <deployment-name> -n <namespace> --replicas=<lower-number>Step 7: Verify Pod Schedules Successfully
After making changes, monitor the pod's status to confirm it transitions to Running.
kubectl get pod <pod-name> -n <namespace> -w
kubectl logs <pod-name> -n <namespace> --tail=50Step 8: Implement Preventive Monitoring
Set up alerts for ResourceQuota usage to prevent future pipeline blocks.
# Example: Create a Prometheus alert rule for high quota usage.
# - alert: NamespaceQuotaHighUsage
# expr: kube_resourcequota_created * on(namespace) group_left kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.8
# for: 5mArchitect's Pro Tip
"This often happens when CI/CD pipelines deploy to a shared development namespace. A single team or a runaway job can consume the entire quota, blocking all other deployments. Consider implementing per-team or per-pipeline namespaces with dedicated quotas."
Frequently Asked Questions
Can I bypass a ResourceQuota by deploying to a different namespace?
Yes, ResourceQuotas are namespace-scoped. Deploying to a namespace without a quota or with higher limits is a valid workaround, but it bypasses the intended resource governance.
Why does my pod fail even though `kubectl top nodes` shows plenty of resources?
ResourceQuotas enforce limits at the namespace level, independent of actual node capacity. A namespace can be 'full' even if the cluster has free resources.
What's the difference between `requests` and `limits` in a ResourceQuota?
A quota for `requests.cpu` governs the sum of all pod resource requests (guaranteed minimum). A quota for `limits.cpu` governs the sum of all pod resource limits (maximum allowed). A pod must satisfy both to be scheduled.