Configuration
Resource Requests and Limits
Resource requests and limits are the foundational mechanism for controlling CPU and memory allocation within a cluster. They ensure that containers receive sufficient resources to function while preventing rogue processes from starving the entire node. You utilize these configurations to balance application performance, cost efficiency, and overall cluster stability.
Understanding Resource Requests
Resource requests represent the minimum guaranteed amount of CPU and memory a container needs to operate. When the scheduler selects a node for a pod, it calculates the total sum of requests across all containers on that node to ensure the node has enough capacity to satisfy those requirements. This mechanism is crucial for cluster planning; it prevents the scheduler from placing pods on overloaded nodes that cannot physically host them. If a node does not have enough unallocated resources to meet the sum of container requests, the pod will remain in a pending state until capacity becomes available. By setting accurate requests, you provide the system with the metadata required to make intelligent placement decisions, ensuring that critical applications receive the resources they need to boot reliably and maintain baseline performance under load without contention from competing workloads.
resources:
requests:
memory: "256Mi" # Guaranteed RAM at startup
cpu: "500m" # Half a core guaranteed allocationImplementing Resource Limits
While requests define the baseline, limits define the hard ceiling for resource usage. A limit prevents a single container from consuming all the resources of a host node, which would otherwise result in a cascading failure for other pods running on that same machine. Memory limits are particularly strict: if a process attempts to exceed its defined memory limit, the system kernel will terminate the process with an Out of Memory (OOM) error. CPU limits act differently, utilizing a mechanism called Completely Fair Scheduler throttling. When a container exceeds its CPU limit, it is prevented from executing for the remainder of the period until its usage drops below the threshold. Understanding this distinction is vital for production stability, as it allows administrators to create 'noisy neighbor' protection, ensuring that an application memory leak or an runaway loop remains contained within its specific process boundary.
resources:
limits:
memory: "512Mi" # Hard ceiling, process dies if exceeded
cpu: "1000m" # Maximum allowed processing powerQoS Classes and Eviction Priority
The system automatically assigns a Quality of Service (QoS) class to pods based on the relationship between their requests and limits. There are three tiers: Guaranteed, Burstable, and BestEffort. A 'Guaranteed' pod is created when every container has both CPU and memory requests equal to their limits. These are the most stable pods and are the last to be evicted during node-level resource pressure. 'Burstable' pods occur when requests are lower than limits, allowing for flexible usage but making them more susceptible to eviction if the node runs low on resources. 'BestEffort' pods have no requests or limits defined and are the first to be killed when memory exhaustion occurs. Understanding these classifications is essential for architecting highly available systems where critical infrastructure components must be prioritized over transient background batch tasks.
# BestEffort Pod (No resources defined)
# Burstable Pod (Request < Limit)
# Guaranteed Pod (Request == Limit)Managing CPU Throttling Risks
CPU throttling is a frequent cause of latency spikes that are often difficult to diagnose. When a container reaches its CPU limit, the system suspends its ability to execute instructions until the next quota period begins, causing a sudden halt in request processing. This manifests as increased response latency for users, even if the application appears idle on average. To avoid this, developers should avoid setting overly aggressive CPU limits for performance-sensitive applications unless strictly necessary for multi-tenancy isolation. By carefully observing metrics, you can identify if your application is being throttled and adjust your limits to provide a buffer for short, bursty workloads without compromising the throughput of the underlying system. Balancing performance and isolation requires constant monitoring of the specific period metrics to ensure that the scheduler-enforced caps are not inadvertently hindering legitimate application traffic patterns.
resources:
requests:
cpu: "200m" # Low baseline
limits:
cpu: "1000m" # High burst, monitor for throttlingBest Practices for Right-Sizing
Right-sizing containers is an iterative process that requires real-time telemetry data. You should never guess your resource requirements; instead, observe the actual consumption of your pods in production over a representative time window. Start by setting requests slightly above the 90th percentile of typical usage and limits at a level that provides a safety margin for unexpected spikes. Over-provisioning leads to significant financial waste and poor density, while under-provisioning leads to frequent restarts and system instability. Use monitoring tools to visualize the delta between requested and actual usage across your entire fleet. By systematically refining these values, you maximize the efficiency of your hardware, reduce operational costs, and ensure that your applications have exactly the amount of capacity required to function optimally without triggering unnecessary OOM events or CPU throttling during peak traffic periods.
resources:
requests:
memory: "128Mi" # Tune based on actual p90 metrics
limits:
memory: "256Mi" # Provide buffer for spikesKey points
- Requests define the guaranteed minimum resources reserved for a container.
- Limits establish the absolute maximum resources a container can consume.
- Memory limits trigger OOM kills when exceeded, while CPU limits trigger throttling.
- The scheduler uses requests to determine which node has capacity for a new pod.
- Guaranteed QoS pods have requests equal to limits and have the lowest eviction priority.
- BestEffort pods have no defined requests or limits and are evicted first under pressure.
- CPU throttling creates latency spikes that can cause intermittent application performance issues.
- Right-sizing requires monitoring historical usage metrics to balance cost and stability effectively.
Common mistakes
- Mistake: Setting requests equal to limits for all containers. Why it's wrong: This forces the container into the 'Guaranteed' QoS class, which may prevent effective node oversubscription. Fix: Use different values based on actual baseline vs peak load patterns.
- Mistake: Forgetting to set any requests or limits. Why it's wrong: Kubernetes will not reserve resources, potentially leading to node instability and unpredictable scheduling. Fix: Always define resource requests for every container in a pod.
- Mistake: Setting a CPU limit that is too low. Why it's wrong: This causes 'CPU throttling' where processes are paused when they hit the limit, significantly increasing latency. Fix: Increase CPU limits or remove them if the application is latency-sensitive.
- Mistake: Miscalculating memory units. Why it's wrong: Confusing 'Mi' (Mebibytes) with 'M' (Megabytes) leads to unexpected OOM kills. Fix: Use the standard Kubernetes binary suffix 'Mi' for memory and 'm' for millicores.
- Mistake: Assuming limits define hard hardware boundaries. Why it's wrong: Limits are enforced by the Linux kernel via Cgroups; they are not absolute performance guarantees. Fix: Monitor actual application metrics to tune limits rather than guessing.
Interview questions
What are resource requests and limits in Kubernetes, and why do we define them for pods?
Resource requests and limits are the primary mechanisms for managing compute resources in a cluster. A 'request' is the amount of CPU or memory that Kubernetes guarantees to a container; the scheduler uses this value to place the pod on a node with sufficient capacity. A 'limit' is the maximum amount of resource a container can consume. We define these to ensure stable cluster performance, prevent a single container from causing 'noisy neighbor' issues by consuming all host resources, and to assist the scheduler in efficient bin-packing of workloads.
What happens if a container in a Kubernetes pod attempts to exceed its defined memory limit?
If a container attempts to exceed its defined memory limit, the Linux kernel, via the cgroup controller, will trigger an 'Out of Memory' (OOM) kill. The container will be terminated immediately to protect the host node from crashing. Kubernetes will then notice the container has terminated and, depending on the Pod's restart policy, will attempt to restart it. If the application continues to exceed its memory limit, the pod will enter a 'CrashLoopBackOff' state, signaling that the allocated memory resources are insufficient for the workload requirements.
Explain how Kubernetes handles CPU limits compared to how it handles memory limits.
The key difference is that CPU is a 'compressible' resource, while memory is 'non-compressible.' If a container hits its CPU limit, Kubernetes does not kill the process; instead, it throttles the container's execution, forcing it to wait for available CPU cycles. This increases latency but keeps the process alive. Conversely, memory is non-compressible; once it is used, it cannot be reclaimed without killing the process. Thus, hitting a memory limit results in an OOM kill, whereas hitting a CPU limit results in performance degradation through throttling.
Compare defining resource requirements at the pod level versus using a LimitRange object.
Defining resources at the pod level gives developers explicit control over individual application needs, ensuring every workload is sized correctly according to its specific profile. However, this is manual and prone to human error. A LimitRange object is a cluster-level policy that automatically applies default requests and limits to pods that lack them, or restricts the min/max values developers can set. While pod-level definition provides precision, LimitRange acts as a safety net to ensure that no pod enters the cluster without baseline resource constraints, preventing resource starvation across the namespace.
Why is it considered a best practice to set resource requests equal to resource limits for production workloads?
Setting requests equal to limits creates a 'Guaranteed' Quality of Service (QoS) class for the pod. By doing this, you ensure the pod is allocated exactly the resources it asks for, and those resources are not oversubscribed on the node. This provides predictable performance and prevents the pod from being evicted during node pressure scenarios. If requests are lower than limits (Burstable QoS), the pod might be granted excess capacity during idle times, but it risks being throttled or evicted during periods of high contention, making 'Guaranteed' the safest choice for critical production services.
How would you determine the appropriate resource requests and limits for an application that you are deploying to Kubernetes for the first time?
I would start by deploying the application without hard-coded limits and monitoring its performance using tools like the Metrics Server or Prometheus to observe actual usage patterns over time. After gathering data on average and peak consumption, I would set my requests slightly above the average usage to ensure efficient scheduling and set my limits slightly above the observed peak to provide a buffer for transient spikes. Over time, I would iterate on these values based on real-world telemetry, ensuring the settings balance cost-efficiency with application stability and the risk of OOM kills.
Check yourself
1. What happens to a pod if the container exceeds its memory limit?
- A.The kernel throttles the CPU usage until memory usage drops
- B.Kubernetes automatically increases the memory limit
- C.The container is terminated and potentially restarted with an OOMKilled status
- D.The pod is evicted to another node with more memory
Show answer
C. The container is terminated and potentially restarted with an OOMKilled status
Memory is a non-compressible resource; the kernel cannot 'throttle' it, so it must terminate the process. Throttling only applies to CPU. Kubernetes does not auto-scale limits, and eviction is triggered by node-level pressure, not single-container limits.
2. If a pod requests 500m CPU and sets a limit of 1000m, what does this guarantee?
- A.The pod will always get 1000m CPU at all times
- B.The pod is guaranteed 500m CPU, and can burst up to 1000m if available
- C.The pod will be throttled if it exceeds 500m CPU
- D.The pod can use all available CPU on the node regardless of the 1000m limit
Show answer
B. The pod is guaranteed 500m CPU, and can burst up to 1000m if available
Requests are reserved resources. Limits define the maximum cap. The container is guaranteed its request, but may access unused node CPU up to the limit. Throttling only occurs when exceeding the 1000m limit.
3. Which of the following describes the 'Burstable' Quality of Service class?
- A.Requests are equal to limits for all containers in the pod
- B.No requests or limits are set for any container in the pod
- C.The pod has at least one request/limit set, but they are not equal
- D.The pod is only assigned to nodes with unlimited CPU resources
Show answer
C. The pod has at least one request/limit set, but they are not equal
Burstable occurs when requests are defined but not equal to limits. 'Guaranteed' requires all requests to equal limits. 'BestEffort' is when no requests or limits exist. Nodes do not have 'unlimited' resources.
4. Why is it recommended to set CPU requests even if you don't strictly require a minimum amount?
- A.To allow the Kubernetes scheduler to place the pod on a node with sufficient capacity
- B.To prevent the node from entering a 'Ready' state
- C.To ensure the application uses more CPU than it actually needs
- D.To bypass the need for memory requests
Show answer
A. To allow the Kubernetes scheduler to place the pod on a node with sufficient capacity
The scheduler uses requests to calculate the remaining capacity of a node. Without requests, the scheduler cannot account for the pod's footprint, leading to node overloading. This has no effect on node state or CPU consumption efficiency.
5. What is the primary difference between how CPU and memory are managed when limits are reached?
- A.CPU is a compressible resource that can be throttled; memory is incompressible and leads to termination
- B.Memory is a compressible resource; CPU is incompressible
- C.Both are managed by terminating the process when the limit is reached
- D.Neither resource is actually enforced by the kernel
Show answer
A. CPU is a compressible resource that can be throttled; memory is incompressible and leads to termination
CPU can be restricted by time-slicing (throttling), while memory cannot be partially allocated or throttled without crashing the process. Termination is the standard outcome for memory, but not for CPU.