Kubernetes · lesson 9 of 10
Intermediate
6Advanced
1Also part of Kubernetes
5- + Deploying FastAPI with Docker and Kubernetes: Images, Signals, and Probes from FastAPI
- + Deployment Strategies Compared: Rolling, Blue-Green, and Canary from DevOps
- + GitOps in Practice: Git as the Source of Truth from DevOps
- + Kubernetes Pod Security Admission: Enforcing the Restricted Profile from Cloud Security
- + CIS Kubernetes Benchmark Checks with kube-bench in kind from Cloud Security
Horizontal Pod Autoscaling: Real Replicas, Real CPU Numbers
Deploy a CPU-hungry container behind an HPA and watch replica count climb in real time as metrics-server reports real utilization — no simulated numbers, just kubectl get hpa polled while it happens.
By SkillByExample · Published
Every earlier lesson in this track sets a fixed replicas: count and leaves it there.
HorizontalPodAutoscaler changes that number for you, based on metrics metrics-server
collects for real from running pods.
A deployment that requests less CPU than it uses
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cpu-app
spec:
replicas: 1
selector:
matchLabels:
app: cpu-app
template:
metadata:
labels:
app: cpu-app
spec:
containers:
- name: cpu-app
image: vish/stress
args: ["-cpus", "1"]
resources:
requests:
cpu: "100m"
limits:
cpu: "200m"
vish/stress -cpus 1 is a small reference image that pegs one CPU core flat out — deliberately
chosen so this lesson doesn’t need a separate load generator. It requests 100m but its
limit is 200m, so once running it will actually consume close to 200m: 200% of what it asked
for.
kubectl apply -f deployment.yaml
deployment.apps/cpu-app created
Enabling metrics-server (minikube)
The HPA controller doesn’t measure CPU itself — it reads aggregated metrics from
metrics-server, which minikube ships as an addon:
minikube addons enable metrics-server
* metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
- Using image registry.k8s.io/metrics-server/metrics-server:v0.8.1
* The 'metrics-server' addon is enabled
Applying the HorizontalPodAutoscaler
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: cpu-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: cpu-app
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
kubectl apply -f hpa.yaml
kubectl get hpa cpu-app
horizontalpodautoscaler.autoscaling/cpu-app created
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
cpu-app Deployment/cpu-app cpu: <unknown>/50% 1 5 0 4s
<unknown> — metrics-server hasn’t completed its first scrape of the new pod yet. This is
expected, not a misconfiguration.
Watching real replicas climb
Polling the same command over the next minute, with no changes made by hand:
kubectl get hpa cpu-app
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
cpu-app Deployment/cpu-app cpu: 201%/50% 1 5 1 28s
kubectl get hpa cpu-app
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
cpu-app Deployment/cpu-app cpu: 201%/50% 1 5 4 37s
kubectl get hpa cpu-app
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
cpu-app Deployment/cpu-app cpu: 201%/50% 1 5 5 47s
Real utilization (201%) never changes across these three checks — each vish/stress pod uses
the same ~200m regardless of how many replicas exist. What changes is REPLICAS: 1 → 4 → 5 in
under 20 seconds, the controller adding pods each sync cycle until it hits maxReplicas: 5 and
stops, still 201% over target.
What actually happened, from the HPA’s own events
kubectl describe hpa cpu-app
Name: cpu-app
Namespace: default
Reference: Deployment/cpu-app
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 201% (201m) / 50%
Min replicas: 1
Max replicas: 5
Deployment pods: 5 current / 5 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 37s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 22s horizontal-pod-autoscaler New size: 5; reason:
Two real SuccessfulRescale events, not one — the controller took two steps to get from 1 to
5 rather than jumping straight there, a deliberate rate limit on how fast it scales up in one
cycle.
Practice
1. Lower averageUtilization to 10 and re-apply. Does it change the final replica count?
No — maxReplicas: 5 is still the ceiling. A lower target would want more replicas (further
over target), but the HPA still can’t exceed maxReplicas, so it converges to 5 either way;
only the target percentage shown in kubectl get hpa changes.
2. Set minReplicas and maxReplicas to the same value. What happens to the HPA?
kubectl patch hpa cpu-app -p '{"spec":{"minReplicas":3,"maxReplicas":3}}'
kubectl get hpa cpu-app
horizontalpodautoscaler.autoscaling/cpu-app patched
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
cpu-app Deployment/cpu-app cpu: <unknown>/50% 3 3 3 47s
REPLICAS jumps straight to 3 — the Deployment scales to meet minReplicas immediately,
independent of whatever CPU data exists. TARGETS goes back to <unknown> for a stretch,
though: the patch created brand-new pods (old ones were replaced to satisfy the new minimum),
and those new pods need their own fresh metrics-server scrape before utilization shows up
again — the same <unknown> warm-up from this lesson’s very first kubectl get hpa repeats
itself here, this time triggered by a config change rather than a first-ever creation.
3. Delete the HPA but leave the Deployment running. What happens to replica count?
Nothing changes — deleting the HPA removes the controller managing the replica count, not the
replicas themselves. The Deployment stays at whatever count the HPA last set (5, from this
lesson) until something else changes it, exactly like editing replicas: by hand and then
never touching it again.
4. Delete the Deployment (removing all load) and time how long it takes the HPA's scale-down stabilization window to matter — recreate it at low load and watch replicas *shrink*.
Re-apply deployment.yaml with args: ["-cpus", "0"] swapped in (idle, near-zero CPU) instead
of ["-cpus", "1"], apply the same hpa.yaml, and watch kubectl get hpa cpu-app every 30
seconds. Utilization should drop toward 0% almost immediately, but REPLICAS won’t drop from
whatever minReplicas allows until roughly 5 minutes have passed with sustained low
utilization — the scale-down stabilization window this lesson’s FAQ mentions, timed for
yourself rather than taken on faith.
5. What real number would you expect if averageUtilization were set to 500 instead of 50, given this lesson's pod actually uses ~201%?
kubectl get hpa would show cpu: 201%/500% and REPLICAS would stay at minReplicas: 1 —
201% is already below a 500% target, so the HPA has no reason to scale up at all. This is the
inverse of the lesson’s own setup: the gap between usage and target, not the absolute CPU
number, is what drives the decision either way.
Next: packaging this same Deployment and Service as a real Helm chart, instead of hand-applying individual YAML files.