Skip to main content
Kubernetes · lesson 9 of 10
Kubernetes intermediate Lesson 9 of 10

Horizontal Pod Autoscaling: Real Replicas, Real CPU Numbers

Deploy a CPU-hungry container behind an HPA and watch replica count climb in real time as metrics-server reports real utilization — no simulated numbers, just kubectl get hpa polled while it happens.

By SkillByExample · Published

kubernetes hpa autoscaling metrics-server

Every earlier lesson in this track sets a fixed replicas: count and leaves it there. HorizontalPodAutoscaler changes that number for you, based on metrics metrics-server collects for real from running pods.

A deployment that requests less CPU than it uses

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cpu-app
  template:
    metadata:
      labels:
        app: cpu-app
    spec:
      containers:
        - name: cpu-app
          image: vish/stress
          args: ["-cpus", "1"]
          resources:
            requests:
              cpu: "100m"
            limits:
              cpu: "200m"

vish/stress -cpus 1 is a small reference image that pegs one CPU core flat out — deliberately chosen so this lesson doesn’t need a separate load generator. It requests 100m but its limit is 200m, so once running it will actually consume close to 200m: 200% of what it asked for.

kubectl apply -f deployment.yaml
deployment.apps/cpu-app created

Enabling metrics-server (minikube)

The HPA controller doesn’t measure CPU itself — it reads aggregated metrics from metrics-server, which minikube ships as an addon:

minikube addons enable metrics-server
* metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
  - Using image registry.k8s.io/metrics-server/metrics-server:v0.8.1
* The 'metrics-server' addon is enabled

Applying the HorizontalPodAutoscaler

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: cpu-app
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
kubectl apply -f hpa.yaml
kubectl get hpa cpu-app
horizontalpodautoscaler.autoscaling/cpu-app created
NAME      REFERENCE            TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
cpu-app   Deployment/cpu-app   cpu: <unknown>/50%   1         5         0          4s

<unknown> — metrics-server hasn’t completed its first scrape of the new pod yet. This is expected, not a misconfiguration.

Watching real replicas climb

Polling the same command over the next minute, with no changes made by hand:

kubectl get hpa cpu-app
NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
cpu-app   Deployment/cpu-app   cpu: 201%/50%   1         5         1          28s
kubectl get hpa cpu-app
NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
cpu-app   Deployment/cpu-app   cpu: 201%/50%   1         5         4          37s
kubectl get hpa cpu-app
NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
cpu-app   Deployment/cpu-app   cpu: 201%/50%   1         5         5          47s

Real utilization (201%) never changes across these three checks — each vish/stress pod uses the same ~200m regardless of how many replicas exist. What changes is REPLICAS: 1 → 4 → 5 in under 20 seconds, the controller adding pods each sync cycle until it hits maxReplicas: 5 and stops, still 201% over target.

What actually happened, from the HPA’s own events

kubectl describe hpa cpu-app
Name:                                                  cpu-app
Namespace:                                             default
Reference:                                             Deployment/cpu-app
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  201% (201m) / 50%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       5 current / 5 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  37s   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  22s   horizontal-pod-autoscaler  New size: 5; reason:

Two real SuccessfulRescale events, not one — the controller took two steps to get from 1 to 5 rather than jumping straight there, a deliberate rate limit on how fast it scales up in one cycle.

Practice

1. Lower averageUtilization to 10 and re-apply. Does it change the final replica count?

No — maxReplicas: 5 is still the ceiling. A lower target would want more replicas (further over target), but the HPA still can’t exceed maxReplicas, so it converges to 5 either way; only the target percentage shown in kubectl get hpa changes.

2. Set minReplicas and maxReplicas to the same value. What happens to the HPA?
kubectl patch hpa cpu-app -p '{"spec":{"minReplicas":3,"maxReplicas":3}}'
kubectl get hpa cpu-app
horizontalpodautoscaler.autoscaling/cpu-app patched
NAME      REFERENCE            TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
cpu-app   Deployment/cpu-app   cpu: <unknown>/50%   3         3         3          47s

REPLICAS jumps straight to 3 — the Deployment scales to meet minReplicas immediately, independent of whatever CPU data exists. TARGETS goes back to <unknown> for a stretch, though: the patch created brand-new pods (old ones were replaced to satisfy the new minimum), and those new pods need their own fresh metrics-server scrape before utilization shows up again — the same <unknown> warm-up from this lesson’s very first kubectl get hpa repeats itself here, this time triggered by a config change rather than a first-ever creation.

3. Delete the HPA but leave the Deployment running. What happens to replica count?

Nothing changes — deleting the HPA removes the controller managing the replica count, not the replicas themselves. The Deployment stays at whatever count the HPA last set (5, from this lesson) until something else changes it, exactly like editing replicas: by hand and then never touching it again.

4. Delete the Deployment (removing all load) and time how long it takes the HPA's scale-down stabilization window to matter — recreate it at low load and watch replicas *shrink*.

Re-apply deployment.yaml with args: ["-cpus", "0"] swapped in (idle, near-zero CPU) instead of ["-cpus", "1"], apply the same hpa.yaml, and watch kubectl get hpa cpu-app every 30 seconds. Utilization should drop toward 0% almost immediately, but REPLICAS won’t drop from whatever minReplicas allows until roughly 5 minutes have passed with sustained low utilization — the scale-down stabilization window this lesson’s FAQ mentions, timed for yourself rather than taken on faith.

5. What real number would you expect if averageUtilization were set to 500 instead of 50, given this lesson's pod actually uses ~201%?

kubectl get hpa would show cpu: 201%/500% and REPLICAS would stay at minReplicas: 1 — 201% is already below a 500% target, so the HPA has no reason to scale up at all. This is the inverse of the lesson’s own setup: the gap between usage and target, not the absolute CPU number, is what drives the decision either way.

Next: packaging this same Deployment and Service as a real Helm chart, instead of hand-applying individual YAML files.

Frequently Asked Questions

Why did kubectl get hpa show <unknown> right after I created it?
The HorizontalPodAutoscaler controller and metrics-server both run on their own sync loops — metrics-server needs to scrape the pod at least once (its scrape interval is typically 15-60s) before any CPU number exists for the HPA to read. A brand-new pod plus a brand-new HPA almost always shows <unknown> for the first cycle or two; it resolves itself within about 30 seconds without any action from you.
Does the HPA look at CPU usage directly, or something else?
It looks at usage as a percentage of each pod's CPU *request*, not an absolute number. A pod requesting 100m that's actually using 201m reports 201% utilization — that's why setting an unrealistically low request makes a deployment scale aggressively even under modest real load, and why HPA tuning is really about choosing the right request value first.
Why did the deployment stop scaling at 5 replicas instead of continuing to grow?
maxReplicas is a hard ceiling the HPA controller will never cross, regardless of how far over target the utilization is. This lesson's HPA had maxReplicas: 5 and CPU utilization was still 201% (well above the 50% target) when it hit that ceiling — it simply stayed at 5 rather than scaling further.
Why didn't this lesson show the deployment scaling back down?
HPA's default scale-down behavior waits out a stabilization window (5 minutes by default) after the last scale-up before removing any replicas, specifically to avoid flapping under bursty load. That's a real, correct default — but it makes scale-down too slow to demonstrate inline here; Practice 4 below has you trigger and time it yourself.