What does "reconciliation" mean in Kubernetes?

A controller repeatedly compares the desired state in an object's spec with what actually exists in the cluster, then creates, updates or deletes resources to close the gap. If you delete a Pod that a ReplicaSet owns, the ReplicaSet sees one Pod too few and creates a replacement.

When should I use a StatefulSet instead of a Deployment?

Use a StatefulSet when each Pod needs a stable name, a stable DNS name through a headless Service, and its own PersistentVolumeClaim that stays bound to it across restarts. Databases such as PostgreSQL, and systems like Kafka or Elasticsearch, are the usual cases. Stateless web and API servers belong in a Deployment.

Are a StatefulSet's PersistentVolumeClaims deleted with it?

No. Deleting a StatefulSet leaves the PVCs created from its volumeClaimTemplates in place, so deleting the workload cannot destroy its data. Delete them yourself with kubectl delete pvc once you are sure the data is no longer needed.

Why can't I scale a DaemonSet with kubectl scale?

A DaemonSet has no replicas field and no scale subresource, so kubectl scale fails with a NotFound error. It runs one Pod on every node that matches it, so its Pod count changes only when nodes are added, removed or relabelled.

How do I run a Kubernetes CronJob immediately to test it?

Run kubectl create job --from=cronjob/ . That copies the CronJob's jobTemplate into an ordinary Job that starts at once, without changing the schedule. The CronJob's LAST SCHEDULE column stays unchanged, because the CronJob itself did not fire.

Workload Controllers | SkillByExample

The reconciliation loop

Every Kubernetes controller runs the same loop, forever:

Observe: read the current state from the API server.
Diff: compare it with the desired state in the object’s spec.
Act: create, update or delete objects to close the gap.

You never tell Kubernetes “start 3 Pods”. You declare replicas: 3, and a controller works out what to do. Each section below applies one controller’s manifest and shows what it did.

The outputs come from a single-node minikube cluster started with minikube start -p rw-k8s --driver=docker. The profile name becomes the node name, so NODE shows rw-k8s rather than minikube. Pod hashes, IPs and ages will differ on yours.

Deployment (stateless apps)

The controller you’ll use most. It manages rolling updates and rollbacks for interchangeable Pods.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web
          image: nginx:1.25
          ports:
            - containerPort: 80

$ kubectl apply -f deployment.yaml
deployment.apps/web-app created
$ kubectl rollout status deployment/web-app
Waiting for deployment "web-app" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "web-app" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "web-app" rollout to finish: 2 of 3 updated replicas are available...
deployment "web-app" successfully rolled out
$ kubectl get deployment web-app
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
web-app   3/3     3            3           2s
$ kubectl get replicasets -l app=web-app
NAME                 DESIRED   CURRENT   READY   AGE
web-app-58477f86f4   3         3         3       2s
$ kubectl get pods -l app=web-app
NAME                       READY   STATUS    RESTARTS   AGE
web-app-58477f86f4-jpbpm   1/1     Running   0          2s
web-app-58477f86f4-k67lh   1/1     Running   0          2s
web-app-58477f86f4-nnhhd   1/1     Running   0          2s

You applied one object and got four more: a ReplicaSet and three Pods. The names chain together. 58477f86f4 is a hash of the Pod template, shared by the ReplicaSet and its Pods, and each Pod adds a random suffix. Practice task 1 deletes one of these Pods to see which controller brings it back.

ReplicaSet (managed by the Deployment)

You rarely create ReplicaSets yourself. The Deployment creates a new one every time the Pod template changes, which is how a rolling update works:

$ kubectl set image deployment/web-app web=nginx:1.27
deployment.apps/web-app image updated
$ kubectl rollout status deployment/web-app
Waiting for deployment "web-app" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web-app" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "web-app" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "web-app" rollout to finish: 1 old replicas are pending termination...
deployment "web-app" successfully rolled out
$ kubectl get replicasets -l app=web-app
NAME                 DESIRED   CURRENT   READY   AGE
web-app-58477f86f4   0         0         0       12s
web-app-5ff6fd6876   3         3         3       4s

A new template hash means a new ReplicaSet, 5ff6fd6876. The Deployment scaled it up one Pod at a time (maxSurge: 1) and scaled the old one down only as new Pods became available (maxUnavailable: 0). The old ReplicaSet is kept at 0 replicas, not deleted. That is what makes a rollback fast:

$ kubectl rollout history deployment/web-app
deployment.apps/web-app 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

Each revision is one of those ReplicaSets. kubectl rollout undo scales revision 1 back up instead of building anything new. CHANGE-CAUSE is <none> because nothing set the kubernetes.io/change-cause annotation.

StatefulSet (stateful apps)

For workloads that need a stable identity and ordered startup and shutdown, such as databases and message queues. This one reads its password from a Secret and claims a volume per Pod.

# statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless   # must reference a headless service
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:15
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pg-secret
                  key: POSTGRES_PASSWORD
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:            # each pod gets its OWN PVC
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

serviceName must name a headless Service. That Service is what gives each Pod a DNS name:

# postgres-headless.yaml
apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432

kubectl create secret generic pg-secret --from-literal=POSTGRES_PASSWORD=lesson
kubectl apply -f postgres-headless.yaml -f statefulset.yaml
kubectl rollout status statefulset/postgres --timeout=300s
kubectl get pods -l app=postgres
kubectl get pvc

$ kubectl create secret generic pg-secret --from-literal=POSTGRES_PASSWORD=lesson
secret/pg-secret created
$ kubectl apply -f postgres-headless.yaml -f statefulset.yaml
service/postgres-headless created
statefulset.apps/postgres created
$ kubectl rollout status statefulset/postgres --timeout=300s
Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 3 new pods have been updated...
$ kubectl get pods -l app=postgres
NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          3s
postgres-1   1/1     Running   0          2s
postgres-2   1/1     Running   0          1s
$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-postgres-0   Bound    pvc-4edc0189-c964-4b66-8cbf-704387397e3b   5Gi        RWO            standard       <unset>                 3s
data-postgres-1   Bound    pvc-33a95c1f-0cd4-4dd0-88a4-411c40ed937d   5Gi        RWO            standard       <unset>                 2s
data-postgres-2   Bound    pvc-661ff88e-8f6e-47f3-a943-e91d29372ba9   5Gi        RWO            standard       <unset>                 1s

The Pods are named by ordinal, postgres-0 to postgres-2, and their ages step down by a second each: the controller creates a Pod only after the previous one is Ready. They became Ready quickly because this manifest has no readiness probe, so “Ready” only means the container started. Each Pod also got its own claim, named <template>-<pod>, from volumeClaimTemplates.

Each Pod’s DNS name comes from the headless Service:

$ kubectl run dns-test --image=busybox:1.36 --rm -i --restart=Never -- nslookup postgres-1.postgres-headless.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53


Name:	postgres-1.postgres-headless.default.svc.cluster.local
Address: 10.244.0.20

pod "dns-test" deleted from default namespace
$ kubectl get pod postgres-1 -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
postgres-1   1/1     Running   0          11s   10.244.0.20   rw-k8s   <none>           <none>

postgres-1.postgres-headless resolves to that one Pod’s IP, so a replica can always find the primary at postgres-0.postgres-headless. Practice task 2 shows that the name and the claim both survive the Pod being deleted.

Key differences from a Deployment:

	Deployment	StatefulSet
Pod names	Template hash + random suffix	Stable ordinal (app-0, app-1)
Startup order	Parallel	Ordered (0 → 1 → 2)
Storage	Shared or none	Each pod gets its own PVC
DNS	Service only	Individual pod DNS names

DaemonSet (one Pod per node)

A DaemonSet runs one copy of a Pod on every node. It suits log collectors, monitoring agents and network plugins.

# daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd:v1.16
          volumeMounts:
            - name: varlog
              mountPath: /var/log
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

$ kubectl apply -f daemonset.yaml
daemonset.apps/log-collector created
$ kubectl rollout status daemonset/log-collector --timeout=300s
Waiting for daemon set "log-collector" rollout to finish: 0 of 1 updated pods are available...
daemon set "log-collector" successfully rolled out
$ kubectl get daemonset log-collector
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
log-collector   1         1         1       1            1           <none>          1s
$ kubectl get pods -l app=log-collector -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
log-collector-7mxz9   1/1     Running   0          2s    10.244.0.23   rw-k8s   <none>           <none>

There is no replicas in the manifest. DESIRED 1 is the number of nodes, and the single Pod landed on the only node, rw-k8s. The hostPath volume mounts that node’s own /var/log, which is why an agent like this has to run on every node. When a node joins the cluster, the DaemonSet controller starts a Pod on it, and when a node is removed, its Pod goes with it.

Job (run to completion)

A Job runs Pods until a set number of them succeed. It suits database migrations, batch processing and one-off scripts. This one fakes three migration steps with busybox:

# job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  completions: 1
  backoffLimit: 3        # retry up to 3 times on failure
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              for v in 001 002 003; do
                echo "applying migration $v"
                sleep 1
              done
              echo "schema up to date"

$ kubectl apply -f job.yaml
job.batch/db-migrate created
$ kubectl get jobs
NAME         STATUS    COMPLETIONS   DURATION   AGE
db-migrate   Running   0/1           0s         0s
$ kubectl wait --for=condition=complete job/db-migrate --timeout=120s
job.batch/db-migrate condition met
$ kubectl get jobs
NAME         STATUS     COMPLETIONS   DURATION   AGE
db-migrate   Complete   1/1           6s         6s
$ kubectl get pods -l job-name=db-migrate
NAME               READY   STATUS      RESTARTS   AGE
db-migrate-47m64   0/1     Completed   0          6s
$ kubectl logs job/db-migrate
applying migration 001
applying migration 002
applying migration 003
schema up to date

kubectl wait blocks until the Job’s complete condition is true, which is the right way to wait in a script. The Pod stays behind in Completed, not deleted, so its logs are still readable. kubectl logs job/db-migrate picks one of the Job’s Pods for you. Unlike a Deployment, nothing restarts a Pod that exits 0. Practice task 4 shows what backoffLimit does when it exits non-zero.

CronJob (scheduled Jobs)

A CronJob creates a Job from its jobTemplate on a cron schedule.

# cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"     # every day at 2 AM (cron syntax)
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: busybox:1.36
              command: ["sh", "-c", "echo backup started $(date -u +%FT%TZ); echo wrote backup-$(date -u +%F).tar"]

kubectl apply -f cronjob.yaml
kubectl get cronjobs
kubectl create job backup-now --from=cronjob/nightly-backup
kubectl wait --for=condition=complete job/backup-now --timeout=120s
kubectl get jobs
kubectl logs job/backup-now

$ kubectl apply -f cronjob.yaml
cronjob.batch/nightly-backup created
$ kubectl get cronjobs
NAME             SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
nightly-backup   0 2 * * *   <none>     False     0        <none>          0s
$ kubectl create job backup-now --from=cronjob/nightly-backup
job.batch/backup-now created
$ kubectl wait --for=condition=complete job/backup-now --timeout=120s
job.batch/backup-now condition met
$ kubectl get jobs
NAME         STATUS     COMPLETIONS   DURATION   AGE
backup-now   Complete   1/1           4s         4s
db-migrate   Complete   1/1           6s         16s
$ kubectl logs job/backup-now
backup started 2026-10-11T17:20:20Z
wrote backup-2026-10-11.tar

--from=cronjob/... copies the jobTemplate into an ordinary Job that runs now, which is how you test a schedule without waiting for 2 AM. LAST SCHEDULE is <none> because the CronJob itself has never fired. TIMEZONE <none> means the schedule follows the time zone of the cluster’s controller manager, usually UTC. Set spec.timeZone to pin it.

Clean up before the Practice tasks, which start from an empty namespace. The PVCs need their own delete:

$ kubectl delete -f deployment.yaml -f postgres-headless.yaml -f statefulset.yaml -f daemonset.yaml -f job.yaml -f cronjob.yaml
deployment.apps "web-app" deleted from default namespace
service "postgres-headless" deleted from default namespace
statefulset.apps "postgres" deleted from default namespace
daemonset.apps "log-collector" deleted from default namespace
job.batch "db-migrate" deleted from default namespace
cronjob.batch "nightly-backup" deleted from default namespace
$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-postgres-0   Bound    pvc-4edc0189-c964-4b66-8cbf-704387397e3b   5Gi        RWO            standard       <unset>                 6m55s
data-postgres-1   Bound    pvc-33a95c1f-0cd4-4dd0-88a4-411c40ed937d   5Gi        RWO            standard       <unset>                 6m54s
data-postgres-2   Bound    pvc-661ff88e-8f6e-47f3-a943-e91d29372ba9   5Gi        RWO            standard       <unset>                 6m53s
$ kubectl delete pvc -l app=postgres
persistentvolumeclaim "data-postgres-0" deleted from default namespace
persistentvolumeclaim "data-postgres-1" deleted from default namespace
persistentvolumeclaim "data-postgres-2" deleted from default namespace
$ kubectl delete secret pg-secret
secret "pg-secret" deleted from default namespace

The claims outlived their StatefulSet, and they carry its app=postgres label, so one selector deletes them all. Left in place, they would be re-attached by the next StatefulSet named postgres, still at 5Gi.

Controller comparison

Controller	Use case	Pods have stable identity?	Persistent storage per pod?
Deployment	Stateless web/API servers	No	No
StatefulSet	Databases, queues	Yes	Yes
DaemonSet	Node-level agents (logging, monitoring)	No	No
Job	One-off batch tasks, migrations	No	No
CronJob	Scheduled recurring tasks	No	No

Learning outcomes

You can:

Explain the reconciliation loop and why controllers are central to Kubernetes
Choose the right controller for stateless apps, databases, node agents, and batch jobs
Write Deployment, StatefulSet, DaemonSet, Job, and CronJob manifests
Explain why StatefulSets suit databases: ordinal names, per-Pod DNS and per-Pod claims
Follow reconciliation through kubectl get replicasets, rollout status and controller events

Practice

The outputs come from the same single-node rw-k8s profile, starting from the empty namespace the cleanup above leaves.

Apply the web-app Deployment, delete one of its Pods, then use the ReplicaSet’s events to show which controller replaced it.

Solution

kubectl apply -f deployment.yaml
kubectl rollout status deployment/web-app
kubectl get pods -l app=web-app
kubectl delete pod web-app-58477f86f4-6xkkd      # use a name from your own list
kubectl get pods -l app=web-app
kubectl describe replicaset -l app=web-app | grep SuccessfulCreate

$ kubectl apply -f deployment.yaml
deployment.apps/web-app created
$ kubectl rollout status deployment/web-app
Waiting for deployment "web-app" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "web-app" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "web-app" rollout to finish: 2 of 3 updated replicas are available...
deployment "web-app" successfully rolled out
$ kubectl get pods -l app=web-app
NAME                       READY   STATUS    RESTARTS   AGE
web-app-58477f86f4-6xkkd   1/1     Running   0          3s
web-app-58477f86f4-7nmtz   1/1     Running   0          3s
web-app-58477f86f4-w49xv   1/1     Running   0          3s
$ kubectl delete pod web-app-58477f86f4-6xkkd
pod "web-app-58477f86f4-6xkkd" deleted from default namespace
$ kubectl get pods -l app=web-app
NAME                       READY   STATUS    RESTARTS   AGE
web-app-58477f86f4-7g8bf   1/1     Running   0          2s
web-app-58477f86f4-7nmtz   1/1     Running   0          14s
web-app-58477f86f4-w49xv   1/1     Running   0          14s
$ kubectl describe replicaset -l app=web-app | grep SuccessfulCreate
  Normal  SuccessfulCreate  14s   replicaset-controller  Created pod: web-app-58477f86f4-6xkkd
  Normal  SuccessfulCreate  14s   replicaset-controller  Created pod: web-app-58477f86f4-w49xv
  Normal  SuccessfulCreate  14s   replicaset-controller  Created pod: web-app-58477f86f4-7nmtz
  Normal  SuccessfulCreate  2s    replicaset-controller  Created pod: web-app-58477f86f4-7g8bf

The replacement has a new random suffix and a 2-second age. It was created by the replicaset-controller, not the Deployment: the ReplicaSet saw 2 Pods against a desired 3 and acted. The Deployment only manages ReplicaSets. The template hash, 58477f86f4, is the same as in the lesson, because the Pod template is identical.

Run the StatefulSet with replicas: 2 and storage: 100Mi, creating pg-secret first. Create a database in postgres-0, delete that Pod, and check what comes back: the name, the PVC, and the database.

Solution

In statefulset.yaml, set replicas: 2 and storage: 100Mi. Then add a --- line and the Service from postgres-headless.yaml after it, so one file holds both.

Postgres can take a few seconds to accept connections after its Pod is Ready, and the replacement postgres-0 may not exist yet right after the delete. If a psql call fails with could not connect to server, or kubectl wait reports NotFound, wait and retry, e.g. until kubectl exec postgres-0 -- pg_isready -U postgres; do sleep 1; done. Then:

kubectl create secret generic pg-secret --from-literal=POSTGRES_PASSWORD=practice
kubectl apply -f statefulset.yaml
kubectl rollout status statefulset/postgres --timeout=300s
kubectl get pods -l app=postgres
kubectl get pvc
kubectl exec postgres-0 -- psql -U postgres -c 'CREATE DATABASE survives;'
kubectl delete pod postgres-0
kubectl wait --for=condition=Ready pod/postgres-0 --timeout=120s
kubectl get pods -l app=postgres
kubectl exec postgres-0 -- psql -U postgres -Atc 'SELECT datname FROM pg_database WHERE datname = $$survives$$;'

$ kubectl create secret generic pg-secret --from-literal=POSTGRES_PASSWORD=practice
secret/pg-secret created
$ kubectl apply -f statefulset.yaml
statefulset.apps/postgres created
service/postgres-headless created
$ kubectl rollout status statefulset/postgres --timeout=300s
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 2 new pods have been updated...
$ kubectl get pods -l app=postgres
NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          3s
postgres-1   1/1     Running   0          1s
$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-postgres-0   Bound    pvc-74de9633-c7f3-431d-8bec-960f90c5d44b   100Mi      RWO            standard       <unset>                 3s
data-postgres-1   Bound    pvc-f1474fc5-61de-4676-ae85-5eec2373172d   100Mi      RWO            standard       <unset>                 1s
$ kubectl exec postgres-0 -- psql -U postgres -c 'CREATE DATABASE survives;'
CREATE DATABASE
$ kubectl delete pod postgres-0
pod "postgres-0" deleted from default namespace
$ kubectl wait --for=condition=Ready pod/postgres-0 --timeout=120s
pod/postgres-0 condition met
$ kubectl get pods -l app=postgres
NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          1s
postgres-1   1/1     Running   0          17s
$ kubectl exec postgres-0 -- psql -U postgres -Atc 'SELECT datname FROM pg_database WHERE datname = $$survives$$;'
survives

apply reports the objects in file order, so the Service after --- comes second. The replacement is called postgres-0 again, not a random name. It re-attached data-postgres-0, so the survives database is still there. The claims are 100Mi, not the lesson’s 5Gi, only because the lesson’s cleanup deleted the old ones. A Deployment Pod would have come back with a new name and no claim of its own.

Apply the DaemonSet and count its Pods on this one-node cluster. Then predict what kubectl scale daemonset log-collector --replicas=3 does, and run it.

Solution

kubectl apply -f daemonset.yaml
kubectl rollout status daemonset/log-collector --timeout=300s
kubectl get daemonset
kubectl get pods -l app=log-collector -o wide
kubectl scale daemonset log-collector --replicas=3

$ kubectl apply -f daemonset.yaml
daemonset.apps/log-collector created
$ kubectl rollout status daemonset/log-collector --timeout=300s
Waiting for daemon set "log-collector" rollout to finish: 0 of 1 updated pods are available...
daemon set "log-collector" successfully rolled out
$ kubectl get daemonset
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
log-collector   1         1         1       1            1           <none>          1s
$ kubectl get pods -l app=log-collector -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
log-collector-k9m2k   1/1     Running   0          1s    10.244.0.33   rw-k8s   <none>           <none>
$ kubectl scale daemonset log-collector --replicas=3
Error from server (NotFound): the server could not find the requested resource

DESIRED 1 comes from the node count, not from a field you set. A DaemonSet has no replicas and no scale endpoint, which is what the NotFound refers to. The only way to get more Pods is to add nodes.

Turn the Job into one that always fails: use busybox:1.36, print a message and exit 1, set backoffLimit: 2 and restartPolicy: Never. How many Pods run, and how does the Job end?

Solution

# job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  completions: 1
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: busybox:1.36
          command: ["sh", "-c", "echo migrating; echo 'table users already exists' >&2; exit 1"]

$ kubectl apply -f job.yaml
job.batch/db-migrate created
$ kubectl wait --for=condition=Failed job/db-migrate --timeout=300s
job.batch/db-migrate condition met
$ kubectl get jobs
NAME         STATUS   COMPLETIONS   DURATION   AGE
db-migrate   Failed   0/1           42s        42s
$ kubectl get pods -l job-name=db-migrate
NAME               READY   STATUS   RESTARTS   AGE
db-migrate-5hp4s   0/1     Error    0          30s
db-migrate-vh8w9   0/1     Error    0          42s
db-migrate-wsvnt   0/1     Error    0          10s
$ kubectl logs -l job-name=db-migrate --prefix
[pod/db-migrate-5hp4s/migrate] table users already exists
[pod/db-migrate-5hp4s/migrate] migrating
[pod/db-migrate-vh8w9/migrate] table users already exists
[pod/db-migrate-vh8w9/migrate] migrating
[pod/db-migrate-wsvnt/migrate] table users already exists
[pod/db-migrate-wsvnt/migrate] migrating
$ kubectl describe job db-migrate | grep -E 'Pods Statuses|BackoffLimitExceeded'
Pods Statuses:    0 Active (0 Ready) / 0 Succeeded / 3 Failed
  Warning  BackoffLimitExceeded  6s    job-controller  Job has reached the specified backoff limit

Three Pods: the first attempt plus two retries. With restartPolicy: Never, each retry is a new Pod, so every attempt’s logs survive for inspection. The gaps between Pod ages (42s, 30s, 10s) grow, 12 seconds then 20, because of the retry back-off. The order of stdout and stderr lines from one Pod is not guaranteed: here the stderr line came first in all three, although the script echoes migrating first.

Make nightly-backup fire on its own: change its schedule to every minute, wait for the first Job the CronJob creates, and compare that Job’s name and the LAST SCHEDULE column with what kubectl create job --from gave you in the lesson.

Solution

In cronjob.yaml, change the schedule line to:

  schedule: "*/1 * * * *"   # every minute, for testing

kubectl apply -f cronjob.yaml
kubectl get cronjob nightly-backup
until kubectl get jobs -o name | grep nightly-backup-; do sleep 5; done
kubectl wait --for=condition=complete job/nightly-backup-29862323 --timeout=120s   # your Job's name
kubectl get cronjob nightly-backup
kubectl logs job/nightly-backup-29862323
date -u -d @$((29862323 * 60)) +%FT%TZ
kubectl get jobs

$ kubectl apply -f cronjob.yaml
cronjob.batch/nightly-backup created
$ kubectl get cronjob nightly-backup
NAME             SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
nightly-backup   */1 * * * *   <none>     False     0        <none>          0s
$ until kubectl get jobs -o name | grep nightly-backup-; do sleep 5; done
job.batch/nightly-backup-29862323
$ kubectl wait --for=condition=complete job/nightly-backup-29862323 --timeout=120s
job.batch/nightly-backup-29862323 condition met
$ kubectl get cronjob nightly-backup
NAME             SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
nightly-backup   */1 * * * *   <none>     False     0        6s              36s
$ kubectl logs job/nightly-backup-29862323
backup started 2026-10-11T17:23:00Z
wrote backup-2026-10-11.tar
$ date -u -d @$((29862323 * 60)) +%FT%TZ
2026-10-11T17:23:00Z
$ kubectl get jobs
NAME                      STATUS     COMPLETIONS   DURATION   AGE
db-migrate                Failed     0/1           96s        96s
nightly-backup-29862323   Complete   1/1           3s         11s

This time LAST SCHEDULE is set, because the CronJob fired rather than you. A scheduled Job is named <cronjob>-<scheduled time in minutes since the Unix epoch>: 29862323 minutes is exactly 17:23:00 UTC, the same second the log line printed. The name is fixed per scheduled minute, so the controller cannot create the same run twice. A Job from --from gets whatever name you choose. Once you are done, set the schedule back, or the CronJob keeps creating a Job every minute.

Clean up with:

kubectl delete deployment web-app
kubectl delete -f statefulset.yaml -f daemonset.yaml -f job.yaml -f cronjob.yaml
kubectl get jobs
kubectl get pvc
kubectl delete pvc data-postgres-0 data-postgres-1
kubectl delete secret pg-secret

$ kubectl delete deployment web-app
deployment.apps "web-app" deleted from default namespace
$ kubectl delete -f statefulset.yaml -f daemonset.yaml -f job.yaml -f cronjob.yaml
statefulset.apps "postgres" deleted from default namespace
service "postgres-headless" deleted from default namespace
daemonset.apps "log-collector" deleted from default namespace
job.batch "db-migrate" deleted from default namespace
cronjob.batch "nightly-backup" deleted from default namespace
$ kubectl get jobs
No resources found in default namespace.
$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-postgres-0   Bound    pvc-74de9633-c7f3-431d-8bec-960f90c5d44b   100Mi      RWO            standard       <unset>                 2m14s
data-postgres-1   Bound    pvc-f1474fc5-61de-4676-ae85-5eec2373172d   100Mi      RWO            standard       <unset>                 2m12s
$ kubectl delete pvc data-postgres-0 data-postgres-1
persistentvolumeclaim "data-postgres-0" deleted from default namespace
persistentvolumeclaim "data-postgres-1" deleted from default namespace
$ kubectl delete secret pg-secret
secret "pg-secret" deleted from default namespace

nightly-backup-29862323 went with the CronJob that created it, because the CronJob is its owner. The PVCs did not go with the StatefulSet. Kubernetes keeps them by default so deleting the workload can’t destroy the data, which is why they need their own delete.

The reconciliation loop

Deployment (stateless apps)

ReplicaSet (managed by the Deployment)

StatefulSet (stateful apps)

DaemonSet (one Pod per node)

Job (run to completion)

CronJob (scheduled Jobs)

Controller comparison

Learning outcomes

Practice

Frequently Asked Questions