Security, RBAC, and Observability

Kubernetes Security Model

Kubernetes is designed to run multi-tenant workloads, so security is layered:

Request → Authentication → Authorization (RBAC) → Admission Control → etcd

RBAC — Roles and Bindings

Role (namespace-scoped)

# role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
  - apiGroups: [""]           # "" = core API group
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]

ClusterRole (cluster-wide)

# clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-manager
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

RoleBinding — Grant a Role to a User/ServiceAccount

# rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-binding
  namespace: default
subjects:
  - kind: User
    name: alice
    apiGroup: rbac.authorization.k8s.io
  - kind: ServiceAccount
    name: ci-bot
    namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f role.yaml -f rolebinding.yaml

# Check what a user/SA can do
kubectl auth can-i list pods --as alice
kubectl auth can-i delete deployments --as alice
kubectl auth can-i delete pods --as system:serviceaccount:default:ci-bot

# List all role bindings in a namespace
kubectl get rolebindings -n default
kubectl describe rolebinding read-pods-binding

ServiceAccounts

Every pod runs as a ServiceAccount. By default it’s the default SA with minimal permissions.

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-bot
  namespace: default

# Use the ServiceAccount in a Pod
spec:
  serviceAccountName: ci-bot
  containers:
    - name: app
      image: my-app:1.0

kubectl create serviceaccount ci-bot
kubectl get serviceaccount

Pod Security — Non-Root Containers

spec:
  securityContext:
    runAsNonRoot: true          # pod-level: all containers run as non-root
    runAsUser: 1000
    fsGroup: 2000

  containers:
    - name: app
      image: my-app:1.0
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true    # container can't write to /
        capabilities:
          drop:
            - ALL                        # drop all Linux capabilities

Network Policies

Control which pods can talk to which other pods.

# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}          # applies to all pods in namespace
  policyTypes:
    - Ingress
  ingress: []              # no ingress allowed by default

# Allow only frontend → backend traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080

Observability

Logs

# Stream logs from a pod
kubectl logs -f my-pod

# Logs from a specific container in a multi-container pod
kubectl logs my-pod -c sidecar

# Logs from a previous container run (after crash)
kubectl logs my-pod --previous

# Aggregate logs across all pods matching a label
kubectl logs -l app=api --all-containers=true --tail=100

Events

# Cluster-wide events (sorted by time — most useful for debugging)
kubectl get events --sort-by='.lastTimestamp'

# Events in a specific namespace
kubectl get events -n production

# Events for a specific resource
kubectl describe pod my-pod | grep -A 20 Events
kubectl describe node worker-1 | grep -A 20 Events

Metrics

# Enable metrics server on minikube
minikube addons enable metrics-server

# Pod CPU and memory usage
kubectl top pods
kubectl top pods -n production --sort-by=cpu

# Node resource usage
kubectl top nodes

Health Checks — Debugging Failed Pods

# Pod is in CrashLoopBackOff
kubectl describe pod <name>     # look at: State, Last State, Events
kubectl logs <name> --previous  # logs from last crash

# Pod is Pending
kubectl describe pod <name>     # look for: Insufficient cpu/memory, No nodes available

# Pod is not Ready
kubectl describe pod <name>     # look at: Readiness probe failures in Events

# Generic debugging — run a temporary debug pod
kubectl run debug --rm -it --image=busybox -- sh

# Copy files out of a pod
kubectl cp my-pod:/var/log/app.log ./app.log

Monitoring Stack (Production)

The standard Kubernetes observability stack:

# Install kube-prometheus-stack via Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

# Access Grafana locally
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
# Open http://localhost:3000 (admin / prom-operator)

Learning Outcomes

You can:

• Write Role, ClusterRole, RoleBinding manifests for least-privilege access
• Verify permissions with kubectl auth can-i
• Secure pods with non-root contexts and capability drops
• Apply NetworkPolicy to restrict inter-pod traffic
• Diagnose issues using logs, events, and kubectl describe
• Understand the standard monitoring stack (Prometheus + Grafana)

Practice

The outputs come from a single-node minikube profile named practice-k8s. With the default profile, your node and its system Pods are named minikube instead.

1. Apply role.yaml and rolebinding.yaml, then use kubectl auth can-i to answer: can alice list pods, read pod logs, delete pods, list pods in kube-system, or list secrets?

kubectl apply -f role.yaml -f rolebinding.yaml
kubectl auth can-i list pods --as alice
kubectl auth can-i get pods --subresource=log --as alice
kubectl auth can-i delete pods --as alice
kubectl auth can-i list pods --as alice -n kube-system
kubectl auth can-i list secrets --as alice

$ kubectl apply -f role.yaml -f rolebinding.yaml
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/read-pods-binding created
$ kubectl auth can-i list pods --as alice
yes
$ kubectl auth can-i get pods --subresource=log --as alice
yes
$ kubectl auth can-i delete pods --as alice
no
$ kubectl auth can-i list pods --as alice -n kube-system
no
$ kubectl auth can-i list secrets --as alice
no

2. Create the ci-bot ServiceAccount and give it the deployment-manager ClusterRole in the default namespace only. Check whether it can delete deployments in default and in kube-system, and whether it can delete pods.

# ci-bot-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-bot-deployments
  namespace: default
subjects:
  - kind: ServiceAccount
    name: ci-bot
    namespace: default
roleRef:
  kind: ClusterRole            # a ClusterRole, granted only inside "default"
  name: deployment-manager
  apiGroup: rbac.authorization.k8s.io

$ kubectl create serviceaccount ci-bot
serviceaccount/ci-bot created
$ kubectl apply -f clusterrole.yaml -f ci-bot-binding.yaml
clusterrole.rbac.authorization.k8s.io/deployment-manager created
rolebinding.rbac.authorization.k8s.io/ci-bot-deployments created
$ kubectl auth can-i delete deployments --as system:serviceaccount:default:ci-bot
yes
$ kubectl auth can-i delete deployments --as system:serviceaccount:default:ci-bot -n kube-system
no
$ kubectl auth can-i delete pods --as system:serviceaccount:default:ci-bot
no
$ kubectl auth can-i list pods --as system:serviceaccount:default:ci-bot
yes

3. Run the “Pod Security” settings on a real Pod. Use busybox:1.36 running sleep 3600, then check which user the container runs as and whether it can write to /.

# secure-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: locked-down
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
  containers:
    - name: app
      image: busybox:1.36
      command: ["sleep", "3600"]
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

$ kubectl apply -f secure-pod.yaml
pod/locked-down created
$ kubectl wait --for=condition=Ready pod/locked-down
pod/locked-down condition met
$ kubectl exec locked-down -- id
uid=1000 gid=0(root) groups=0(root),2000
$ kubectl exec locked-down -- touch /hello
touch: /hello: Read-only file system
command terminated with exit code 1

4. Write a Pod that crashes on its first start and runs normally after that, using an emptyDir as a marker. Then use kubectl logs and kubectl logs --previous to see both starts.

# crasher.yaml
apiVersion: v1
kind: Pod
metadata:
  name: crasher
spec:
  volumes:
    - name: state
      emptyDir: {}            # survives container restarts, not Pod deletion
  containers:
    - name: app
      image: busybox:1.36
      volumeMounts:
        - name: state
          mountPath: /state
      command:
        - sh
        - -c
        - |
          if [ -f /state/crashed ]; then
            echo "second start: running"; sleep 3600
          fi
          touch /state/crashed
          echo "first start"; echo "fatal: DATABASE_URL is not set" >&2; exit 1

$ kubectl apply -f crasher.yaml
pod/crasher created
$ kubectl wait --for=condition=Ready pod/crasher --timeout=120s
pod/crasher condition met
$ kubectl get pod crasher
NAME      READY   STATUS    RESTARTS     AGE
crasher   1/1     Running   1 (1s ago)   2s
$ kubectl logs crasher
second start: running
$ kubectl logs crasher --previous
first start
fatal: DATABASE_URL is not set
$ kubectl describe pod crasher | grep -A 3 'Last State'
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 11 Oct 2026 10:53:08 +0530

5. Enable metrics-server and use kubectl top to find the hungriest Pod in kube-system and the node’s overall usage.

minikube -p practice-k8s addons enable metrics-server   # drop -p with the default profile
kubectl top pods -n kube-system --sort-by=memory
kubectl top nodes

$ minikube -p practice-k8s addons enable metrics-server
* metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
  - Using image registry.k8s.io/metrics-server/metrics-server:v0.8.1
* The 'metrics-server' addon is enabled
$ kubectl top pods -n kube-system --sort-by=memory
NAME                                   CPU(cores)   MEMORY(bytes)   
kube-apiserver-practice-k8s            64m          233Mi           
kube-controller-manager-practice-k8s   26m          70Mi            
etcd-practice-k8s                      38m          51Mi            
kube-scheduler-practice-k8s            10m          28Mi            
storage-provisioner                    3m           20Mi            
kube-proxy-9dr62                       1m           20Mi            
metrics-server-9d74bb658-9rtvr         4m           20Mi            
coredns-7d764666f9-lwqfk               3m           17Mi            
$ kubectl top nodes
NAME           CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
practice-k8s   272m         2%       1195Mi          15%

Clean up with:

kubectl delete -f crasher.yaml -f secure-pod.yaml
kubectl delete -f ci-bot-binding.yaml -f clusterrole.yaml -f rolebinding.yaml -f role.yaml
kubectl delete serviceaccount ci-bot

$ kubectl delete -f crasher.yaml -f secure-pod.yaml
pod "crasher" deleted from default namespace
pod "locked-down" deleted from default namespace
$ kubectl delete -f ci-bot-binding.yaml -f clusterrole.yaml -f rolebinding.yaml -f role.yaml
rolebinding.rbac.authorization.k8s.io "ci-bot-deployments" deleted from default namespace
clusterrole.rbac.authorization.k8s.io "deployment-manager" deleted
rolebinding.rbac.authorization.k8s.io "read-pods-binding" deleted from default namespace
role.rbac.authorization.k8s.io "pod-reader" deleted from default namespace
$ kubectl delete serviceaccount ci-bot
serviceaccount "ci-bot" deleted from default namespace