Security, RBAC, and Observability

Kubernetes Security Model

Kubernetes is designed to run multi-tenant workloads, so security is layered:

Request → Authentication → Authorization (RBAC) → Admission Control → etcd

RBAC — Roles and Bindings

Role (namespace-scoped)

# role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
  - apiGroups: [""]           # "" = core API group
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]

ClusterRole (cluster-wide)

# clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-manager
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

RoleBinding — Grant a Role to a User/ServiceAccount

# rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-binding
  namespace: default
subjects:
  - kind: User
    name: alice
    apiGroup: rbac.authorization.k8s.io
  - kind: ServiceAccount
    name: ci-bot
    namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f role.yaml rolebinding.yaml

# Check what a user/SA can do
kubectl auth can-i list pods --as alice
kubectl auth can-i delete deployments --as alice
kubectl auth can-i delete pods --as system:serviceaccount:default:ci-bot

# List all role bindings in a namespace
kubectl get rolebindings -n default
kubectl describe rolebinding read-pods-binding

ServiceAccounts

Every pod runs as a ServiceAccount. By default it’s the default SA with minimal permissions.

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-bot
  namespace: default

# Use the ServiceAccount in a Pod
spec:
  serviceAccountName: ci-bot
  containers:
    - name: app
      image: my-app:1.0

kubectl create serviceaccount ci-bot
kubectl get serviceaccount

Pod Security — Non-Root Containers

spec:
  securityContext:
    runAsNonRoot: true          # pod-level: all containers run as non-root
    runAsUser: 1000
    fsGroup: 2000

  containers:
    - name: app
      image: my-app:1.0
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true    # container can't write to /
        capabilities:
          drop:
            - ALL                        # drop all Linux capabilities

Network Policies

Control which pods can talk to which other pods.

# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}          # applies to all pods in namespace
  policyTypes:
    - Ingress
  ingress: []              # no ingress allowed by default

# Allow only frontend → backend traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080

Observability

Logs

# Stream logs from a pod
kubectl logs -f my-pod

# Logs from a specific container in a multi-container pod
kubectl logs my-pod -c sidecar

# Logs from a previous container run (after crash)
kubectl logs my-pod --previous

# Aggregate logs across all pods matching a label
kubectl logs -l app=api --all-containers=true --tail=100

Events

# Cluster-wide events (sorted by time — most useful for debugging)
kubectl get events --sort-by='.lastTimestamp'

# Events in a specific namespace
kubectl get events -n production

# Events for a specific resource
kubectl describe pod my-pod | grep -A 20 Events
kubectl describe node worker-1 | grep -A 20 Events

Metrics

# Enable metrics server on minikube
minikube addons enable metrics-server

# Pod CPU and memory usage
kubectl top pods
kubectl top pods -n production --sort-by=cpu

# Node resource usage
kubectl top nodes

Health Checks — Debugging Failed Pods

# Pod is in CrashLoopBackOff
kubectl describe pod <name>     # look at: State, Last State, Events
kubectl logs <name> --previous  # logs from last crash

# Pod is Pending
kubectl describe pod <name>     # look for: Insufficient cpu/memory, No nodes available

# Pod is not Ready
kubectl describe pod <name>     # look at: Readiness probe failures in Events

# Generic debugging — run a temporary debug pod
kubectl run debug --rm -it --image=busybox -- sh

# Copy files out of a pod
kubectl cp my-pod:/var/log/app.log ./app.log

Monitoring Stack (Production)

The standard Kubernetes observability stack:

# Install kube-prometheus-stack via Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

# Access Grafana locally
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
# Open http://localhost:3000 (admin / prom-operator)

Learning Outcomes

You can:

• Write Role, ClusterRole, RoleBinding manifests for least-privilege access
• Verify permissions with kubectl auth can-i
• Secure pods with non-root contexts and capability drops
• Apply NetworkPolicy to restrict inter-pod traffic
• Diagnose issues using logs, events, and kubectl describe
• Understand the standard monitoring stack (Prometheus + Grafana)