Kubernetes · lesson 5 of 10
Intermediate
6Advanced
1Also part of Kubernetes
5- + Deploying FastAPI with Docker and Kubernetes: Images, Signals, and Probes from FastAPI
- + Deployment Strategies Compared: Rolling, Blue-Green, and Canary from DevOps
- + GitOps in Practice: Git as the Source of Truth from DevOps
- + Kubernetes Pod Security Admission: Enforcing the Restricted Profile from Cloud Security
- + CIS Kubernetes Benchmark Checks with kube-bench in kind from Cloud Security
Kubernetes Services — Hands-On
Run a Service on a real cluster: ClusterIP, DNS names across namespaces, NodePort, Ingress routing, port-forwarding, and debugging empty endpoints.
By SkillByExample · Updated
The examples run on a local minikube cluster (Kubernetes v1.35, docker driver). Pod name hashes, IPs and ages will differ on your cluster.
Connecting Pods with a Service
A Deployment runs two nginx Pods labelled app=web; a ClusterIP Service selects them by that label.
# full-example.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web # ← Service selects on this label
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
selector:
app: web # ← match Pods with label app=web
ports:
- port: 80
targetPort: 80
type: ClusterIP
kubectl apply -f full-example.yaml
kubectl rollout status deployment/web
kubectl get pods,svc
$ kubectl apply -f full-example.yaml
deployment.apps/web created
service/web-service created
$ kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "web" rollout to finish: 1 of 2 updated replicas are available...
deployment "web" successfully rolled out
$ kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
pod/web-c77955c54-5f4bl 1/1 Running 0 18s
pod/web-c77955c54-6n8l8 1/1 Running 0 18s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28s
service/web-service ClusterIP 10.99.235.177 <none> 80/TCP 18s
The Service got a virtual IP (10.99.235.177) that no Pod owns — it stays fixed while the
Pods behind it come and go. EXTERNAL-IP is <none> because ClusterIP is internal only, so
the test has to run from inside the cluster:
kubectl run curl-test --rm -i --restart=Never --image=curlimages/curl:8.10.1 -- curl -sI http://web-service
$ kubectl run curl-test --rm -i --restart=Never --image=curlimages/curl:8.10.1 -- curl -sI http://web-service
HTTP/1.1 200 OK
Server: nginx/1.25.5
Date: Sat, 10 Oct 2026 20:16:53 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 16 Apr 2024 14:29:59 GMT
Connection: keep-alive
ETag: "661e8b67-267"
Accept-Ranges: bytes
pod "curl-test" deleted from default namespace
--rm -i --restart=Never makes a one-shot Pod: it runs curl, streams the output back and is
deleted. The bare name web-service resolved because the curl Pod is in the same namespace.
Service Discovery via DNS
For several requests in a row, a long-lived client Pod plus kubectl exec is more reliable than
kubectl run --rm, whose attach can miss the output of a command that finishes instantly. A
second Service, payments, lives in a finance namespace.
kubectl create namespace finance
kubectl create deployment payments -n finance --image=nginx:1.25
kubectl expose deployment payments -n finance --port=80
kubectl run client --image=curlimages/curl:8.10.1 -- sleep 3600
kubectl wait --for=condition=Ready pod/client
kubectl exec client -- sh -c 'for h in web-service web-service.default web-service.default.svc.cluster.local payments.finance.svc.cluster.local payments; do curl -s -o /dev/null -w "%{http_code} $h\n" http://$h; done'
$ kubectl create namespace finance
namespace/finance created
$ kubectl create deployment payments -n finance --image=nginx:1.25
deployment.apps/payments created
$ kubectl expose deployment payments -n finance --port=80
service/payments exposed
$ kubectl run client --image=curlimages/curl:8.10.1 -- sleep 3600
pod/client created
$ kubectl wait --for=condition=Ready pod/client
pod/client condition met
$ kubectl exec client -- sh -c 'for h in web-service web-service.default web-service.default.svc.cluster.local payments.finance.svc.cluster.local payments; do curl -s -o /dev/null -w "%{http_code} $h\n" http://$h; done'
200 web-service
200 web-service.default
200 web-service.default.svc.cluster.local
200 payments.finance.svc.cluster.local
000 payments
command terminated with exit code 6
The full form is <service>.<namespace>.svc.cluster.local, and the two shorter forms work from
the default namespace. Bare payments fails with curl exit code 6 (could not resolve host):
the short name only searches the caller’s own namespace, so a cross-namespace call needs at least
payments.finance.
The search list that makes short names work is in every Pod’s /etc/resolv.conf. A busybox Pod
shows it, and nslookup shows the lookups:
kubectl run dns-debug --image=busybox:1.36 -- sleep 3600
kubectl wait --for=condition=Ready pod/dns-debug
kubectl exec dns-debug -- cat /etc/resolv.conf
kubectl exec dns-debug -- nslookup web-service.default.svc.cluster.local
kubectl exec dns-debug -- nslookup web-service
$ kubectl run dns-debug --image=busybox:1.36 -- sleep 3600
pod/dns-debug created
$ kubectl wait --for=condition=Ready pod/dns-debug
pod/dns-debug condition met
$ kubectl exec dns-debug -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
$ kubectl exec dns-debug -- nslookup web-service.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: web-service.default.svc.cluster.local
Address: 10.99.235.177
$ kubectl exec dns-debug -- nslookup web-service
Server: 10.96.0.10
Address: 10.96.0.10:53
** server can't find web-service.cluster.local: NXDOMAIN
** server can't find web-service.svc.cluster.local: NXDOMAIN
** server can't find web-service.cluster.local: NXDOMAIN
Name: web-service.default.svc.cluster.local
Address: 10.99.235.177
** server can't find web-service.svc.cluster.local: NXDOMAIN
command terminated with exit code 1
10.96.0.10 is the cluster DNS Service, and the address returned is the ClusterIP from the first
example. ndots:5 means any name with fewer than five dots is tried with each search suffix
first. busybox’s nslookup prints every suffix it tried, including the NXDOMAIN misses, and
exits 1 even though web-service.default.svc.cluster.local resolved. The misses are normal; read
the Name: line.
Exposing Externally with NodePort
A NodePort Service opens the same port (here 31000) on every node and forwards it to the selected Pods.
# service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: web-nodeport
spec:
selector:
app: web
ports:
- port: 80
targetPort: 80
nodePort: 31000 # accessible at <any-node-ip>:31000
type: NodePort
kubectl apply -f service-nodeport.yaml
kubectl get svc web-nodeport
minikube ssh "curl -sI http://localhost:31000" | head -2
minikube service web-nodeport --url
$ kubectl apply -f service-nodeport.yaml
service/web-nodeport created
$ kubectl get svc web-nodeport
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web-nodeport NodePort 10.110.87.233 <none> 80:31000/TCP 0s
$ minikube ssh "curl -sI http://localhost:31000" | head -2
HTTP/1.1 200 OK
Server: nginx/1.25.5
$ minikube service web-nodeport --url
http://127.0.0.1:50734
! Because you are using a Docker driver on windows, the terminal needs to be open to run it.
80:31000/TCP reads as “Service port 80, node port 31000”. A NodePort Service still gets a
ClusterIP, so in-cluster callers can keep using web-service-style names. On the node itself,
port 31000 answers.
On Linux with a VM driver, minikube service --url prints http://<minikube ip>:31000 and
returns. With the docker driver on Windows or macOS the node’s IP is not routable from your
machine, so minikube opens a tunnel on a random local port (50734 above) and keeps the command
running. curl -sI http://127.0.0.1:50734 from a second terminal returned HTTP/1.1 200 OK.
Ingress — HTTP Routing to Multiple Services
Ingress routes by hostname and path through one controller. On minikube the controller is an addon:
minikube addons enable ingress
$ minikube addons enable ingress
* ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
* After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
- Using image registry.k8s.io/ingress-nginx/controller:v1.14.3
- Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.7
- Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.7
* Verifying ingress addon...
* The 'ingress' addon is enabled
The Ingress needs a second backend to route to. api-service runs http-echo, which answers
every request with a fixed string:
# api.yaml — a second backend for the Ingress to route to
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: echo
image: hashicorp/http-echo:1.0
args: ["-listen=:5678", "-text=hello from api"]
---
apiVersion: v1
kind: Service
metadata:
name: api-service
spec:
selector:
app: api
ports:
- port: 80
targetPort: 5678
The Ingress sends myapp.local/api to api-service and everything else on that host to web-service.
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: myapp.local
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service # route /api → api-service
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: web-service # route / → web-service
port:
number: 80
kubectl apply -f api.yaml
kubectl apply -f ingress.yaml
kubectl get ingress
minikube ssh "curl -s -H 'Host: myapp.local' http://localhost/api"
minikube ssh "curl -s -H 'Host: myapp.local' http://localhost/ | grep title"
minikube ssh "curl -s -o /dev/null -w '%{http_code}\n' http://localhost/"
$ kubectl apply -f api.yaml
deployment.apps/api created
service/api-service created
$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/app-ingress created
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
app-ingress nginx myapp.local 192.168.49.2 80 42s
$ minikube ssh "curl -s -H 'Host: myapp.local' http://localhost/api"
hello from api
$ minikube ssh "curl -s -H 'Host: myapp.local' http://localhost/ | grep title"
<title>Welcome to nginx!</title>
$ minikube ssh "curl -s -o /dev/null -w '%{http_code}\n' http://localhost/"
404
ADDRESS stays empty for the first few seconds, until the controller picks the Ingress up.
The controller listens on port 80 of the minikube node, so minikube ssh reaches it on any
OS. On Linux you can instead add <minikube ip> myapp.local to /etc/hosts and curl
http://myapp.local/api from your machine. ingressClassName: nginx names the controller that
should handle this Ingress. The rewrite-target: / annotation makes the controller send every
matched request to the backend as /, so /api/users arrives as /. http-echo answers any
path the same way, so it changes nothing here, but a real API would see the rewritten path.
Routing is by the Host header as well as the path. The same request without
Host: myapp.local matches no rule and gets the controller’s default 404, even though it hit
the same IP and port.
Port Forwarding (Local Dev)
kubectl port-forward tunnels a local port into the cluster through the API server, with no
Service type change needed. It runs in the foreground, so curl from a second terminal.
kubectl port-forward pod/web-c77955c54-5f4bl 8080:80
# second terminal:
curl -s http://localhost:8080 | grep title
$ kubectl port-forward pod/web-c77955c54-5f4bl 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
$ curl -s http://localhost:8080 | grep title
<title>Welcome to nginx!</title>
Each forwarded connection logs a Handling connection line. Stop the forward with Ctrl+C.
Forwarding to a Service looks the same, but it does not spread requests across Pods:
kubectl port-forward service/web-service 8080:80
# second terminal: four requests, then count them in each Pod's access log
for i in 1 2 3 4; do curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/pf-test; done
kubectl logs web-c77955c54-5f4bl | grep -c 'pf-test HTTP/1.1" 404'
kubectl logs web-c77955c54-6n8l8 | grep -c 'pf-test HTTP/1.1" 404'
$ kubectl port-forward service/web-service 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
$ for i in 1 2 3 4; do curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/pf-test; done
404
404
404
404
$ kubectl logs web-c77955c54-5f4bl | grep -c 'pf-test HTTP/1.1" 404'
0
$ kubectl logs web-c77955c54-6n8l8 | grep -c 'pf-test HTTP/1.1" 404'
4
The 404s are expected, since /pf-test is not a real page; they leave an easy-to-count line in
nginx’s access log. All four requests went to 6n8l8. port-forward service/... resolves the
Service to one Pod when it starts and pins every connection to it, so use it to reach a service,
not to test load balancing.
Service Debugging Checklist
Run the checks in order against a working Service first, so you know what healthy looks like.
kubectl get pods -l app=web
kubectl describe service web-service | grep -A2 Selector
kubectl get pods --show-labels
kubectl get endpoints web-service
kubectl get endpointslices -l kubernetes.io/service-name=web-service
$ kubectl get pods -l app=web
NAME READY STATUS RESTARTS AGE
web-c77955c54-5f4bl 1/1 Running 0 6m19s
web-c77955c54-6n8l8 1/1 Running 0 6m19s
$ kubectl describe service web-service | grep -A2 Selector
Selector: app=web
Type: ClusterIP
IP Family Policy: SingleStack
$ kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
api-7b74bfcf98-t7mhv 1/1 Running 0 99s app=api,pod-template-hash=7b74bfcf98
client 1/1 Running 0 5m16s run=client
dns-debug 1/1 Running 0 5m4s run=dns-debug
web-c77955c54-5f4bl 1/1 Running 0 6m19s app=web,pod-template-hash=c77955c54
web-c77955c54-6n8l8 1/1 Running 0 6m19s app=web,pod-template-hash=c77955c54
$ kubectl get endpoints web-service
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
web-service 10.244.0.3:80,10.244.0.4:80 6m19s
$ kubectl get endpointslices -l kubernetes.io/service-name=web-service
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-service-rwl4t IPv4 80 10.244.0.3,10.244.0.4 6m19s
The selector app=web matches exactly the two web- Pods, and their Pod IPs appear as
endpoints. The old Endpoints API still answers but warns from v1.33; EndpointSlices hold the
same data and are what kube-proxy reads.
Now a Service whose selector matches nothing. kubectl create service sets the selector to
app=<name>, and no Pod has app=web-typo:
kubectl create service clusterip web-typo --tcp=80:80
kubectl describe service web-typo | grep Selector
kubectl get endpointslices -l kubernetes.io/service-name=web-typo
kubectl exec client -- curl -s -m 3 http://web-typo
$ kubectl create service clusterip web-typo --tcp=80:80
service/web-typo created
$ kubectl describe service web-typo | grep Selector
Selector: app=web-typo
$ kubectl get endpointslices -l kubernetes.io/service-name=web-typo
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-typo-ckwlt IPv4 <unset> <unset> 0s
$ kubectl exec client -- curl -s -m 3 http://web-typo
command terminated with exit code 7
The name resolves, because DNS records exist for every Service whether or not it has Pods, but
the connection is refused (curl exit 7) because there is nowhere to send it. Empty endpoints mean
either the selector matches no labels or the matching Pods are not Ready. kubectl describe service plus kubectl get pods --show-labels tells you which.
Practice
- Scale
webto three replicas and confirm the Service picked up the new Pod without being edited.
Solution
kubectl scale deployment web --replicas=3
kubectl rollout status deployment/web
kubectl get endpointslices -l kubernetes.io/service-name=web-service
$ kubectl scale deployment web --replicas=3
deployment.apps/web scaled
$ kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 2 of 3 updated replicas are available...
deployment "web" successfully rolled out
$ kubectl get endpointslices -l kubernetes.io/service-name=web-service
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-service-rwl4t IPv4 80 10.244.0.3,10.244.0.4,10.244.0.15 6m34s
The new Pod carries app=web from the Deployment’s template, so it joins the endpoints as soon
as it is Ready. The Service object never changed.
- Start a curl Pod in the
financenamespace and callweb-serviceby its bare name, then by a name that works.
Solution
kubectl run fclient -n finance --image=curlimages/curl:8.10.1 -- sleep 3600
kubectl wait -n finance --for=condition=Ready pod/fclient
kubectl exec -n finance fclient -- curl -s -o /dev/null -w "%{http_code}\n" http://web-service
kubectl exec -n finance fclient -- curl -s -o /dev/null -w "%{http_code}\n" http://web-service.default
$ kubectl run fclient -n finance --image=curlimages/curl:8.10.1 -- sleep 3600
pod/fclient created
$ kubectl wait -n finance --for=condition=Ready pod/fclient
pod/fclient condition met
$ kubectl exec -n finance fclient -- curl -s -o /dev/null -w "%{http_code}\n" http://web-service
000
command terminated with exit code 6
$ kubectl exec -n finance fclient -- curl -s -o /dev/null -w "%{http_code}\n" http://web-service.default
200
From finance, the search list starts with finance.svc.cluster.local, so the bare name has no
match, and the resolver fails after working through every suffix. Adding the namespace is enough.
- Fix
web-typoso it serves thewebPods, without deleting and recreating it.
Solution
kubectl set selector service web-typo app=web
kubectl exec client -- curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://web-typo
kubectl get endpointslices -l kubernetes.io/service-name=web-typo
kubectl exec client -- curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://web-typo
$ kubectl set selector service web-typo app=web
service/web-typo selector updated
$ kubectl exec client -- curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://web-typo
200
$ kubectl get endpointslices -l kubernetes.io/service-name=web-typo
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-typo-ckwlt IPv4 80 10.244.0.15,10.244.0.3,10.244.0.4 34s
$ kubectl exec client -- curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://web-typo
200
Changing the selector updates the EndpointSlice right away, but kube-proxy then has to reprogram the node’s rules. A request fired in the same instant can still get exit code 7. If yours does, repeat it a second later.
- Port-forward local port 9090 to the
apibackend. Trydeployment/api 9090:80first and explain the failure, then make it work.
Solution
kubectl port-forward deployment/api 9090:80
# second terminal:
curl -s http://localhost:9090; echo "exit=$?"
$ kubectl port-forward deployment/api 9090:80
Forwarding from 127.0.0.1:9090 -> 80
Forwarding from [::1]:9090 -> 80
Handling connection for 9090
E1011 01:53:59.058087 17964 portforward.go:424] "Unhandled Error" err=<
an error occurred forwarding 9090 -> 80: error forwarding port 80 to pod a0646b912eb49cb069d9ca157f6b959c1bb9073c9ecc9e502f219eff512d85c8, uid : exit status 1: 2026/10/10 20:23:59 socat[9335] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused
>
error: lost connection to pod
$ curl -s http://localhost:9090; echo "exit=$?"
exit=56
The first connection kills the whole forward: kubectl exits with lost connection to pod, and
curl gets exit 56 (connection closed with no response).
With a Pod or Deployment target, the remote port is the container port, and http-echo listens
on 5678, not 80. Forward through the Service instead (or use deployment/api 9090:5678):
kubectl port-forward service/api-service 9090:80
# second terminal:
curl -s http://localhost:9090
$ kubectl port-forward service/api-service 9090:80
Forwarding from 127.0.0.1:9090 -> 5678
Forwarding from [::1]:9090 -> 5678
Handling connection for 9090
$ curl -s http://localhost:9090
hello from api
With a Service target, 80 is the Service port, and kubectl translates it to the targetPort.
That is why the log says -> 5678.
Clean up everything from this lesson with:
kubectl delete -f ingress.yaml -f api.yaml -f service-nodeport.yaml -f full-example.yaml
kubectl delete pod client dns-debug
kubectl delete service web-typo
kubectl delete namespace finance
$ kubectl delete -f ingress.yaml -f api.yaml -f service-nodeport.yaml -f full-example.yaml
ingress.networking.k8s.io "app-ingress" deleted from default namespace
deployment.apps "api" deleted from default namespace
service "api-service" deleted from default namespace
service "web-nodeport" deleted from default namespace
deployment.apps "web" deleted from default namespace
service "web-service" deleted from default namespace
$ kubectl delete pod client dns-debug
pod "client" deleted from default namespace
pod "dns-debug" deleted from default namespace
$ kubectl delete service web-typo
service "web-typo" deleted from default namespace
$ kubectl delete namespace finance
namespace "finance" deleted
Deleting the namespace also deletes everything in it, including payments and fclient.