DevOps · lesson 5 of 8
Intermediate
7- 2 Choosing a CI/CD Toolchain: The Same Weighted Framework as Choosing a Cloud
- 3 CI/CD Pipeline Patterns: Sequential, Parallel, and Staged
- 4 Deployment Strategies Compared: Rolling, Blue-Green, and Canary
- 5 GitOps in Practice: Git as the Source of Truth
- 6 Computing DORA Metrics From Your Own Git History
- 7 Terraform vs. Ansible: A Weighted Decision Framework
- 8 Observability Synthesis: Logs, Metrics, Traces, and Alerts as One Signal
Also part of DevOps
12- + PR Workflow: Rebasing, Squash Merge, and Merge Commits from Git & GitHub
- + GitHub Actions: Workflows, Jobs, and a Real Failing Step from GitHub Actions
- + Jenkins CI: Pipeline Basics from Jenkins
- + Bash DevOps Scripts from Bash
- + Introduction to Docker from Docker
- + Deployments and Rollouts from Kubernetes
- + Terraform Fundamentals (IaC) from Terraform
- + Terraform CI/CD with GitHub Actions (plan/apply) from Terraform
- + Ansible Idempotency, Verified: Run It Twice and Watch changed=0 from Ansible
- + Prometheus Scrapes You — It's Pull, Not Push from Monitoring & Observability
- + A Firing Alert Doesn't Mean Anyone Was Notified from Monitoring & Observability
- + A CI Security Gate: Gitleaks, Bandit and pip-audit in One Workflow from DevSecOps (AppSec in the pipeline)
Also part of: Kubernetes
GitOps in Practice: Git as the Source of Truth
Build a real sync loop that reads desired state from a git repo, applies it to a separate cluster-state file, and self-heals manual drift back to what git says.
By SkillByExample · Published
A commit changes what should be running. A sync script changes what actually is — and reverses anyone who skips the first step.
Two states, one file, one script
The desired state lives in a git repo. The cluster state lives as a plain JSON file representing “what’s actually deployed right now” — a stand-in for querying a real cluster:
$ mkdir gitops-repo && cd gitops-repo && git init -q
$ echo '{"image": "app:v1", "replicas": 2}' > deployment.json
$ git add deployment.json && git commit -q -m "initial deployment: v1, 2 replicas"
The sync script reads both, and only touches the cluster side if they disagree:
# sync.py
import json, os
DESIRED = "gitops-repo/deployment.json"
CLUSTER = "cluster-state/deployment.json"
with open(DESIRED) as f:
desired = json.load(f)
cluster = None
if os.path.exists(CLUSTER):
with open(CLUSTER) as f:
cluster = json.load(f)
if cluster == desired:
print(f"No drift: cluster matches git ({desired})")
else:
print(f"Drift detected: cluster={cluster} desired={desired}")
os.makedirs(os.path.dirname(CLUSTER), exist_ok=True)
with open(CLUSTER, "w") as f:
json.dump(desired, f)
print(f"Applied: cluster now matches git ({desired})")
$ py sync.py
Drift detected: cluster=None desired={'image': 'app:v1', 'replicas': 2}
Applied: cluster now matches git ({'image': 'app:v1', 'replicas': 2})
$ py sync.py
No drift: cluster matches git ({'image': 'app:v1', 'replicas': 2})
The first run has nothing to compare against, so it applies immediately. The second run finds
the two states already equal and does nothing — this idempotence is what makes it safe to run
sync.py on a loop or a cron schedule instead of only after a deploy.
A real change flows through git
Bump the version and scale up — as a normal git commit, nothing special:
$ echo '{"image": "app:v2", "replicas": 3}' > gitops-repo/deployment.json
$ git -C gitops-repo commit -am "bump to v2, scale to 3 replicas"
$ py sync.py
Drift detected: cluster={'image': 'app:v1', 'replicas': 2} desired={'image': 'app:v2', 'replicas': 3}
Applied: cluster now matches git ({'image': 'app:v2', 'replicas': 3})
Nothing here talked to a deploy API or ran a script against the cluster directly — the commit changed the desired state, and the next sync noticed the difference and reconciled it. This is the entire GitOps loop: commit, then reconcile.
Drift from outside git gets reverted
Now someone “fixes” the cluster directly — a common real-world shortcut under pressure, and exactly the case GitOps is designed to catch:
$ echo '{"image": "app:v2-hotfix", "replicas": 1}' > cluster-state/deployment.json
$ cat cluster-state/deployment.json
{"image": "app:v2-hotfix", "replicas": 1}
$ py sync.py
Drift detected: cluster={'image': 'app:v2-hotfix', 'replicas': 1} desired={'image': 'app:v2', 'replicas': 3}
Applied: cluster now matches git ({'image': 'app:v2', 'replicas': 3})
The manual edit didn’t go through git, so as far as sync.py is concerned it’s just drift —
the next sync silently overwrites it back to whatever the repo says. That’s not a bug in the
script; it’s the whole value proposition. If the hotfix was actually correct, it belongs in a
commit, not a live edit — otherwise the next sync (or the next engineer’s laptop running this
same script) erases it.
Practice
1. What happens if you run `sync.py` twice in a row right after a drift correction?
The second run prints No drift: cluster matches git (...) — once the sync has applied the desired state, cluster and git agree, so the comparison at the top of the script short-circuits and nothing is written. This is what makes it safe to run continuously rather than only in response to an event.
2. Add a `git log -1 --format=%H` call to `sync.py` and include the commit hash in both print statements.
commit = subprocess.run(["git", "-C", "gitops-repo", "log", "-1", "--format=%H"],
capture_output=True, text=True).stdout.strip()
Include commit[:7] in the f-strings. Now every sync’s output records exactly which commit the cluster is synced to — useful for answering “what commit is actually running in prod right now” without checking a deploy log.
3. Simulate two conflicting manual edits between syncs — first a hotfix, then a second manual edit before `sync.py` ever runs. What does the next sync do?
Overwrite cluster-state/deployment.json twice in a row without running sync.py between edits. Only the second manual edit is ever visible when sync.py finally runs — reading a file always returns its current contents, not its edit history — and that’s what gets compared against and overwritten. Git-based drift detection has no memory of intermediate states the way git log does for the desired-state repo, which is one real argument for keeping frequent, short sync intervals rather than long ones.
4. What if the desired-state file has invalid JSON — a commit with a syntax error?
Change gitops-repo/deployment.json’s content to {"image": "app:v3", "replicas":} (a trailing comma or missing value) and commit it, then run sync.py. json.load raises a json.decoder.JSONDecodeError and the script crashes before touching the cluster state — the last known-good cluster state stays exactly as it was. A broken commit fails loudly rather than silently corrupting the live system, which is a real (if accidental) safety property of reading desired state fresh on every sync.
5. Extend the script to refuse to apply a desired state that reduces `replicas` to 0, printing a warning instead.
if desired.get("replicas") == 0:
print(f"Refusing to apply {desired}: replicas would be 0")
else:
# existing drift-detection and apply logic
This is a small, real example of a policy gate inside a sync loop — the kind of check real GitOps tools (Argo CD’s sync policies, Flux’s admission checks) run before reconciling, so that a bad commit to the desired-state repo doesn’t get applied blindly just because it’s the newest thing in git.