Skip to main content
DevOps · lesson 5 of 8
DevOps intermediate Lesson 5 of 8

Also part of: Kubernetes

GitOps in Practice: Git as the Source of Truth

Build a real sync loop that reads desired state from a git repo, applies it to a separate cluster-state file, and self-heals manual drift back to what git says.

By SkillByExample · Published

devops gitops git

A commit changes what should be running. A sync script changes what actually is — and reverses anyone who skips the first step.

Two states, one file, one script

The desired state lives in a git repo. The cluster state lives as a plain JSON file representing “what’s actually deployed right now” — a stand-in for querying a real cluster:

$ mkdir gitops-repo && cd gitops-repo && git init -q
$ echo '{"image": "app:v1", "replicas": 2}' > deployment.json
$ git add deployment.json && git commit -q -m "initial deployment: v1, 2 replicas"

The sync script reads both, and only touches the cluster side if they disagree:

# sync.py
import json, os

DESIRED = "gitops-repo/deployment.json"
CLUSTER = "cluster-state/deployment.json"

with open(DESIRED) as f:
    desired = json.load(f)

cluster = None
if os.path.exists(CLUSTER):
    with open(CLUSTER) as f:
        cluster = json.load(f)

if cluster == desired:
    print(f"No drift: cluster matches git ({desired})")
else:
    print(f"Drift detected: cluster={cluster} desired={desired}")
    os.makedirs(os.path.dirname(CLUSTER), exist_ok=True)
    with open(CLUSTER, "w") as f:
        json.dump(desired, f)
    print(f"Applied: cluster now matches git ({desired})")
$ py sync.py
Drift detected: cluster=None desired={'image': 'app:v1', 'replicas': 2}
Applied: cluster now matches git ({'image': 'app:v1', 'replicas': 2})
$ py sync.py
No drift: cluster matches git ({'image': 'app:v1', 'replicas': 2})

The first run has nothing to compare against, so it applies immediately. The second run finds the two states already equal and does nothing — this idempotence is what makes it safe to run sync.py on a loop or a cron schedule instead of only after a deploy.

A real change flows through git

Bump the version and scale up — as a normal git commit, nothing special:

$ echo '{"image": "app:v2", "replicas": 3}' > gitops-repo/deployment.json
$ git -C gitops-repo commit -am "bump to v2, scale to 3 replicas"
$ py sync.py
Drift detected: cluster={'image': 'app:v1', 'replicas': 2} desired={'image': 'app:v2', 'replicas': 3}
Applied: cluster now matches git ({'image': 'app:v2', 'replicas': 3})

Nothing here talked to a deploy API or ran a script against the cluster directly — the commit changed the desired state, and the next sync noticed the difference and reconciled it. This is the entire GitOps loop: commit, then reconcile.

Drift from outside git gets reverted

Now someone “fixes” the cluster directly — a common real-world shortcut under pressure, and exactly the case GitOps is designed to catch:

$ echo '{"image": "app:v2-hotfix", "replicas": 1}' > cluster-state/deployment.json
$ cat cluster-state/deployment.json
{"image": "app:v2-hotfix", "replicas": 1}

$ py sync.py
Drift detected: cluster={'image': 'app:v2-hotfix', 'replicas': 1} desired={'image': 'app:v2', 'replicas': 3}
Applied: cluster now matches git ({'image': 'app:v2', 'replicas': 3})

The manual edit didn’t go through git, so as far as sync.py is concerned it’s just drift — the next sync silently overwrites it back to whatever the repo says. That’s not a bug in the script; it’s the whole value proposition. If the hotfix was actually correct, it belongs in a commit, not a live edit — otherwise the next sync (or the next engineer’s laptop running this same script) erases it.

Practice

1. What happens if you run `sync.py` twice in a row right after a drift correction?

The second run prints No drift: cluster matches git (...) — once the sync has applied the desired state, cluster and git agree, so the comparison at the top of the script short-circuits and nothing is written. This is what makes it safe to run continuously rather than only in response to an event.

2. Add a `git log -1 --format=%H` call to `sync.py` and include the commit hash in both print statements.
commit = subprocess.run(["git", "-C", "gitops-repo", "log", "-1", "--format=%H"],
                         capture_output=True, text=True).stdout.strip()

Include commit[:7] in the f-strings. Now every sync’s output records exactly which commit the cluster is synced to — useful for answering “what commit is actually running in prod right now” without checking a deploy log.

3. Simulate two conflicting manual edits between syncs — first a hotfix, then a second manual edit before `sync.py` ever runs. What does the next sync do?

Overwrite cluster-state/deployment.json twice in a row without running sync.py between edits. Only the second manual edit is ever visible when sync.py finally runs — reading a file always returns its current contents, not its edit history — and that’s what gets compared against and overwritten. Git-based drift detection has no memory of intermediate states the way git log does for the desired-state repo, which is one real argument for keeping frequent, short sync intervals rather than long ones.

4. What if the desired-state file has invalid JSON — a commit with a syntax error?

Change gitops-repo/deployment.json’s content to {"image": "app:v3", "replicas":} (a trailing comma or missing value) and commit it, then run sync.py. json.load raises a json.decoder.JSONDecodeError and the script crashes before touching the cluster state — the last known-good cluster state stays exactly as it was. A broken commit fails loudly rather than silently corrupting the live system, which is a real (if accidental) safety property of reading desired state fresh on every sync.

5. Extend the script to refuse to apply a desired state that reduces `replicas` to 0, printing a warning instead.
if desired.get("replicas") == 0:
    print(f"Refusing to apply {desired}: replicas would be 0")
else:
    # existing drift-detection and apply logic

This is a small, real example of a policy gate inside a sync loop — the kind of check real GitOps tools (Argo CD’s sync policies, Flux’s admission checks) run before reconciling, so that a bad commit to the desired-state repo doesn’t get applied blindly just because it’s the newest thing in git.

Frequently Asked Questions

What is GitOps, in one sentence?
GitOps is deploying by committing the desired state to a git repository and running a sync process that continuously reconciles the live system to match — instead of running deploy commands by hand or from a CI script that pushes changes directly.
What makes GitOps different from a normal CI/CD pipeline that runs `kubectl apply` or `terraform apply`?
Direction of control. A push-based pipeline runs commands against the live system whenever a build finishes. GitOps is pull-based and continuous: a sync process repeatedly compares live state to git and corrects any difference, including changes nobody's pipeline made — like a manual hotfix applied directly to the cluster.
What is configuration drift and why does GitOps care about it?
Drift is any difference between what's actually running and what the source of truth (git) says should be running — usually caused by a manual change made outside the normal deploy path. GitOps treats git as authoritative, so a sync loop detects drift and overwrites the live state back to match git, rather than treating the manual change as valid.
Does GitOps require Kubernetes?
No — Kubernetes is the most common target because tools like Argo CD and Flux were built for it, but the underlying pattern (git holds desired state, a sync loop reconciles live state to it) applies to any system that has a way to inspect and apply configuration, including plain servers managed by config management tools.
If someone manually fixes a production issue directly on the cluster, won't GitOps just undo their fix?
Yes, on the next sync — which is exactly the point. The fix needs to go into the git repo, not just the live system, or the next reconciliation reverts it. This is a deliberate design choice: it forces every change to flow through git, so the repo never drifts out of sync with reality.