GitOps: Principles and Practices

GitOps is an operational framework that applies DevOps best practices to infrastructure automation. The core idea: Git is the single source of truth for declarative infrastructure and applications. Changes to the system happen through Git commits, and automated processes ensure the live state matches the desired state in Git.

Here’s how to implement GitOps effectively.

Core Principles

Declarative Configuration

Everything is defined declaratively:

# infrastructure/production/api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: myregistry/api:v1.2.3
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"

Not imperative scripts. Declarative manifests.

Git as Source of Truth

The desired state lives in Git:

infrastructure/
├── base/                 # Shared configurations
│   ├── namespace.yaml
│   └── network-policy.yaml
├── production/           # Production overlay
│   ├── kustomization.yaml
│   ├── api-deployment.yaml
│   └── replicas-patch.yaml
└── staging/              # Staging overlay
    ├── kustomization.yaml
    └── api-deployment.yaml

If it’s not in Git, it shouldn’t exist. If it exists, it should be in Git.

Versioned and Immutable

Every change is:

Committed with a message
Tied to an author
Part of history
Revertible

git log --oneline
abc123 Increase API replicas to 5
def456 Update API to v1.2.3
ghi789 Add resource limits to API deployment

Automatic Synchronization

Automated agents continuously reconcile:

Git Repository (desired state)
       ↓
   GitOps Agent (ArgoCD, Flux)
       ↓
   Compares with cluster
       ↓
   Applies differences
       ↓
Kubernetes Cluster (actual state)

The cluster converges to the declared state automatically.

GitOps Workflow

Standard Flow

Developer → PR → Review → Merge → Sync → Deploy

Developer makes changes to manifests
PR created for review
Review by team (and automated checks)
Merge to main branch
Sync agent detects change
Deploy agent applies to cluster

No `kubectl apply` in Production

Direct cluster access is for emergencies only:

# Bad - bypasses GitOps
kubectl apply -f deployment.yaml

# Good - make change in Git
git checkout -b update-deployment
# Edit files
git commit -m "Update deployment"
git push && create PR

Direct changes create drift from Git state.

Handling Emergencies

When you must change directly:

Make the emergency change
Immediately commit the change to Git
Document why bypass was necessary
Review and improve process

The goal: Git always reflects actual state.

Repository Structure

Single Repository

Everything in one repo:

repo/
├── apps/
│   ├── api/
│   └── web/
├── infrastructure/
│   ├── cert-manager/
│   └── ingress/
└── environments/
    ├── staging/
    └── production/

Pros: Simple, atomic changes across components Cons: Scales poorly, permissions difficult

Split Repositories

Application and infrastructure separate:

# app-manifests/
apps/
├── api/
└── web/

# infra-manifests/
infrastructure/
├── cert-manager/
└── ingress/

Pros: Clear separation, independent ownership Cons: Coordinating changes is harder

App Repository Pattern

Application code and manifests together:

# api-service/
src/
tests/
Dockerfile
k8s/
├── base/
└── overlays/
    ├── staging/
    └── production/

Pros: Developers own their deployment Cons: Platform changes need many PRs

Tooling

Flux

# Install Flux
flux bootstrap github \
  --owner=myorg \
  --repository=infrastructure \
  --path=clusters/production

Flux components:

Source Controller: Watches Git repos
Kustomize Controller: Applies Kustomize overlays
Helm Controller: Manages Helm releases
Notification Controller: Sends alerts

ArgoCD

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Create Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/manifests
    targetRevision: main
    path: apps/api/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ArgoCD features:

Web UI for visualization
Multi-cluster support
RBAC integration
Health status monitoring

Comparison

Feature	Flux	ArgoCD
Architecture	Distributed	Centralized
UI	Basic	Rich web UI
Multi-cluster	Native	Supported
Helm	Controller	Native
Learning curve	Moderate	Lower

Both are excellent. Choose based on team preference.

Kustomize for Environments

Base and Overlays

apps/api/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml
    │   └── replicas-patch.yaml
    └── production/
        ├── kustomization.yaml
        └── replicas-patch.yaml

base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

overlays/production/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
patches:
  - replicas-patch.yaml
images:
  - name: myregistry/api
    newTag: v1.2.3

Common Customizations

Replicas:

# replicas-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 5

Resources:

# resources-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
      - name: api
        resources:
          requests:
            memory: "512Mi"
            cpu: "200m"

Image Updates

Manual Updates

Update image tag in Git:

# Update this line
image: myregistry/api:v1.2.3
# To
image: myregistry/api:v1.2.4

Simple, explicit, auditable.

Automated Image Updates

Flux Image Automation:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: api
spec:
  image: myregistry/api
  interval: 1m
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: api
spec:
  imageRepositoryRef:
    name: api
  policy:
    semver:
      range: ">=1.0.0"
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: api
spec:
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux
        email: flux@example.com
      messageTemplate: "Update {{.AutomationName}}"
    push:
      branch: main
  update:
    path: ./apps/api
    strategy: Setters

Flux automatically creates commits when new images are available.

Security Considerations

Secrets in Git

Don’t store plain secrets in Git. Options:

Sealed Secrets:

# Encrypt secret
kubeseal --cert pub-cert.pem < secret.yaml > sealed-secret.yaml

SOPS:

# Encrypt values
sops --encrypt --age age1... secrets.yaml > secrets.enc.yaml

External Secrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    name: api-secrets

RBAC for Repositories

Control who can change what:

Separate repos for different environments
Branch protection rules
Required reviews for production changes
Signed commits for auditability

Key Takeaways

Everything is declarative and version controlled in Git
Git is the single source of truth; cluster state follows Git
No direct cluster modifications (except emergencies)
Use overlays (Kustomize) for environment-specific configuration
Choose Flux or ArgoCD based on team preference
Automate image updates carefully—sometimes manual is better
Never store plain secrets in Git; use encryption or external secrets
Apply RBAC to repositories with branch protection and required reviews

GitOps brings software development rigor to operations. Start with one application, learn the workflow, then expand.