Infrastructure Configuration Drift

Annotated Case Study

Kubernetes: Where Configurability Meets Entropy

Kubernetes is the clearest existing lens for understanding configuration drift in practice. It is highly configurable by design — clusters expose hundreds of tunable parameters across workloads, namespaces, admission controllers, resource quotas, and networking policies. That expressiveness is the point. But it is also why Kubernetes is acknowledged to affect 40% of users with configuration drift, with measurable impacts on environment stability.

The mechanism is straightforward. Under operational pressure — a degrading workload, an urgent memory limit adjustment, a security patch applied mid-incident — engineers reach for kubectl commands that directly modify live cluster state. The change works. The incident resolves. The documentation does not get updated. A week later, another engineer adjusts the same resource using the YAML definition in Git, not knowing the live state has already diverged. The cluster now reflects two partial realities.

What drift looks like operationally in Kubernetes:

• Workload performance degradation from outdated images in running pods while Git manifests reference current versions.
• Deployment failures caused by inconsistent memory limits across namespaces — some adjusted manually, some still at declared defaults.
• Security vulnerabilities from settings that have drifted from secure defaults established at cluster setup.
• Admission controller rejections for resources whose live state no longer matches expected schema.

Each of these failure modes shares a common structure: the running system and the declared system have become different systems, and the team does not know it until something breaks.

The Cost of Not Knowing

The operational cost of unmanaged drift is not hypothetical. Komodor's 2025 Enterprise Kubernetes Report indicates teams spend an average of 34 workdays annually troubleshooting drift-related incidents. That figure does not include the cost of the outages themselves — only the investigation time.

Real incidents illustrate the pattern:

• Reddit Engineering's November 2024 outage stemmed from a daemonset configuration issue — a component that had drifted from its declared state.
• Cyble experienced a four-hour production disruption in 2025 from cluster misconfiguration, disrupting threat intelligence operations during peak ransomware activity.
• Spotify's 2018 incident required three hours of cluster reconstruction after accidental deletion, a direct consequence of state not being reliably reproducible from declared configuration.

Nearly 80% of production incidents trace to recent system changes, and configuration drift fuels approximately 70% of outages according to a 2025 Gartner finding cited in Komodor's report.

Beyond direct outage costs, drift manifests as quiet infrastructure bloat. Approximately 65% of workloads run idle below half their requested resources — the result of resource allocations that were manually adjusted upward during incidents and never reconciled downward.

Why Manual Changes Are the Root Cause

The primary mechanism through which drift originates is direct manual modification of live systems — kubectl edit, console modifications, direct server configuration. When team members bypass version control and Infrastructure as Code processes to make direct changes, they create a split between what the system knows about itself (Git) and what the system actually is (running state).

The problem is not that engineers make manual changes. Under pressure, direct intervention is often the right call. The problem is undocumented manual changes — changes that do not get reconciled back into the declared state after the incident clears. GitOps literature frames this precisely: drift occurs when someone makes a manual change that circumvents the version control system.

Key Principles

1. You cannot detect drift without a baseline

The precondition for any drift management strategy is a version-controlled baseline: a single source of truth for what the declared desired state actually is. Without a clearly defined and version-controlled baseline, determining what constitutes drift becomes subjective. You cannot tell if something has moved if you do not know where it started.

This means Infrastructure as Code is not optional for teams that want to manage drift. Terraform, AWS CloudFormation, Kubernetes manifests in Git — the specific tool is less important than the discipline: every resource's desired state must exist in version control, and that version must be authoritative.

2. Undocumented overrides accumulate into emergent complexity

Single overrides are manageable. The danger is accumulation. As multiple team members make undocumented modifications over time to address immediate operational needs, each individually reasonable change compounds with the others. The result is a system whose actual behavior cannot be predicted from its declared configuration — a system that must be reverse-engineered from observation.

Teams that override framework conventions without systematic documentation accumulate what practitioners call stray configs: custom YAML files, environment toggles, duplicated dependency versions, inconsistent overrides. Regular audits to identify and strip unnecessary overrides are necessary to maintain the benefits of declared configuration.

3. Document deviations at the time of deviation

The practice that separates well-maintained infrastructure from chaotic infrastructure is how systematically exceptions are documented. A lightweight "Deviation Log" — or more formally, an Architecture Decision Record (ADR) — records why each override was made, and what state was intentionally chosen in place of the default.

ADRs are endorsed as best practice by AWS, Microsoft, and the UK Government Digital Service. The mechanism is simple: when you deviate from declared state intentionally, you write down what you did and why. This practice directly addresses the exclusion problem in drift detection — it tells your tooling which deviations are legitimate, and it tells future engineers which inconsistencies are decisions versus accidents.

4. Start with detection before enabling auto-remediation

Most teams implementing drift management begin with detection and alerting while incrementally building confidence in automatic remediation. Enabling auto-sync before exclusion lists are complete creates a new class of incident: the reconciliation loop, where a GitOps controller repeatedly reverts a legitimate change made by another controller (Istio, cert-manager, Crossplane).

The recommended sequence is: define the baseline in Git, enable detection and alerting, identify legitimate variations that need exclusion, document those exclusions, then gradually enable auto-remediation for well-understood resources.

Compare & Contrast

GitOps Continuous Reconciliation vs. Immutable Infrastructure

Both GitOps and immutable infrastructure prevent drift, but they do so through fundamentally different mechanisms — and they have different prerequisites, costs, and limits.

GitOps with continuous reconciliation uses tools like ArgoCD and Flux to continuously compare live cluster state against the desired state declared in Git. When drift is detected, the system can alert operators or automatically reconcile back to declared state. The Git repository is the single source of truth; controllers automatically revert manual changes during reconciliation cycles.

The limit of this approach is that drift can still occur between reconciliation cycles. A manual change made at 14:00 may not be detected and reverted until 14:05. For most production workloads that window is acceptable; for security-sensitive environments it may not be.

Immutable infrastructure prevents drift by eliminating the mutable state that allows servers to gradually diverge. Resources cannot be modified after creation. All changes are implemented by building a new image, checking it into version control, and deploying replacement resources through automated processes. There is no mechanism for undocumented post-deployment modifications because there is no in-place modification at all.

The limit of this approach is operational cost: it requires a full image pipeline, eliminates hotfixes, and mandates that every change — including emergency patches — go through the build process. For teams still dependent on in-place configuration management, immutable infrastructure is a significant architectural prerequisite, not just a tooling choice.

Policy-as-Code as a Compliance Layer

Policy-as-code engines like Open Policy Agent (OPA) operate differently from both GitOps and immutable infrastructure. Rather than correcting drift after it occurs or preventing mutation, policy-as-code defines compliance boundaries that are enforced at admission time — before misconfiguration reaches the running system.

OPA provides a declarative language for specifying what configurations are allowed. Policies codified in version control with automated enforcement identify configuration deviations early, before they escalate into operational problems. In Kubernetes, this typically runs as an admission webhook — any resource definition that violates policy is rejected at apply time.

The relationship between policy-as-code and escape hatches from convention (covered in module 03) is direct: policy-as-code is how you enforce that escape hatches are used deliberately. A team that has documented its intentional deviations in ADRs can codify those exceptions into OPA policy — making the escape hatch an explicitly permitted path rather than a silent inconsistency.

Boundary Conditions

Where GitOps reconciliation breaks down

GitOps assumes that the Git repository contains a complete and accurate desired state for everything in the cluster. That assumption fails when:

• Stateful workloads hold runtime state that should not be reverted. A database with live transactions is not the same as a stateless deployment. Auto-reconciliation against a stale Git manifest can destroy data.
• Other controllers legitimately modify resources. Istio injects sidecars. cert-manager updates certificate resources. Crossplane manages cloud resources. These controllers legitimately modify resources, and auto-remediating their changes creates reconciliation loops that destabilize clusters.
• Emergency changes need to survive until the next Git push. If auto-sync is enabled without a recovery path, a necessary hotfix will be silently reverted before it can be committed.

The practical answer to all three cases is the exclusion list — resources and fields explicitly excluded from reconciliation. But this requires deliberate planning. Best practice is to build exclusion lists before enabling auto-sync, not after. Most teams start conservative and tighten exclusions over time as they build confidence.

Where immutable infrastructure is inappropriate

Immutable infrastructure is effective at preventing drift but imposes a cost structure that is incompatible with some operational contexts:

• Long-lived, stateful infrastructure (databases, legacy applications that cannot be containerized) does not fit the replace-not-repair model.
• Environments with slow or expensive image pipelines where build times are measured in tens of minutes may find that every configuration change requires a full release cycle — eliminating the operational flexibility that motivated configurability in the first place.
• Organizations that have not yet established a version-controlled IaC baseline cannot benefit from immutable infrastructure. The prerequisite to immutability is a complete, authoritative declaration of what should be deployed.

The exclusion problem: intentional vs. accidental drift

Detection tooling cannot inherently distinguish between drift that is intentional and drift that is accidental. A resource that differs from its Git manifest might be the result of a deliberate architectural decision or an undocumented hotfix from six months ago. Without documentation, these are indistinguishable.

This is why the exclusion problem is genuinely hard. Determining what constitutes legitimate drift versus acceptable variation is non-trivial and context-dependent. The technical answer (exclusion lists) is only effective if it is paired with the organizational answer (ADRs and deviation logs). A team that auto-excludes resources without documenting why will eventually have an exclusion list that is as undocumented as the drift it was meant to manage.