Systems Thinking and Dynamics

Core Concepts

System Dynamics: The Foundation

When something goes wrong in production — a cascade failure, an SLO breach that refuses to stay fixed, an incident that recurs despite three separate postmortems — the instinct is often to look for the broken part. Which service failed? Which engineer made the wrong call? Which configuration value was wrong?

System dynamics, developed by Jay Forrester at MIT in the 1950s, offers a different explanation. It shows how internal system structure — feedback loops, stocks and flows, delays, and time dependencies — routinely produces instability and outcomes that no individual component or actor intended. The problem is often not a broken part. It is the system's own dynamics operating exactly as structured.

The core vocabulary of system dynamics gives us precise language for this:

• Stocks are quantities at a point in time: the number of open incidents, the size of your on-call queue, the level of technical debt.
• Flows are quantities measured over intervals: new incidents arriving per hour, tickets closed per sprint, debt accumulated per release.
• Feedback loops connect the output of a system back to its input, creating circular causality.

The behavior you observe in a system over time — whether it oscillates, explodes, stabilizes, or collapses — is largely determined by these structural elements.

---

Feedback Loops: The Engine of System Behavior

Feedback loops are fundamental to understanding complex system behavior. There are two kinds, and they behave very differently.

Balancing (negative) feedback loops counteract change. They sense a gap between current state and desired state, and apply a corrective force. An incident response process is a balancing loop: as the number of open incidents rises, on-call engineers are paged, the incident response workflow kicks in, and eventually the count is brought back down. Auto-scaling is a balancing loop: as CPU utilization rises, new instances are provisioned until utilization returns to target.

Reinforcing (positive) feedback loops amplify change. An increase leads to a further increase; a decrease leads to a further decrease. Viral growth is a reinforcing loop. So is a death spiral: as a service degrades, clients retry more aggressively, which increases load, which degrades the service further.

A feedback process operates when information about the current state of a controlled system is transmitted back to a control mechanism, which then modifies its actions to move the system toward a desired state. This sounds simple, but in practice the loop can be long, slow, noisy, or broken at several points — and each deficiency produces its own failure mode.

Fig 1

The gain around positive feedback loops determines whether systems exhibit exponential expansion, decline, or approach some limit. This matters because the gain is adjustable — and adjusting it is one of the moderate leverage points for system intervention.

---

Delays: Why Systems Oscillate

Feedback loops do not operate instantaneously. Between an event, its detection, and the corrective response lies a delay. And delays are where systems go wrong.

Delays in feedback loops introduce nonlinear behavior, where the timing of information, material, or decision flows becomes a critical leverage point. Long delays between action and consequence can lead to system oscillation, instability, and counterintuitive outcomes.

Consider a deployment pipeline with a 40-minute feedback loop — code committed, CI runs, staging deployed, load tests executed, alert fires. By the time the alert fires, the engineer may have already merged three more commits. The corrective action (roll back commit A) is now delayed, the system has changed further, and the fix may address the wrong state.

Managerial decision-making experiments demonstrate systematic divergence from optimal behavior due to delays and feedback dynamics. Individuals making rational decisions within their local context can generate irrational system-level outcomes. This is not a failure of intelligence — it is a structural property of systems with long feedback delays.

---

Emergence: The Irreducibility of System Behavior

One of the most consequential — and most resisted — ideas in systems thinking is emergence. Emergent properties are features of a system that cannot be found in any of its individual components.

Safety in complex systems is an emergent property that can only be analyzed and managed at the system level, not through examination of individual components in isolation. Accidents arise from interactions among multiple components — technical, human, and organizational — none of which by themselves would cause the accident.

This directly contradicts the instinct to decompose and conquer. A service may have 99.99% availability, well-written runbooks, an experienced on-call rotation, and a carefully reviewed architecture. None of that tells you what will happen when all three are under stress simultaneously during a partial network partition that affects only some regions, degrades (but does not break) DNS, and coincides with a botched rollback.

STAMP and CAST account for nonlinear interactions among system components that linear causal analysis approaches cannot capture. Traditional accident analysis assumes direct causal chains where individual failures lead to specific outcomes. But in complex systems, accidents often emerge from interactions — not from any single broken component.

Resilience engineering explicitly addresses safety in complex sociotechnical systems where failures result from interactions among multiple components, human factors, organizational processes, and software systems. This represents a departure from the assumption that failures are primarily localized and can be prevented by auditing components one at a time.

You cannot predict emergence from components alone — not because you lack information, but because emergence is a property of relationships, not parts.

---

Causal Texture and Environmental Complexity

Emery and Trist's concept of causal texture adds another dimension: the environment a system operates in is itself structured, and that structure varies. Some environments are stable and predictable ("placid"). Others are turbulent fields — where the environment itself is changing faster than any organization can fully track, and where cause and effect are entangled across organizational boundaries.

Production systems in 2026 often operate in environments closer to turbulent fields than placid ones: cloud provider outages, upstream dependency changes, security vulnerability disclosures, load patterns shaped by global events. Contemporary global systems exhibit increasing levels of tight coupling and interactive complexity, and that complexity propagates inward to any system coupled to them.

Different causal textures require different responses. A highly turbulent environment calls for greater organizational flexibility, more frequent sensing of the environment, and design choices that reduce tight coupling — not more detailed procedures.

---

Leverage Points: Where to Push the System

Given that system behavior emerges from structure, the natural question is: where can you intervene to change that behavior?

Donella Meadows' leverage points framework ranks intervention types from least to most effective. The hierarchy is counterintuitive: the interventions that are easiest to implement are generally the least powerful, while the most powerful interventions are often the hardest to execute and the least obvious.

Fig 2

At the bottom: parameter changes. Adjusting system parameters — constants such as SLO targets, alert thresholds, timeout values, or capacity numbers — represents the weakest leverage point. These changes are easiest to implement and often the first response to incidents. They are also the most likely to be insufficient. Changing an alert threshold does not change the feedback structure that caused the alert to fire in the first place.

Physical structure and buffers. The structure of material stocks and flows represents a weak-to-moderate leverage point. In software systems, this maps to infrastructure topology, deployment architecture, and capacity buffers. Buffer capacity determines a system's ability to absorb shocks — more spare capacity means more time to respond before things cascade. But relying on buffers without addressing underlying dynamics remains a limited strategy.

Information flows. The structure of information flows — who has access to what information, when, and in what form — represents a moderate-strength leverage point. In engineering organizations, this includes observability architecture, how alerts are routed, who receives postmortem reports, whether on-call engineers can see dependencies' health, and whether leadership receives lagging or leading indicators. Changing what information reaches which decision-maker can produce significant behavioral change without touching any parameter.

Feedback loop strengths. Strengthening stabilizing feedback mechanisms improves system resilience and capacity to maintain homeostasis. Making on-call feedback faster, shrinking the deployment feedback loop from 40 minutes to 4, or adding automated rollback — these adjust how strongly the system responds to deviation, which is meaningfully more powerful than adjusting what threshold triggers the response.

System rules. The rules of a system — incentives, punishments, and constraints — represent a moderate leverage point, more impactful than parameter changes but less powerful than structural redesign. A team that is measured on feature velocity and is not accountable for operational events will behave differently from a team that carries its own pager. Changing what is rewarded or penalized is more powerful than tuning the SLO.

System goals. Changing what a system is fundamentally designed to accomplish has profound effects on all subordinate structures, rules, and information flows. Moving from "ship features" to "deliver reliable user outcomes" is not a policy change — it is a goal change that reshapes what work is prioritized, how trade-offs are made, and what gets measured.

Mental models and paradigms. Mental models and paradigms represent the highest-leverage points for system change. The belief that reliability problems are caused by individual mistakes is a paradigm. So is the belief that enough procedures can eliminate variability. The ability to transcend paradigms — to recognize the rules of the game itself and shift to fundamentally different framings — is the most powerful intervention identified by Meadows, though also the hardest to execute.

---

System Archetypes: Recurring Failure Patterns

System dynamics research has identified recurring structural patterns — archetypes — that appear across industries and contexts. Two are especially common in engineering organizations.

Fixes that fail. A problem-solving intervention produces immediate positive effects but generates undetected side effects that eventually negate or worsen the original problem. The relief from the quick fix reduces pressure to address root causes, so root causes remain. Eventually the problem returns, often in a different form. Another fix is applied. The cycle continues, each iteration leaving the system slightly more entangled.

In production systems: an alert fires too often and is generating noise, so the threshold is raised. The immediate problem (noisy alert) is resolved. The unintended side effect: the signal-to-noise ratio improves by reducing signal. Real problems now fire the alert less reliably. The fundamental issue — that something is wrong with the system the alert monitors — is never addressed.

Shifting the burden. Pressures from worsening problems lead organizations to apply quick symptomatic fixes rather than implement more difficult fundamental solutions. This creates a reinforcing dynamic: the temporary fix alleviates pressure to address root causes, dependencies on the temporary fix grow, and the original problem intensifies.

In production systems: a service has a memory leak. The team sets up a cron job to restart the service every night. The restart alleviates the immediate user-facing symptom. But now there is less urgency to fix the leak, the restart becomes load-bearing infrastructure, and over time other systems begin depending on the restart schedule. The fundamental problem — the leak — becomes harder to address because fixing it would require decommissioning the restart, which is now depended upon.

---

Worked Example

Applying Systems Thinking to an Incident Recurrence

A team experiences repeated incidents around database connection pool exhaustion. After the third incident in two months, leadership asks for a permanent fix.

Round 1: Parameter change (lowest leverage). The team increases the connection pool size from 100 to 200. Incidents stop for three weeks. Then they resume at the new threshold. The fix worked temporarily but the underlying demand growth was never addressed. This is a textbook "fix that fail."

Round 2: Buffer change (weak-moderate leverage). The team adds read replicas to distribute load. This buys more headroom. But the application query patterns are inefficient — N+1 queries, missing indexes, unbounded pagination. The new replicas buy time, not resolution.

Round 3: Information flow change (moderate leverage). The team adds slow query logging and surfaces it in the team's observability dashboard. Now engineers can see the specific queries driving exhaustion before they cross the alert threshold. Several N+1 patterns are identified and fixed within a sprint. Connection pool exhaustion incidents drop significantly.

Round 4 (not taken): Rules change. The team does not yet have a policy requiring query analysis in code review, or an SLO for database query latency. So the efficiency of the information flow depends on engineers proactively checking the dashboard. The information flow improvement is real but fragile.

Where to push next: Introducing database query performance as part of the definition of done, and tracking query latency in the service's SLO, would move the intervention to a higher leverage tier — changing what is measured and enforced, not just what is visible.

This example illustrates the hierarchy in practice: each intervention is more durable than the last, and each one requires more organizational effort.

---

Common Misconceptions

"We just need to tune the right parameters."

Parameter changes represent the weakest leverage point in Meadows' hierarchy. Changing timeouts, thresholds, and resource allocations is often necessary but rarely sufficient. Systems with structural feedback problems will reassert those problems at the new parameter values.

"If we write enough procedures, we can eliminate variability."

Complexity and variability are inherent, unavoidable features of real work that cannot be eliminated through procedural control alone. Workers in complex operational environments face competing goals, time pressure, and unexpected situations that require continuous adaptation. Procedures capture what was anticipated, not what will happen.

"Problems come from broken parts."

Accidents in complex systems often result from emergent properties arising from interactions between multiple components — none of which by themselves would cause the accident. Searching for the single broken component misses the interaction. This is why postmortems that conclude "root cause: engineer made wrong call" tend to produce fixes that fail.

"My system behaves the way it does because of what's happening externally."

System dynamics reveals how internal system structure — feedback loops, delays, and information flows — often produces organizational instability and counterintuitive behaviors. External events can trigger dynamics, but the shape of the response is determined by internal structure. The same external shock applied to two differently structured systems produces two completely different trajectories.

---

Compare & Contrast

Reinforcing vs. Balancing Feedback Loops in Production

	Balancing loop	Reinforcing loop
Effect	Counteracts change; moves toward equilibrium	Amplifies change; drives exponential growth or collapse
Response to perturbation	Returns toward set point (if functioning)	Accelerates away from initial state
Example	Auto-scaling CPU utilization	Retry storms under degradation
When it's a problem	When delay is long enough to cause oscillation	When there is no balancing loop to stop it
Leverage action	Strengthen the loop (faster, more accurate feedback)	Reduce the gain (rate limiting, backoff, circuit breakers)

High-Leverage vs. Low-Leverage Interventions

	Low leverage	High leverage
What changes	Parameter values (thresholds, sizes, rates)	Information flows, rules, feedback structure, goals, paradigms
Ease of implementation	Easy and fast	Difficult and slow
Durability	Temporary; problem re-emerges at new value	Structural; changes the dynamic
Where it appears in incident response	Corrective actions: "increase timeout to 30s"	Corrective actions: "change SLO to include dependency latency"; "make query perf part of code review"
Risk of misuse	Creates false sense of resolution	Can face organizational resistance; benefits lag

---

Active Exercise

Map a Recent Incident to the Leverage Hierarchy

Take the most recent significant incident your team has experienced. Work through the following questions individually, then discuss as a team.

1. Describe the feedback loop that failed. Was it a balancing loop that should have corrected a problem but did not? A reinforcing loop that escalated faster than expected? Was there a delay involved? How long was it?

2. Identify the corrective actions from the postmortem. For each action item, place it on the leverage hierarchy:

   • Parameter change (timeout, threshold, resource allocation)?
   • Buffer change (added capacity, added redundancy)?
   • Information flow change (new alert, better dashboard, faster feedback)?
   • Rules change (changed what gets measured or enforced)?
   • Goal or paradigm change?

3. Ask the "fixes that fail" question. For each corrective action: is there a plausible side effect that could negate this fix over time? Does the fix reduce pressure to address a more fundamental issue?

4. Identify the highest-leverage intervention that was not taken. Based on your analysis: what would a higher-leverage action have looked like? What prevented the team from taking it?

5. Check for the shifting burden pattern. Is there any temporary mitigation (a cron job, a manual process, an exception in the code) that has become load-bearing infrastructure? What is the original problem it was meant to address?

Aim to complete steps 1–3 in 20 minutes, then discuss as a group for 15.