Empirical Methodology for Prompts

Core Concepts

The Replication Problem in Prompt Engineering

Prompt engineering has a dirty secret: most published technique comparisons are not statistically validated. A systematic survey of prompt engineering techniques found that zero-shot techniques show a general lack of statistically significant differences across nearly all tested methods. Even more troubling — when techniques are applied in slightly different experimental setups, claimed benefits often disappear or become inconsistent.

This is not a minor caveat — it is a structural problem. The prompt engineering field is in a state similar to early nutrition research or early psychology before pre-registration: the majority of published findings represent plausible stories, not validated effects.

For a practitioner, this has a practical implication: your intuition about which prompt is "better" is probably not evidence that it is better. A prompt that scores higher on three test cases may score lower on fifty. Without a principled evaluation design, you are pattern-matching on noise.

Self-Consistency Sampling

Self-consistency is one of the few techniques with solid empirical grounding. The core idea: instead of treating a single model response as the answer, sample multiple independent completions and aggregate by majority vote.

The canonical validation comes from chain-of-thought prompting on GSM8K (Grade School Math), where self-consistency produced a +17.9% improvement over the greedy decoding baseline. Critically, this gain is not tied to a specific prompt wording. When evaluated across three manually written CoT prompt sets, self-consistency consistently outperformed the greedy baseline regardless of the exact prompt format — which is a rare empirical property for a prompt engineering technique to have.

Self-consistency does not tell you whether your prompt is good. It tells you whether the model's reasoning is stable. Instability under sampling is a signal of a poorly specified task or ambiguous instructions — and that signal is actionable.

What makes self-consistency useful as an evaluation tool beyond just improving accuracy is that the variance across samples is itself diagnostic. A prompt where 9 of 10 samples agree is expressing a well-specified problem. A prompt where samples scatter across 5 different answers is showing you a specification problem.

The Tripartite Structure of Prompt Acts

Before you can evaluate a prompt, you need a clear model of what it is doing. Austin's speech act theory provides a useful framework. Every prompt has three levels:

• Locutionary: the literal text of the prompt — the tokens you send
• Illocutionary: what you are doing by sending it — requesting, instructing, constraining, eliciting
• Perlocutionary: the causal effect on the model's output — what actually happens as a result

Most prompt evaluation mistakes happen because practitioners measure the locutionary level (did I include the right keywords?) while caring about the perlocutionary level (did the output improve?). The illocutionary level — what action the prompt actually performs in context — is the load-bearing part that is most often left implicit and unexamined.

A prompt evaluation design must specify what perlocutionary effect it is testing for, or it is not testing anything meaningful.

Grounding in Multi-Turn Interactions

Conversation analysis describes how participants in a dialogue use grounding moves to establish shared understanding. Turn-constructional units and transition-relevance places define the structural points where speaker change is legitimate and where confirmation of understanding is expected.

In multi-turn prompt sequences, grounding failures are the primary source of compounding errors. If the model has misunderstood the task in turn 2 and the prompt in turn 3 doesn't check for this, subsequent turns silently build on a broken foundation. Structuring your multi-turn interactions with explicit grounding checkpoints — places where the model must demonstrate its current understanding before proceeding — makes failures observable rather than invisible.

Feedback Loops as Hypothesis Validation

Iterative software development is built on the principle that each development cycle is an engineering hypothesis tested against real-world constraints. Prompt engineering is no different. The prompt is the hypothesis. The evaluation is the test. The feedback loop is how you learn.

Continuous experimentation makes consequences the ultimate arbiter of good decisions — not theoretical reasoning about why a prompt should work. This is operationally significant: a good feedback loop design lets you deploy prompt variations and measure their effects without requiring manual review of every output.

---

Key Principles

1. Separate signal from noise before you optimize. The replication crisis in prompt engineering is a signal-to-noise problem. Any prompt comparison based on fewer than ~30 test cases with no statistical framing is noise until proven otherwise.

2. Variance is diagnostic. A prompt that produces high agreement across self-consistency samples is better specified than one that doesn't. Use variance as a quality signal, not just accuracy.

3. Evaluation must match what you actually care about. Mismatched evaluation metrics — measuring locutionary properties when you care about perlocutionary effects — is one of the most common ways prompt evaluation fails silently.

4. Grounding failures compound. In multi-turn sequences, an early misunderstanding that goes unchecked amplifies through every subsequent turn. Build grounding checkpoints into the sequence explicitly.

5. Short feedback cycles beat large batch retrospectives. Fast feedback loops allow rapid empirical evidence gathering and incremental correction. Waiting for a large accumulated sample before reviewing prompt performance means fixing problems late.

6. Diagnostic information requires multiple data sources. Effective root cause analysis integrates structured error data, execution traces, and contextual information. The same applies to prompt debugging: a single failed output tells you little; a structured trace of what the model received, what it reasoned, and where it diverged tells you a lot.

---

Step-by-Step Procedure

Minimum Viable Prompt Evaluation

This procedure applies when you are comparing two or more prompt candidates and need a result you can trust.

Step 1: Specify the perlocutionary target. Write down what the output must do to count as correct. This is not about format — it is about the actual effect the output needs to have. If you cannot write this down precisely, your evaluation will be invalid regardless of what else you do.

Step 2: Assemble a test set of at least 30 cases. The test cases should span the range of inputs you expect in production, not just the cases that were convenient to write. Skewed test sets produce skewed results.

Step 3: Run each prompt candidate with self-consistency sampling (N=10–20 samples per case). Record the majority vote answer and the inter-sample agreement rate for each case. The agreement rate is your signal quality indicator.

Step 4: Score using your perlocutionary target. Apply your Step 1 criteria to the majority-vote outputs. Use a binary pass/fail score unless you have a justified reason for a graded scale.

Step 5: Compare with statistical framing. A McNemar test or a simple bootstrapped confidence interval on the accuracy difference is enough. You are looking for: does the difference persist under resampling? If it does not, you have noise.

Step 6: Check the failure cases, not just the aggregate score. Examine the cases where one prompt succeeds and the other fails. Look for whether the failures share a structural property — if they do, that is a specification gap to address.

---

Designing Diagnostic Multi-Turn Sequences

Step 1: Map the information dependencies. List what the model needs to know at each turn to produce a correct output at the final turn. Draw a dependency graph if the chain is long.

Step 2: Insert explicit grounding checkpoints. At each turn where a critical understanding must hold, prompt the model to confirm its current state before proceeding. This can be as simple as asking it to summarize what it is doing and why.

Step 3: Instrument the output for diagnostics. Ask the model to emit structured information alongside its answer — its confidence level, the alternatives it considered, or the decision points it encountered. This is the prompt equivalent of structured error logs: it gives you causal chain visibility without needing to re-run the sequence to understand what happened.

Step 4: Design your feedback loop around the grounding checkpoints. Log the grounding checkpoint outputs separately from the final output. When the final output fails, the checkpoint logs tell you where in the chain the failure originated.

---

Worked Example

Scenario: You are evaluating two prompts for a code review assistant. Prompt A is your current production prompt. Prompt B adds an instruction to "think step by step before identifying issues." You ran both on 10 test cases and Prompt B got 8/10 right versus Prompt A's 6/10. Should you ship Prompt B?

Step 1: Is 10 cases enough? No. A difference of 2 correct responses on 10 cases is within the range of random noise. The confidence interval on a 20 percentage point difference with N=10 spans from roughly "no difference" to "B is much better." This result is not evidence of anything.

Step 2: Apply self-consistency to the 10 cases you have. Run each prompt 10 times per case. Look at the agreement rate. If Prompt B has higher agreement (e.g., 8–9/10 responses agree) while Prompt A has lower agreement (e.g., 5–6/10), that is a real signal — B is producing more stable reasoning even on the small sample. If agreement rates are similar, the 2-case difference is likely noise.

Step 3: Expand to 30+ cases before deciding. If the self-consistency signal looks promising, expand the test set. If the pattern holds at 30 cases and the confidence interval excludes zero, you have evidence. If it disappears, the original result was noise.

Step 4: Examine the failures. On the cases where Prompt A fails and Prompt B succeeds, what do they have in common? If they all involve multi-file context or ambiguous variable names, you have found a structural gap in Prompt A's specification — and now you can address it directly instead of hoping "step by step" continues to help.

---

Active Exercise

Task: Run a self-consistency evaluation on a prompt you use regularly.

1. Take a prompt you use in production or development that you believe works well.
2. Identify 15 representative inputs — ideally drawn from real usage.
3. Run each input through the prompt 8 times (N=8 samples per input). Vary the temperature if your inference setup allows it; otherwise accept whatever variance the model produces.
4. For each input, calculate the inter-sample agreement rate: how many of the 8 samples produce the same answer?
5. Find the 3 inputs with the lowest agreement rate. Read the disagreeing outputs carefully.

Questions to answer in writing:

• What do the low-agreement inputs have in common?
• Is the disagreement a sign of a genuine specification ambiguity in the prompt, or is it a sign that the task is inherently hard?
• Based on what you found: what is one change you would make to the prompt, and what prediction does it make about agreement rates?

This exercise is diagnostic, not evaluative. You are not trying to prove the prompt is good or bad — you are practicing reading variance as a signal.