# Prompting Practices That Survived Real Testing

> Most prompting advice is untested. Three practices held up: position-swap judging, de-anchoring a worked example, and dropping temperature.

Canonical: https://brandonlazovic.dev/articles/prompting-practices-that-survived/  
Author: Brandon Lazovic  
Published: 2026-08-20

## The short version

- Most prompting advice never meets a test that could disprove it. Three practices here earned their place because each carried a measurement that could have failed.
- A 2026 study of five LLM judges found style bias reaching 0.76 to 0.92, while position bias, the one most bias checks test for, sat at 0.04 or below.
- On a production check of 2,988 blinded paired judgements, a cheap model's self-agreement across a position swap fell to as low as 5.6% against a chance floor near 33%; a larger judge model raised it to 70.5% through 88.0% on the same pairs.
- OpenAI's current reasoning models reject the temperature parameter outright. Determinism now comes from a schema the API guarantees will match, an explicit reasoning-effort setting, and a prompt that lists every valid value.

Most prompting advice never meets a test that could kill it. Three practices survived one anyway inside a real pipeline: judging an AI output with the answer order swapped, replacing a prompt's worked example with a neutral one, and giving up the temperature setting a current reasoning model refuses to accept. Because each one carried a real measurement rather than a hunch, the result could have gone the other way. One did, partway; a cheap model graded thousands of paired outputs and turned out to be reading which side an answer sat on rather than what it said, a result that also points at a second, larger bias most practitioners never check for at all. What follows is the check behind each practice.

> A 2026 study of five LLM judges found style bias reaching 0.76 to 0.92, while position bias, the one most bias checks test for, sat at 0.04 or below.

## Does swapping the order of two answers change what an AI judge decides?

Swapping the order changes the verdict often enough to make it meaningless. Given two candidate outputs, a judge model should reach the same decision no matter which slot either one lands in. Run every pair through the judge twice, once in each order, and check whether the pick flips when nothing about the content changed. A pick that flips means the judge graded a slot, not an answer.

I ran that exact check inside a production pipeline on 2,988 blinded paired judgements, scored against several grading criteria with a three-way verdict of win, lose, or tie. The pipeline is personal and Claude-tooling work, not a client engagement, so none of these figures touch client data. Across the swap, chance agreement sits near 33% for that shape of question.

The cheap model doing the judging, `gpt-5.4-nano`, OpenAI's smallest and cheapest tier, landed between 5.6% and 27.7% self-agreement depending on the criterion; it also picked the same letter, A or B, on 72% to 98% of pairs regardless of which candidate actually held that slot. That range hides how much the individual criteria disagreed with each other, some barely above a coin flip and others closer to random three-way guessing.

Switching the judge to `gpt-5.6-terra`, the larger tier used here for judging rather than generation, raised self-agreement to between 70.5% and 88.0% on the identical pairs, and that gap decided which model grades the pipeline's output today, the same kind of measured tradeoff that sets the [sub-agent delegation cost curve](/articles/sub-agent-delegation-context-cost/) elsewhere in this pipeline.

To rule out an artifact of how the two answer labels were listed, rather than a genuine slot effect, three different orderings of the same output schema were tested first. The bias held across all three. That points at the model's reading of position rather than at how the schema happened to be written.

## Does fixing position bias fix an AI judge?

Fixing position bias leaves an AI judge only partly fixed, because a second, larger bias runs underneath it. A 2026 evaluation of five judge models from four separate providers, using a controlled 225-pair test with content held identical across formatting styles, found style bias reaching 0.76 to 0.92, a strong preference for markdown-formatted answers over plain prose. [1]

On the same models, position bias sat at 0.04 or below. [1] The bias most checklists test for turns out to be the smaller one here, by a wide margin.

Even the paper's own position-swap runs only cut style bias by roughly 0.12 on average, a side effect of correcting for order rather than a targeted fix for style, which leaves most of the 0.76-to-0.92 gap standing. [1] Swap position and stop there, and a practitioner has closed the bias worth 0.04 while leaving one worth roughly twenty times that untouched.

My own read here, offered as inference rather than the paper's stated conclusion, is that a position swap catches an effect a reader can literally see by reversing two candidates on a page. Style bias hides inside content that looks unrelated to the verdict, which is exactly why a check built for order never touches it. Before trusting a judge's reasoning about which candidate won, strip formatting from both, or hold it constant.

![Bar chart comparing two measured biases in the same set of LLM judges. Position bias, the one most bias checklists test for, sits at a ceiling of 0.04 or below, shown as a short solid segment. Style bias, a strong preference for markdown-formatted answers over plain prose, ranges from 0.76 to 0.92 in the same 2026 study, shown as the much longer hatched remainder scaled to the range's own low end. A callout below states that even that low end runs roughly twenty times position bias's ceiling.](/figures/prompting-practices-that-survived/bias-comparison.png)

*Soumik, 2026 (arXiv 2604.23178); five judge models from four providers, 225 pairs, content held identical across formatting styles.*

## Does a worked example in a prompt bias the output toward itself?

For how an example is placed and framed, the published evidence says yes, though not under the name this piece gives the check. Few-shot prompting means giving the model a handful of worked examples before the real question. Permuting the order of those examples, with the same examples and the same task, can swing accuracy from near chance to near state-of-the-art on GPT-3-era models. [2] A good ordering for one model does not transfer to another. [2]

A separate study found the model's answer skews toward whatever example sits nearest the end of the prompt, or toward whatever answer was common in the model's training data, independent of which example was actually correct for the task at hand. [3]

Neither paper studied the specific move this piece is naming a de-anchoring check: swapping a prompt's one worked example for a neutral substitute and rerunning to see whether the output pattern survives. If the pattern holds with a different example in place, the pattern belongs to the data. If it disappears, it belonged to the example.

Zhao and colleagues built a fuller correction for a related failure: feed the model a content-free input, the literal string "N/A", read off which answer it defaults to with no real content at all, then use that default to recalibrate its predictions. [3] It's like asking a witness what they would say about a case they have never actually seen, to find out which answer they favor by default before trusting anything they say about a real one.

Where the fix stops being a rule of thumb is the point the analogy breaks: the paper turns the content-free answer into calibration weights instead of a courtroom judgment call. Their correction improved accuracy by up to 30 percentage points on some tasks, a large enough number to take the underlying bias seriously well outside the classification tasks they tested. [3]

A de-anchoring check is the cheaper, rougher cousin of that fix. It flags whether a bias exists; building the fuller correction described above is a separate, bigger project.

## How do you get deterministic output from a model that rejects temperature?

Deterministic output on OpenAI's current reasoning models comes from the output contract instead of the sampler. In my own pipeline runs, passing any temperature value besides the model's default returns a 400 error every time. That is a first-hand observation rather than a documented claim, since neither OpenAI's API reference nor its reasoning guide mentions temperature in connection with these models.

Determinism instead comes from three things stacked together. One is an output schema the API guarantees will match. Another is an explicit reasoning-effort setting rather than the model's own default. And the third is a prompt that lists every legal value instead of leaving the model to infer them.

OpenAI's own reference for the chat completion endpoint lists `reasoning_effort` as a flat, optional field rather than a nested object, and as of this week it names seven allowed values: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`. [4] That count matters here. Before writing this, I checked a pipeline-standards note that named five of those seven and flagged `max` as unusable outside a different API surface. It was accurate when it was written.

Today it sits one revision behind the vendor's own page, which is the exact failure this whole piece argues against: a hazard list is a snapshot, and the fix is to check it against the live source again instead of trusting the last one taken of it.

The same reference deprecates the older `max_tokens` field in favor of `max_completion_tokens`, which now counts a reasoning model's internal reasoning tokens against the cap as well as what it prints. [4] On the schema side, OpenAI describes Structured Outputs as reliably matching a model's response to a developer-supplied JSON Schema, strict enough that a parsed response is "guaranteed to match the schema you requested." [5]

Pair that guarantee with a prompt that spells out a field's full set of legal values, and the output stops drifting between runs even though the setting that used to enforce that is no longer there to pass.

None of this costs materially more than the old approach. The three tiers of the current model family run $0.20 to $5.00 per million input tokens depending on which one a job needs, a figure confirmed against the live pricing page the same day this was written rather than carried over from an older note. [6]

## What should you actually run before trusting an AI judge?

Run the position swap this afternoon on whatever prompt already judges your pipeline's output. Pick ten to twenty paired candidates, judge each pair twice, once in each order, and record whether the verdict changes between the two runs.

Divide the number of pairs that agreed by the total tested, and compare that rate to the chance floor for your verdict schema: roughly 33% for a three-way win, lose, or tie answer, and roughly 50% for a straight either/or.

A self-agreement rate within about ten points of that chance floor means the judge is reading which side an answer landed on rather than what the answer said, and every verdict it has produced up to that point is noise rather than signal. A rate in the 70s or higher, closer to what `gpt-5.6-terra` showed on the pairs described above, is worth trusting enough to act on.

Even then, a passing position-swap score is a floor rather than a clearance, because the style-bias finding above never runs through this check at all. Keep formatting constant across both candidates on the next pass. That is the bias a clean position-swap score will not show you.

## Sources

1. Soumik, Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines (arXiv 2604.23178) — https://arxiv.org/html/2604.23178v1
2. Lu et al., Fantastically Ordered Prompts and Where to Find Them (arXiv 2104.08786) — https://arxiv.org/abs/2104.08786
3. Zhao et al., Calibrate Before Use: Improving Few-Shot Performance of Language Models (arXiv 2102.09690) — https://arxiv.org/abs/2102.09690
4. OpenAI API reference: Create chat completion — https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create
5. OpenAI: Structured Outputs guide — https://developers.openai.com/api/docs/guides/structured-outputs
6. OpenAI: API pricing — https://developers.openai.com/api/docs/pricing