Self-Consistency Prompting: Vote Across Samples

Self consistency prompting means you run the same hard question several times with enough randomness for different reasoning paths, pull each final answer, then take the majority vote. Wang et al. (2023) showed that correct chains often agree while wrong chains scatter. You leave with a plain vote loop, rules for sample count (N), how to extract answers without messy regex, a cost check before you multiply spend, and a clear split: when the vote lifts accuracy versus when it burns tokens for no gain. Soft tool note: PromptMake /text helps you lock a clean baseline prompt before you wrap it in a sampler.

What self consistency prompting is

A single chain-of-thought reply is one story the model tells about how it got to an answer. That story can be right. It can also sound careful and still land on the wrong label or number. Self consistency prompting treats the answer as something you measure across samples, not something you trust from one fluent paragraph. You keep the prompt fixed, raise temperature so paths diverge, run N completions, normalize each final answer into a comparable key, then pick the key that appears most often.

The technique fits tasks with a discrete, checkable outcome: a class label, a yes/no flag, a short ID, a multiple-choice letter, a numeric result with a fixed unit. Open-ended drafts, brand voice rewrites, and brainstorm lists fight the vote because "majority" has no honest meaning when every sample is a different essay. Teams that gain the most run low-volume, high-cost-of-error checks: compliance flags, dosage math with a verified formula, config validation, quiz-style knowledge gates inside an app.

Skip self consistency prompting when latency budgets are tight, when the answer is free-form, and when a reasoning-class model already clears your eval with one call. A vote multiplies prompt tokens and completion tokens by N. You pay for agreement, so agreement has to matter.

How the vote across samples works

Four pieces make the loop: a fixed prompt that asks for visible reasoning plus a parseable final answer, a sampler that returns N completions at a non-zero temperature, an extractor that maps each completion to a canonical answer key, and a voter that picks the mode (or a weighted mode if you score confidence). Fail the loop early if you cannot write the extractor in ten lines. If humans disagree on how to read the "answer," code will disagree too.

Keep reasoning and answer in separate regions so extraction stays boring. XML tags, markdown headers, or a last-line "ANSWER: …" convention all work. Pick one delimiter and freeze it across samples. Temperature in the 0.5-0.8 band is the usual research range for path diversity on chat models. Temperature 0 collapses samples toward the same chain, which defeats the point. Temperature near 1.0 can scatter answers so hard that you never get a majority and you still paid N times.

N is a budget knob, not a virtue. N=3 catches many unstable failures. N=5-7 is a common production ceiling when error cost is high. N=20-40 belongs to paper benchmarks and rare offline audits. Log every sample's raw answer key, the vote, and whether a human later agreed. Without that log, you will raise N forever and never prove the spend.

Extract the answer before you vote

Write the prompt so the model ends with a machine-readable field. Example contract: "Put intermediate work under <analysis>. Put the final choice alone under <answer> as one of: approve, reject, escalate." Your extractor reads only <answer>, lowercases it, and maps aliases ("APPROVED" → approve). Reject samples that miss the tag or invent a fourth label. Those samples are invalid votes, not creative flexibility.

For numbers, pin units and rounding in the prompt: "Answer in USD whole dollars, no commas, no currency symbol." Normalize "1200", "$1,200", and "1200.00" to the same key before you count. For multiple choice, force a single letter. For boolean gates, force true/false strings. Soft free text inside the answer field breaks majority counting; move explanation to the analysis block.

Count votes and handle ties

Simple majority works when one key clearly leads. On a tie, pick a policy before production: re-sample once with a new seed, fall back to a reasoning-class model for a single tie-break call, or route to a human. Do not invent a silent preference for the first sample; that reintroduces single-path bias under a voting costume.

Optional: weight samples by a self-reported confidence score only if you calibrated that score on your set. Uncalibrated "confidence: high" lines add noise. Prefer hard agreement on the answer key. If you need ranked candidates for a UI, return the top two keys with counts so a reviewer sees the spread.

A step-by-step self consistency prompting workflow

You can prove the method on a spreadsheet before you write orchestration code. Pick one recurring task with labeled answers. Score a single greedy (temperature 0 or low) CoT baseline. Then run the same prompt at N=3 and N=5 with temperature ~0.7. Compare accuracy and mean cost per correct answer. Ship the vote only when holdout accuracy rises enough to justify the multiplier.

Draft the baseline instruction once. Role fluff and long pep talks do little for a vote. Goal, constraints, allowed labels, and the answer delimiter do the work. If that baseline is still a messy paragraph, harden it in PromptMake /text for the model you will sample, then freeze the string in version control. The sampler should change temperature and seed, not the prompt body, between samples in one vote.

Cap retries. A sample that fails schema is a miss, not a reason to loop forever. Set a maximum of one repair pass that restates the delimiter rules. If repair still fails, mark the sample invalid and continue. Your voter should require a minimum valid count (for example 2 of 3) before it trusts a majority; otherwise fail closed or escalate.

Steps 1-3: Contract, baseline, and sample budget

1. Write the success rule in one line: "Label must match gold; answer field must be one of the allowed set."
2. Score a single-sample CoT or direct prompt on 30-50 holdout cases. Record accuracy and median tokens.
3. Choose N and temperature. Start with N=3 and temperature 0.7 on a chat model. Note the projected cost as N × median tokens × price.

Stop if the single-sample baseline already meets your accuracy bar. Self consistency prompting is an escalation, not a default wrapper for every call.

Steps 4-6: Sample, vote, gate

1. Run N completions with the frozen prompt. Store raw text plus extracted keys.
2. Drop invalid extractions. Majority-vote the rest. Log ties.
3. Score the voted answers on the same holdout. Ship only if accuracy rises and cost-per-correct stays inside budget. Keep the single-sample path behind a flag for a week of shadow traffic.

After a win, version the prompt, N, temperature, and extractor together. Changing the delimiter without changing the extractor silently empties your vote bucket. Treat that bundle like a model artifact.

When self consistency prompting helps vs wastes tokens

The technique earns its keep when wrong answers are expensive, when a single chain flips between a few discrete options, and when you can measure lift on a labeled set. Classic wins look like math word problems with one numeric answer, multi-condition policy checks with a small label set, and brittle classification where temperature-0 CoT oscillates across near-miss labels. Agreement across independent paths is evidence the answer is stable, not that the model "understood" in a human sense. You still need evals.

It wastes tokens when the task is creative or open-ended, when answers cannot be normalized to a shared key, when you already sit on a reasoning-class model that clears the task in one shot, and when N grows because someone read a paper and copied N=40 into a hot path. High-throughput support macros, short rewrites, and JSON field fills with schema-constrained decoding rarely need a vote. Pay for structure and validation instead.

Use this prose gate before you multiply spend:

• Discrete answer you can canonicalize → candidate for a vote
• Error cost high and volume low → strong candidate
• Single-sample eval already green → skip the vote
• Latency SLA under a few seconds on a slow model → skip or shrink N
• Free-form deliverable (email, blog, brainstorm) → skip; majority is meaningless

Cost math stays simple. If one call costs C and accuracy is A0, N samples cost about N×C. You need accuracy A1 such that the business value of (A1−A0) beats (N−1)×C per query. If you cannot name that value, you are guessing. Run the holdout first; raise N only when A1 still climbs.

Common mistakes

Teams sample at temperature 0 and call it self consistency prompting. Paths match, the vote is a rubber stamp, and you still paid N times. Raise temperature enough for disagreement, then let agreement mean something.

Teams vote on full paragraphs. Two samples can share the same decision and still look different in prose. Extract a key. Vote on keys.

Teams wrap every production prompt in N=5 because a benchmark table looked good. Benchmarks optimize for accuracy under research budgets. Your product optimizes for accuracy under a bill and a latency bar. Measure both.

Teams mix self consistency prompting with tree-of-thoughts mid-path search. Those are different shapes. Self-consistency samples full chains and votes finals. Tree of thoughts proposes, scores, and backtracks on partial states. Pick one search story per pipeline stage or you will not know which spend helped.

Teams add "think step by step" on top of a reasoning-class model, then multiply samples. Hidden thinking plus external CoT plus N votes is triple scaffolding. Prefer a short goal-plus-constraints prompt on GPT-5.6 Sol, Claude Opus 5 / Fable 5, or Gemini 3.1 Pro for hard analysis, and reserve multi-sample votes for chat models where a single path still wobbles on your set.

Teams ignore invalid extractions and let empty strings win a plurality. Count only valid keys. Fail closed when valid samples fall below your floor.

Model notes for 2026 and where PromptMake fits

As of mid-2026, chat-tier models remain the natural home for classic self consistency prompting: Claude Sonnet 5, GPT-5.6 Terra or Luna, Gemini 3.5 Flash, and similar fast paths. You control temperature, you see the chain, and N=3 stays affordable for gated checks. Reasoning-class options (GPT-5.6 Sol, Claude Opus 5 or Fable 5, Gemini 3.1 Pro, DeepSeek reasoning tiers) already spend tokens on internal search. Start with one clear call and structured output. Add a multi-sample vote only if your eval shows residual instability that a second independent sample set fixes.

API pattern: same messages array, N create calls (or n>1 where the provider returns multiple choices in one request), shared system prompt, temperature > 0, max tokens large enough for analysis plus answer. Prefer provider-native multiple-choice responses when billing and latency favor one round trip. Still extract and vote in your code; do not trust a vendor "best of" without your canonical key.

PromptMake /text is the soft step before the sampler. Paste a rough task brief, pick the model that will run the samples, and take a structured role-task-format prompt with a hard answer delimiter. Then stop chatting the production string. Put it under version control, wire the voter, and prove lift on holdout. The tool drafts the baseline; the vote proves the baseline under noise.

FAQ

What is self consistency prompting in plain terms?

Self consistency prompting is a vote across samples. You ask the same question several times so the model takes different reasoning paths, then you pick the answer that shows up most often. The method assumes correct paths cluster and incorrect paths diverge. You still need a discrete answer key and a way to extract it from each sample.

How is self consistency prompting different from chain-of-thought?

Chain-of-thought asks for intermediate steps inside one reply. Self consistency prompting runs several of those replies (or several direct samples) and majority-votes the finals. CoT improves a single path. The vote checks whether independent paths agree. Many teams combine them: CoT inside each sample, then a vote across samples.

How many samples (N) should I use?

Start with N=3 on a labeled holdout. Move to N=5 if accuracy still rises and cost stays inside budget. Large N values from research papers fit offline audits, not default hot paths. Choose N from measured lift and dollar cost per query, not from a table in a 2023 paper alone.

Does self consistency prompting work on reasoning models?

Sometimes, but it is a weak default. Reasoning models already allocate hidden thinking tokens. Multiply that by N and the bill grows fast. Prefer a sharp single call with goal, constraints, and format on GPT-5.6 Sol, Claude Opus 5 / Fable 5, or Gemini 3.1 Pro. Add a vote only when your eval shows unstable discrete answers that multiple independent samples stabilize.

When does self consistency prompting waste tokens?

It wastes tokens on open-ended writing, on answers you cannot canonicalize, on tasks a single sample already solves, and on high-QPS paths where N× latency breaks the SLA. It also wastes tokens when temperature is too low for diversity or when you vote on unparsed prose. Fix extraction and measure lift before you raise N.

How do I extract answers for a reliable vote?

Force a tiny final field: a label, a letter, a number with pinned units, or a boolean string. Keep reasoning in a separate block. Normalize case, punctuation, and aliases before you count, and drop samples that miss the field. Vote on the normalized key only; that discipline matters more than fancy aggregators.

How do I start with a free workflow this week?

Pick one discrete task with 30 labeled examples. Write a prompt with a clear <answer> field, or tighten that draft once in PromptMake /text on the free tier. Score one sample, then score a majority of three at temperature about 0.7. Keep the vote only if holdout accuracy rises enough to pay for the extra calls.