Chain-of-Thought Prompting: When to Use It and When to Skip It

Chain-of-thought prompting is the technique where you make a language model show intermediate reasoning before the final answer. Wei et al. (2022) proved it with few-shot examples. Kojima et al. same year showed a single trigger phrase works zero-shot. Wang et al. (2023) added self-consistency: sample multiple chains, majority-vote the answer.

Three years of blog posts treated CoT like vitamins — add to everything. That's wrong. CoT helps when a task decomposes into steps. It hurts or wastes money when the task doesn't. In 2026, reasoning models add another layer: they already chain-of-thought internally, so external CoT often duplicates or interferes.

This is the decision guide: which CoT variant to use, when to skip entirely, and how to not pay 5x tokens for zero accuracy gain.

The one question that decides everything

Before picking a CoT technique, ask: does solving this require intermediate steps a human would write down?

Yes → CoT might help (on the right model class).

No → skip CoT. Classification, lookup, translation, simple extraction, creative brainstorming — adding reasoning steps increases cost and verbosity without improving accuracy.

Examples where CoT helps: math word problems, multi-condition logic, debugging stack traces, comparing three vendor quotes with different fee structures, legal clause analysis with exceptions.

Examples where CoT hurts or wastes tokens: "Is this email spam?", "Summarize in 3 bullets", "Extract the invoice date", "Write a tagline for a coffee shop".

If you can't name the intermediate steps a competent human would take, CoT is probably theater.

Three CoT variants (and what each costs)

Zero-shot CoT

Add a trigger phrase — "Let's think step by step" or "Work through this before answering" — to the end of the prompt. No examples required.

Cost: roughly 1.5–3x tokens vs direct answering because the model generates a longer response.

Best for: quick experiments on chat models, hard reasoning tasks where you don't have curated examples yet.

2026 caveat: skip on reasoning models (GPT-5.5, Opus 4.8 / Fable 5, Gemini 3.1 Pro). See our article on why "think step by step" died for the full explanation.

Few-shot CoT

Provide 2–5 complete examples where each shows question → reasoning steps → final answer. The model pattern-matches the reasoning structure.

Cost: 3–5x tokens (examples in every request) plus longer outputs.

Best for: domain-specific reasoning with a consistent format — medical triage logic, internal policy interpretation, financial calculations with company-specific rules.

Few-shot CoT teaches format and domain patterns. It does not teach reasoning to a model that can't reason. Bad examples propagate bad habits.

Self-consistency

Run the same CoT prompt N times (temperature 0.5–0.8 so paths differ), extract each final answer, take the majority vote.

Cost: N× whatever CoT already costs. N=3 is a common sweet spot; N=40 is research-benchmark territory, not production default.

Best for: high-stakes, low-volume decisions — medical dosage checks, financial compliance flags, critical infrastructure config validation.

Skip for: high-throughput pipelines, creative writing, anything where latency matters more than the last 5% of accuracy.

When to use CoT (decision table in prose)

Use zero-shot CoT when:

• You're on a standard chat model (not GPT-5.5 reasoning / Opus 4.8 thinking / Gemini 3.1 Pro thinking)
• The task is multi-step and the model jumps to wrong answers without scaffolding
• You're prototyping and don't have labeled examples yet
• You want visible reasoning for human review (tutoring, audit trails)

Upgrade to few-shot CoT when:

• Zero-shot produces correct answers but messy or inconsistent format
• Domain jargon or company-specific rules appear in every example
• You need the model to follow a mandated reasoning template (compliance, support tiers)

Add self-consistency when:

• A single wrong answer is expensive
• You can afford 3–5x latency and cost per query
• The task has a discrete verifiable answer (number, class label, yes/no)

Skip CoT entirely when:

• The task is single-step retrieval or classification
• You're on a reasoning model that thinks internally
• Output must be short and structured (use API schema instead of reasoning narration)
• Creative or open-ended generation is the goal
• You already hit accuracy targets with direct prompting — don't fix what works

Chat models vs reasoning models

This split matters more than which CoT variant you pick.

Standard chat models (Gemini 3.5 Flash, Claude Sonnet without thinking): CoT is still a legitimate tool. Zero-shot for quick wins. Few-shot when format matters. Self-consistency when stakes are high.

Reasoning models (GPT-5.5, Claude Opus 4.8 / Fable 5, Gemini 3.1 Pro, DeepSeek R1): external CoT is usually wrong tool. The model allocates hidden thinking tokens. Your job is clear task spec, constraints, output format, and enough max_tokens for both thinking and answer.

Mixing both is a common mistake: reasoning model + "think step by step" + three few-shot CoT examples. Triple scaffolding. Often worse than a five-line direct prompt.

Production pattern in 2026: route simple queries to fast chat models (no CoT), hard analytical queries to reasoning models (no CoT phrase), medium complexity to chat models with structured CoT if benchmarks show a gain.

Structured CoT beats magic phrases

Bare "think step by step" is the 2022 version. The 2026 version on chat models separates reasoning from answer explicitly:

<analysis>Work through constraints and calculations here.</analysis>

<answer>Final answer only — one number / one label / one JSON object.</answer>

Benefits:

• Production code parses <answer> without regex surgery on freeform narration
• Humans can read <analysis> when debugging failures
• The model knows where to stop reasoning and start delivering

Claude responds reliably to XML-style tags. Other models handle markdown headers or labeled sections similarly. Pick one delimiter and stick with it across your pipeline.

When CoT makes accuracy worse

CoT is not monotonically good. Documented failure modes:

Simple tasks: CoT on "What is the capital of France?" adds tokens, adds nothing.

Reasoning models: external CoT can push internal reasoning into visible output, causing narration bloat and occasional answer drift.

Harmful overthinking: on reasoning models, more computation after reaching the correct intermediate result can overwrite it with a wrong final answer. Adding CoT phrases doesn't fix that; it can extend the chain.

Bad few-shot examples: one incorrect step in an example teaches the wrong pattern. Curate examples like unit tests.

Format-heavy tasks: if you need strict JSON, CoT narration fights schema-constrained decoding. Use API structured output, not visible chains.

Cost vs accuracy: be honest about the tradeoff

Benchmark posts claim +20–70% accuracy from CoT variants. Real production numbers depend on your task, model, and eval set. Order of magnitude:

• Zero-shot CoT: modest gain, low setup cost
• Few-shot CoT: bigger gain on domain tasks, ongoing maintenance cost (examples drift)
• Self-consistency N=3: meaningful reliability bump, 3x bill

Run your own eval before shipping. Ten inputs with pass/fail criteria beats a blog post benchmark.

Also consider: CoT on a cheap fast model sometimes beats direct prompting on an expensive model without CoT. Cost optimization and accuracy optimization are the same spreadsheet.

Worked example: support ticket routing

Task: classify support tickets into Billing / Technical / Account and extract urgency (low/medium/high).

Wrong approach: "Let's think step by step about the user's emotional state and..." — overkill, slow, inconsistent labels.

Right approach: direct prompt with 3 few-shot examples showing ticket → label + urgency. No CoT. Pattern matching task, not multi-hop reasoning.

Contrast: task is "Given our SLA table and this ticket history, determine if we're liable for a credit and calculate the amount."

Right approach: few-shot CoT with worked examples showing policy lookup → liability check → calculation → answer. Or route to a reasoning model with a direct spec and structured JSON output.

Same product. Different task shape. Different technique.

CoT and prompt chains (when one prompt isn't enough)

Sometimes the move isn't longer CoT — it's splitting steps across calls.

Chain pattern:

1. Extract facts from messy input (no CoT, just structured extraction)
2. Apply rules to extracted facts (CoT or reasoning model here)
3. Format output for downstream system (schema-constrained, no CoT)

Each step gets the technique it deserves. One mega-CoT prompt trying to extract, reason, and format simultaneously is how instructions get lost in the middle.

Quick reference checklist

Before adding CoT to a prompt, check:

1. Is this a multi-step reasoning task? If no, stop.
2. Am I on a reasoning model? If yes, skip CoT phrases.
3. Do I need visible reasoning or just a correct answer? If just answer, use structured output or hidden thinking.
4. Do I have clean few-shot examples? If no, start zero-shot, don't invent bad examples.
5. Is accuracy worth N× cost? If no, skip self-consistency.
6. Did I measure baseline without CoT? If no, you're guessing.

Frequently asked questions

What's the difference between CoT and few-shot prompting?

Few-shot shows input/output examples — often for format and style. Few-shot CoT specifically shows reasoning steps in the examples. Plain few-shot doesn't require intermediate steps.

Is zero-shot CoT still worth it in 2026?

On standard chat models for hard reasoning tasks, yes. On reasoning models, no. Test both ways on your eval set.

How many few-shot CoT examples do I need?

Two to three strong examples usually suffice. More than five increases cost and lost-in-the-middle risk without proportional gain.

What's the best self-consistency N?

N=3 for production sweet spot. N=5–10 for higher stakes. Beyond that, diminishing returns unless benchmarks on your data prove otherwise.

Should I show CoT reasoning to end users?

Only if transparency is a product requirement. Otherwise parse the final answer and log reasoning internally for debugging.

How does this relate to tree-of-thought?

Tree-of-thought explores multiple reasoning branches and backtracks — powerful for puzzles and planning, 10–30x cost. CoT is a single chain. Use ToT rarely, when search-like exploration is genuinely needed.

Can PromptMake replace manual CoT structuring?

Partially. promptmake.net/text builds role, task, and format structure for your target model from a rough idea. It won't run self-consistency or pick your routing logic — but it saves time on the baseline prompt before you add CoT if benchmarks say you need it.

Related articles in this series

Prompt engineering best practices in 2026 — full rule set and decision tree.

Why "let's think step by step" is dead — deep dive on zero-shot CoT vs reasoning models.

See also: positive framing, lost-in-the-middle, few-shot vs zero-shot, structured output, role prompting, RTF framework, and prompting GPT-5.5 / Opus 4.8 / Gemini 3.1 Pro.

Bottom line

Chain-of-thought is a scalpel, not a supplement. Use it when intermediate steps exist and the model class benefits from visible or example-driven scaffolding. Skip it when the task is simple, when the model already reasons internally, or when structured output matters more than narration.

Start direct. Add CoT only when evals prove it. Escalate from zero-shot to few-shot to self-consistency in that order — not all at once on every prompt because a 2022 paper had good GSM8K numbers.