Why "Let's Think Step by Step" Is Dead in 2026 (And What to Do Instead)

In 2022, Takeshi Kojima and colleagues published a paper with a sentence that became prompt engineering canon: append "Let's think step by step" to a question, and large language models suddenly got better at math, logic, and multi-hop reasoning. No examples. One phrase. GSM8K accuracy jumped from 10.4% to 40.7% on InstructGPT.

For three years, that phrase was everywhere. Blog posts, cheat sheets, system prompts, Twitter threads. It was the closest thing this field had to a universal upgrade button.

In 2026, blindly adding it is often a mistake. Not because the paper was wrong. Because the models changed underneath the trick.

What "let's think step by step" actually did

The original Zero-shot-CoT paper (Kojima et al., 2022) built on Wei et al.'s chain-of-thought work. The insight: LLMs can produce intermediate reasoning steps if you nudge them to. With few-shot CoT, you show example problems where the solution walks through steps before the answer. With zero-shot CoT, you skip the examples and add a trigger phrase instead.

"Let's think step by step" (and close variants like "Let's work this out step by step") activates a different generation mode. The model allocates tokens to reasoning before concluding. On GPT-3.5-era models that jumped straight to answers, this was genuinely transformative.

It worked because those models did not reason by default. You had to prompt reasoning into existence.

Why it breaks on reasoning models

Starting in 2024–2026, labs shipped models with dedicated internal reasoning: GPT-5.5 (and legacy o-series), Claude Opus 4.8 / Fable 5 with extended thinking, Gemini 3.1 Pro with thinking mode, DeepSeek R1. These models allocate a hidden reasoning budget before producing visible output. The thinking happens whether or not you ask for it.

Adding "let's think step by step" to a reasoning model does one or more of these:

1. Wastes tokens on visible narration the model was already doing internally.
2. Pushes the model to dump its reasoning into the answer instead of keeping it in the thinking channel.
3. Constrains the internal search with a visible step format copied from your prompt or few-shot examples.

OpenAI's guidance for GPT-5.5 and reasoning tiers is explicit: trust the model's inherent reasoning. Don't direct the reasoning process. Anthropic and Google echo the same pattern for Opus 4.8 / Fable 5 and Gemini 3.1 Pro thinking modes.

You're not unlocking hidden capability. You're interfering with architecture built specifically to reason without your scaffolding.

The overthinking problem

Recent research on large reasoning models adds another wrinkle. Models can reach the correct answer early in their reasoning trace, then keep thinking, revise, and land on a wrong final answer. Researchers call this harmful overthinking — distinct from verbose overthinking, where extra reasoning is redundant but harmless.

More reasoning is not monotonically better. Telling a model that already overthinks to "think step by step" in the visible prompt can make output longer without making it more correct. Sometimes it makes things worse.

This is the opposite of 2022, when the problem was too little reasoning, not too much.

When the phrase still works

"Dead" is strong. More accurate: dead as a default, alive as a targeted tool.

Still worth trying on:

• Standard chat models without extended thinking (Gemini 3.5 Flash, Claude Sonnet default mode)
• Hard multi-step tasks where the model jumps to wrong answers without intermediate steps
• Math, logic puzzles, debugging, constraint-satisfaction problems on non-reasoning tiers
• Situations where you want visible reasoning for human audit (show your work for a student, not for production parsing)

Skip it on:

• GPT-5.5 with reasoning effort enabled (and legacy o-series API configs)
• Claude Opus 4.8, Fable 5, or Mythos 5 with extended/adaptive thinking
• Gemini 3.1 Pro with thinking mode
• DeepSeek R1, Grok heavy reasoning modes
• Simple extraction, classification, or formatting tasks on any model
• Production pipelines where you parse structured output from the response

Quick test: run the same prompt with and without the phrase. If accuracy doesn't move or output bloats, drop it.

What to do instead (by model class)

Reasoning models: goal, constraints, format

Replace process instructions with outcome instructions.

Instead of: "Let's think step by step. What is the total cost?"

Write: "Calculate the total cost including 8% tax and $15 shipping. Return only the final dollar amount as a number."

Add API-level controls where available:

• GPT-5.5: reasoning_effort (low / medium / high / xhigh)
• Claude Opus 4.8 / Fable 5: thinking.budget_tokens or adaptive effort, separate from output max_tokens
• Gemini 3.1 Pro: thinking budget where supported
• Set completion token limits high enough for both hidden reasoning and visible answer (1024 is often too low)

If you need a brief justification in the output, ask for it explicitly after the answer: "Return JSON with keys total (number) and one_line_rationale (string, max 20 words)." Don't ask for full step-by-step narration.

Standard chat models: structured CoT when needed

When you do want visible reasoning on a non-reasoning model, structure it so reasoning and answer are separable.

Use XML tags or clear headers:

<reasoning>Brief internal work here</reasoning>

<answer>Final answer only</answer>

Or: "First analyze in a <scratchpad> block. Then give the final answer in an <answer> block. Production code will parse only <answer>."

This is evolved CoT — not the magic phrase tacked onto the end, but explicit separation so you can extract what you need.

Any model: split into a chain

For complex tasks, don't ask one prompt to think and answer. Chain two calls:

1. "List the constraints and unknowns in this problem. Do not solve yet."
2. "Given this analysis: [paste output], produce the final recommendation in 3 bullets."

Each step is optimizable. Errors are catchable between steps. This replaces zero-shot CoT's single-prompt hack with something that scales.

The few-shot trap on reasoning models

Zero-shot CoT wasn't the only casualty. Few-shot examples that show "Step 1... Step 2... Answer:" can lock reasoning models into mimicking visible chains from your examples instead of using internal search.

Research revisiting CoT on strong reasoning models found traditional few-shot CoT exemplars often don't beat zero-shot — the model ignores low-signal examples and responds to the task description anyway, or worse, copies the example format into the output.

On reasoning models: zero-shot or one clean example. On chat models: two to three examples if format compliance matters.

Common mistakes people still make

Copy-pasting 2022 prompts into GPT-5.5 without changing anything.

Adding "think step by step" AND extended thinking AND few-shot CoT examples — triple scaffolding, triple noise.

Parsing full CoT narration in production when you only need the final field.

Using reasoning models for high-volume extraction because "it's smarter" — it's slower, costlier, and the extra reasoning buys nothing on a label classification task.

Assuming visible reasoning equals correct reasoning. Long chains can still end wrong.

Before / after examples

Math word problem on Gemini 3.5 Flash (standard chat — CoT still helps)

Before (weak): "A store sells 3 apples for $2 and 5 oranges for $4. Maria buys 6 apples and 10 oranges. Total cost?"

After: same question + "Work through the calculation step by step, then state the final total on the last line."

Reasonable on a non-reasoning model. Visible steps help catch arithmetic errors.

Same problem on GPT-5.5 (reasoning model — drop CoT)

Before (2022 habit): "Let's think step by step. A store sells 3 apples for $2..."

After: "Maria buys 6 apples at 3-for-$2 pricing and 10 oranges at 5-for-$4 pricing. What is the total cost in dollars? Reply with the number only."

Trust internal reasoning. Constrain output shape.

Debugging on Claude Opus 4.8 with extended thinking

Before: "You are an expert debugger. Think step by step through this stack trace..."

After: "This Node.js app throws ECONNREFUSED on Redis connect at startup. Stack trace below. Identify the most likely cause and the single config change to fix it. One paragraph max."

Role is optional. Process instruction removed. Success criteria explicit.

How this connects to the rest of prompt engineering in 2026

The death of default CoT is one symptom of a larger split: chat models vs reasoning models need different playbooks. Same for few-shot (sometimes harmful), role prompts (sometimes fluff), and JSON-in-prompt (replaced by API schemas).

Our prompt engineering best practices guide on this blog covers the full decision tree. This article is the deep cut on one phrase that aged badly.

Frequently asked questions

Should I ever write "let's think step by step" in 2026?

Yes, on standard chat models for hard reasoning tasks where you want visible intermediate steps. No, as a default on reasoning models or simple tasks.

Did the Kojima paper become wrong?

No. The models changed. Zero-shot CoT was correct for 2022-era LLMs. Reasoning models have different inference architecture.

What about "show your work" for education?

Fine for tutoring use cases on chat models where visible reasoning is the product. Separate from production pipelines where you need parseable output.

Does Claude Opus 4.8 / Fable 5 extended thinking replace CoT prompting?

Functionally yes for analytical tasks. Enable thinking, write a clear task spec, don't duplicate with CoT phrases.

What's the one-line replacement?

On reasoning models: state the goal and output format. On chat models: structured <reasoning> / <answer> separation if you need visible steps.

Can PromptMake help if I'm bad at writing these prompts?

promptmake.net/text builds model-specific structure from a rough idea — role, task, format calibrated to your target model. It won't pick reasoning vs chat model for you, but it saves blank-page friction once you know which class you're using.

Bottom line

"Let's think step by step" was a genuine breakthrough on models that didn't reason by default. Reasoning models reason by default. The phrase didn't become false. It became redundant — and on frontier models, actively unhelpful.

Update your templates. Match the prompt to the architecture. And if you're still append-only upgrading every prompt with a 2022 trigger phrase, you're optimizing for a model that doesn't exist anymore.