Show your work
Picture a math teacher who tells a student: "Don't just give me the answer — show your work." Forced to write out each step, the student slows down, catches their own mistakes, and is far more likely to land on the right answer. If they do get it wrong, the teacher can point to the exact line where the logic broke.
That's essentially what "Chain-of-Thought" (CoT) prompting does for AI language models — the systems behind chatbots like ChatGPT. Normally, if you ask one of these models a multi-step question, it tries to jump straight to a final answer, and on anything involving several steps of arithmetic or logic, it often stumbles. Chain-of-Thought prompting simply asks the model to "think out loud" first — to write out its reasoning step by step before giving a final answer.
This isn't a change to the model itself. No retraining, no new code, no extra computing hardware. It's just a different way of asking the question. Yet when Google researchers introduced this idea in 2022, the results were startling: on a set of grade-school math problems, one of Google's large models jumped from getting only about 18% of answers right to getting 58% right, just by being told to reason step by step instead of answering directly. It turned out that inside these very large models, real reasoning ability was already there — it just needed the right kind of prompt to switch it on.
How it works
The magic of Chain-of-Thought lies not in changing the model, but in changing how we ask questions. To understand it, we first need to understand the standard way of interacting with LLMs at the time of its invention: few-shot prompting.
The Foundation: Few-Shot Prompting
Before CoT, the state of the art for guiding an LLM was few-shot prompting. This involves giving the model a handful of examples (the "shots") of the task you want it to perform before you ask your actual question. The model uses these examples as a template for its own answer.
For a simple sentiment analysis task, a few-shot prompt might look like this:
Q: This movie was incredible. The acting was superb.
A: Positive
Q: I would not recommend this restaurant. The food was cold.
A: Negative
Q: I'm not sure how I feel about the book. It was okay.
A: Neutral
Q: The concert was a blast! I can't wait for the next one.
A:The model sees the pattern and will almost certainly complete the last line with Positive. This is effective for simple pattern matching, but it falls apart when a problem requires multiple logical steps. For multi-step math problems, this same pattern-matching habit would often produce a plausible-sounding but incorrect final answer.
The Breakthrough: Wei et al. 2022
In January 2022, a team of Google researchers published the paper that changed everything: "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models ↗". Their insight was to embed the reasoning process inside the examples.
Instead of just providing the question and the final answer in the "shots," they included a step-by-step explanation.
Here's a comparison using a typical word problem from the paper.
Standard Few-Shot Prompting (The Old Way):
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: 11
Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
A: 4
Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?
A: 9When given a new problem, the model tries to leap directly to the answer and often fails.
Few-Shot Prompting with Chain-of-Thought (The New Way):
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.
Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
A: The juggler can juggle 16 balls. Half of the balls are golf balls, so there are 16 / 2 = 8 golf balls. Half of the golf balls are blue, so there are 8 / 2 = 4 blue golf balls. The answer is 4.
Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?
A:By providing examples that "think out loud," the model learns to imitate that behavior. When it sees the new question, it doesn't just output a number. It first generates its own chain of thought, breaking the problem down into sequential, manageable parts. This allocates more computation to the problem, mimicking a more deliberate thought process and dramatically increasing the odds of arriving at the correct answer.
The results were stunning. On the GSM8K benchmark, a set of grade-school math problems, Google's 540-billion parameter PaLM model went from 17.9% accuracy with standard prompting to 58.1% accuracy using Chain-of-Thought [3]. That's a 224% relative improvement from a simple change in the prompt.
An "Emergent Ability" of Scale
Crucially, the researchers found that CoT is an emergent ability. This is a property that doesn't exist in smaller-scale systems but appears once a certain threshold of complexity or size is reached. Chain-of-Thought prompting provides no benefit to smaller models; in fact, it often makes them perform worse [2].
The paper found that CoT only began improving performance in models of around 100 billion parameters or more [2, 3]. Smaller models tended to generate reasoning chains that were "fluent but illogical," leading them further away from the correct answer. Only the largest models had enough latent reasoning capacity to benefit from being prompted to use it — suggesting that the ability was always there, waiting on a prompting method that could activate it.
Zero-Shot CoT: The "Let's think step by step" Hack
While few-shot CoT is powerful, crafting good, diverse examples takes manual effort. A follow-up paper in 2022 by Kojima et al. discovered something even more remarkable: you could trigger a chain of thought with a simple instruction, no examples needed [5].
This technique, called Zero-Shot CoT, involves appending a simple phrase to the end of a question. The most famous one is:
"Let's think step by step."
Let's revisit a problem where a model might fail:
Prompt:
I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I have left?
A model might incorrectly output 10.
Zero-Shot CoT Prompt:
I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I have left? Let's think step by step.
Model Output: `Okay, let's break this down:
You start with 10 apples.
You give away 2 apples to the neighbor, so you have 10 - 2 = 8 apples.
You give away 2 more apples to the repairman, so you have 8 - 2 = 6 apples.
You buy 5 more apples, so you have 6 + 5 = 11 apples.
You eat 1 apple, so you have 11 - 1 = 10 apples. The answer is 10.`
At first glance this looks right, but step 5 appears to contain a reasoning error — a useful reminder that CoT chains, however fluent, aren't automatically sound. Here's a second attempt:
Corrected Model Output with CoT: `Okay, let's break this down:
Start with 10 apples.
Give away 2 to the neighbor: 10 - 2 = 8.
Give away 2 to the repairman: 8 - 2 = 6.
Buy 5 more: 6 + 5 = 11.
Eat 1: 11 - 1 = 10. You have 10 apples left.`
On closer inspection, though, the model's logic was fine all along — the error was in this article's own working, not the model's. That the mistake was so easy to catch is itself the point: it highlights a key benefit of CoT, namely interpretability.
This zero-shot approach is incredibly useful when you don't have well-crafted examples or when prompt length is a concern.
Advanced Variants
The original idea has spawned several powerful variants:
Self-Consistency: This technique, also from Google researchers, runs the same prompt multiple times to generate several different reasoning paths, then takes a majority vote on the final answers — if three paths land on "10" and two land on "9", it goes with "10". This makes the final result more robust by hedging against any single flawed chain of logic [3].
Automatic CoT (Auto-CoT): To get around the manual labor of writing few-shot examples, this method uses Zero-Shot CoT to automatically generate reasoning chains for a set of questions, then uses those generated examples for a more robust few-shot prompt [3, 5].
What this means in practice
For developers building on LLMs, Chain-of-Thought was one of the first and most important tools in the prompt engineering toolkit.
The Upside:
Massive Accuracy Gains for Free: CoT can unlock significant reasoning performance without the enormous cost and complexity of fine-tuning a new model (that is, retraining it on new data). You're simply paying for a few extra words in the prompt and the response.
Improved Interpretability: When a model produces a wrong answer, the chain of thought acts like a debug log — a record of what the program was doing at each step. You can pinpoint the exact logical leap where it went wrong, making it easier to adjust the prompt or spot weaknesses in the model's reasoning [2]. This is a step away from treating the model as an unreadable "black box."
Versatility: The technique works on a wide range of arithmetic, commonsense, and symbolic (logic-puzzle-style) reasoning tasks [2].
The Trade-offs:
Cost and Latency: The reasoning chain isn't free. It adds words to the model's output, which increases both the cost of running it and the time it takes to get a final answer. For an app that needs an instant response, the extra delay from a long chain of reasoning might not be worth it. The core trade-off is speed and cost versus accuracy.
Model Dependency: How well CoT works depends heavily on the underlying model. As of 2025, while it remains a solid strategy overall, its relative value is shrinking for the most advanced systems. A June 2025 Wharton study found that for some of the newest, most capable "reasoning" models, a simple CoT prompt can actually hurt performance on certain tasks [3]. The model gets "over-steered" by the instruction, pushed down a clunkier path than the one it would have found on its own.
It's Not a Cure-All: CoT helps a model organize what it already knows — it doesn't teach it new facts or help it grasp concepts it doesn't already understand. Its biggest weakness shows up when the key information needed to solve a problem isn't stated outright but is only implied [1].
Where this is heading
The original Chain-of-Thought technique is now a basic, well-established part of working with LLMs, but the cutting edge of research has moved on. As of late 2025, the focus is shifting from how to reason to making sure the model has the right information to reason about in the first place.
The Problem with Implicit Information A key limitation of CoT is that it struggles when essential facts aren't spelled out [1]. A model can walk through its steps flawlessly, but if it misses an important, unstated connection buried in the original question, the whole chain of reasoning is built on a shaky foundation. CoT is good at the "reasoning" part of a problem but doesn't help much with the "understanding what's actually being asked" part.
Next-Generation: Iterative Summarization Recognizing this weakness, researchers are building new preparatory steps that run before the reasoning starts. One proposal from early 2025 is called Iterative Summarization Pre-Prompting (ISP²) [1]. Think of it as a step the model takes to organize its notes before it starts "showing its work."
The ISP² process, as described by its authors, works like this:
Extract: The model first pulls out all the people, objects, and facts mentioned in the question, pairing each with its description — these become candidate "key information pairs."
Rate: It then rates how reliable and important each pair seems, prioritizing ones that look significant but whose connections to the rest of the problem are still unclear.
Summarize & Iterate: It takes the two weakest-ranked pairs and asks the model to merge them into a single summary, which forces any hidden relationship between them out into the open. This repeats, pair by pair, until everything has been folded into one consolidated summary.
Reason: That final summary — now containing the previously implicit information stated explicitly — is fed back in with the original question, and a standard Chain-of-Thought prompt can then solve it.
The authors claim this method yields a 7.1% performance improvement over existing methods on tasks where information is implicit [1].
What to Watch Based on these trends, here is a reading of what's next:
Hybrid Prompting: The future isn't about replacing CoT, but layering something on top of it. Expect more multi-stage techniques like ISP² that first organize and clarify the problem, then apply CoT for the actual step-by-step logic. The prompt itself becomes a small program: first understand, then reason.
The Diminishing Returns of Simple Tricks: The Wharton study's finding that CoT's benefit is shrinking for top-tier models is an important signal [3]. As the underlying models get smarter on their own, the simple tricks that used to boost their predecessors will matter less. Prompt engineering will drift away from magic phrases like "Let's think step by step" and toward more deliberate, structured ways of feeding a model information.
Focus on Reliability: As models get better at producing fluent-sounding reasoning, the challenge shifts from getting a chain of thought to getting a trustworthy one. Techniques like Self-Consistency — generating several reasoning paths and checking that they agree — are likely to matter more for high-stakes uses in fields like medicine or law, where a single logical slip can have real consequences.
Chain-of-Thought was a monumental discovery. It taught us that inside these giant neural networks, there are latent capabilities waiting to be unlocked. It moved the goalposts for what we thought was possible with prompting alone. Now, in 2025, it serves as the foundation upon which more complex, reliable, and powerful reasoning structures are being built.