How Reasoning Models Decide How Much to Think

Most people picture a reasoning model as something that "thinks hard" before answering. What they don't realize is that, until very recently, the model thought the same amount regardless of the question — a fixed budget of reasoning tokens for an easy lookup and a genuinely hard proof alike.

That's a problem. It means the model over-computes the trivial and under-computes the difficult. A 2026 paper (Learning When to Think) tackles this head-on, and the result is one of the most immediately usable efficiency tricks in the agent world.

The problem with fixed thinking budgets

Reasoning models are typically trained with reinforcement learning to produce a chain of "thinking" tokens before they answer. The catch: the budget is usually fixed. Every problem gets the same depth of reasoning.

• An easy arithmetic check gets the full treatment.
• A genuinely hard logic puzzle gets the same budget — often not enough.

The waste is twofold: you pay in tokens (money) and in latency (seconds the user waits) on every easy query, while the hard queries still fall short.

The fix: let the model choose

The researchers' idea is disarmingly simple. Instead of an external router deciding how much to think, the model itself decides as the very first token of its response. It picks one of three modes:

• NoThink — answer immediately, no reasoning.
• Short — a brief reasoning pass.
• Long — an extended reasoning pass.

The choice is learned inside the RL loop (GRPO) using a shaped reward and hard per-mode token caps. No separate classifier, no external scheduler. The decision is just... part of the output.

What they actually found

Trained on a 1.5B distilled model over math problems, all three modes emerged without collapsing into a single default. The interesting part:

• Brief modes ended up more accurate than Long on many problems — the router was effectively sorting by difficulty.
• Average response length dropped from 4,796 to 2,811 tokens (a 41% reduction) with near-identical accuracy (0.782 vs 0.796 on MATH500).
• It transferred without retraining to other benchmarks — a 76% token reduction on GSM8K at higher accuracy than the baseline.

Why this matters for agents

Test-time compute allocation is one of the biggest efficiency frontiers in AI right now. For an agent running a long, multi-step task, the cost compounds: every step is a model call. A model that knows when not to think is dramatically cheaper to run in production.

The elegance here is the routing decision living inside the model's first token rather than in a brittle external system. That makes it deployable today, not just a research curiosity.

The honest caveat

This was demonstrated on math-style reasoning. Real agent workloads are messier — but the principle transfers: not every step deserves full reasoning. The agents that win on cost will be the ones that learn to think only when it counts.

If you want the deeper breakdown of what lives inside an agent's reasoning loop, the full walkthrough is in the 5 core components of AI agent architecture, including how the reasoning engine decides what to do next.

Related: The AI Agent Architecture: 5 Core Components · AI Agent Guide · AI agent token budget calculator

Next Steps

See how the reasoning engine sits in the loop, then size the token bill before you run a long agent task.

how do AI agents work — return to the complete AI agent architecture guide.