Every Tokio application eventually meets the same mystery: a task that looks harmless in code review brings an entire runtime to its knees. No panics, no blocked threads, no obvious errors — just latency creeping up everywhere else until some other task barely gets polled at all. The cause is almost always the same: one future that never yields, sitting inside a scheduler that has no way to forcibly preempt it.
Rust gives you async/await but no runtime. That design decision is what makes Tokio fast, but it also means the runtime cannot pause a task mid-poll the way an OS kernel preempts a thread. Tokio’s answer is a cooperative budgeting system layered on top of its work-stealing scheduler, plus a handful of polling policies that decide who runs next. Understanding how those pieces fit together turns “my runtime feels slow” from a guessing game into a diagnosable problem.
This post walks through the mechanics: how a task actually gets polled, what the cooperative budget of 128 operations means in practice, why a stream that is always ready can starve everything else, and how the multi-threaded scheduler decides between the LIFO slot, the local queue, and the global queue. All of the internal numbers here come from the current Tokio source (the 1.53.x series), and every behavior described is verifiable in the repository.
The preemption problem
Consider what a task is in Tokio: an async block or async function, wrapped in a Future, stored in the runtime’s task heap and driven by a worker thread that calls poll on it. A single poll call is allowed to do an unbounded amount of work before it returns Poll::Pending. The compiler inserts yield points only at .await sites — and only when the awaited future actually returns Pending. If the future is immediately ready, execution falls straight through.
That leads to the classic starvation shape, which the Tokio documentation itself uses as its example:
async fn drop_all<I: Stream + Unpin>(mut input: I) {
while let Some(_) = input.next().await {}
}Under load, when the stream always has an item ready, every .await returns immediately. The task never returns Pending, so the worker never gets a chance to poll anything else. Other tasks queue up behind it indefinitely. A preemptive scheduler would suspend this task after a time slice; Tokio cannot, because the state needed to suspend a poll is whatever the future happens to hold across an .await, and there is no language-level timer that can fire mid-function.
The solution is cooperation: futures that perform many ready operations are expected to periodically give control back, and the runtime tracks whether they’re holding up their end of the bargain.
How the cooperative budget works
Deep inside Tokio lives a small module, tokio::task::coop, that implements this deal. The contract is simple: each task poll starts with a budget, and every “yield point” the task consumes — each channel send, each read on a socket that returns data, each recv() that immediately succeeds — costs one unit. When the budget runs out, the next yield point forces the task to return Pending even though the underlying resource is still ready. The runtime reschedules the task and hands it a fresh budget.
The initial budget is 128 operations, a constant in Tokio’s source. The comments around that constant explain the trade-off it encodes: the value has to be low enough that a greedy task cannot starve its siblings for long, but high enough that legitimately deep tasks still make useful progress before yielding. It is deliberately not part of the stable API contract — you shouldn’t write code that depends on exactly 128 — but knowing the magnitude helps when interpreting latency behavior.
Two public tools let you interact with the system directly. The first is consume_budget(), which lets hand-rolled loops participate in the protocol:
use tokio::task;
async fn process_batch(items: Vec<Item>) {
for item in items {
process(item).await;
// Explicitly give the scheduler a chance to run
// other tasks in long, CPU-ish loops.
if task::coop::consume_budget().await.is_err() {
// Budget exhausted: the task will be rescheduled
// and resumed with a fresh budget later.
}
}
}The second is task::unconstrained(), the escape hatch. Wrapping a future in it removes budget enforcement entirely — the future will never be forced to yield. This is the right call for futures whose correctness depends on making progress in one poll (some third-party state machines and hand-rolled poll implementations break if they get spuriously Pending), but it is also exactly how you re-create the starvation bug on purpose, so it deserves a comment wherever it appears.
One subtlety worth knowing: the budget is tracked per task poll, not per future. If your task awaits a child future, the child runs under the same budget. This is why deeply nested ready-to-fire chains can exhaust a budget “unexpectedly” — every operation in the chain draws from the same pool of 128.
Diagnosing budget exhaustion
Budget-forced yields are invisible at the API level — recv() just returns Pending one extra time. The visible symptom is usually throughput flattening under load: a channel consumer that could theoretically drain millions of messages per second instead caps out, because each poll is limited to 128 sends/receives before a reschedule.
The patterns that reliably trigger it:
- A
looparoundrx.recv().awaiton anunbounded_channelwith a hot producer — every recv is instantly ready, so the budget drains at maximum speed. - Batch-processing loops over in-memory collections inside an async fn, with an
.awaiton an always-ready operation per item. - Retry loops that poll a service which responds in microseconds — each attempt costs budget units.
The fix is not to remove the budget but to make the loop a good citizen: call consume_budget() at a sensible interval, batch work so each poll does more per budget unit, or move genuinely CPU-bound sections into spawn_blocking so they don’t occupy a scheduler poll at all.
The waker half of the story
Budgets only matter because of how tasks get back onto the run queue. When a future returns Pending, it must have stored a Waker somewhere — a socket registration, a channel’s waiting list, a timer wheel. When the resource becomes ready, that waker’s wake() is called, and Tokio schedules the task onto a worker’s queue. The next poll gets a fresh budget of 128, and the cycle repeats.
This is also why a task that ignores its budget for too long is so damaging: while one task monopolizes a worker thread, wakeups for other tasks pile up in queues that nobody is draining. Tokio’s metrics (behind the rt feature’s Handle::metrics()) expose queue depth and poll counts, which makes the imbalance visible: one task accumulating poll-time share while every queue it feeds backs up.
A practical review checklist for async code follows directly from this:
- Any unbounded loop with an
.awaitinside should either consume budget explicitly or have a bounded iteration count per poll. - Prefer bounded channels in hot paths — backpressure through
send().awaitnaturally yields, whiletry_send-style always-succeeding paths do not. - Treat
unconstrained()as a lint-worthy escape hatch, not a performance knob.
Inside the multi-threaded scheduler’s polling policy
Budgeting decides how long a single poll runs. The scheduler’s queue policy decides which task gets polled next — and here Tokio’s multi-threaded runtime has a few carefully tuned numbers.
Each worker thread keeps a local run queue, plus there’s one global (injection) queue for spawned tasks and for overflow. To keep global-queue tasks from starving while workers grind through local work, workers periodically poll the global queue. The interval is self-tuning: Tokio tracks an exponentially weighted moving average of how long a task poll takes, and aims to visit the global queue roughly every 200 microseconds, rather than counting a fixed number of tasks. The source caps that at 127 tasks polled per global-queue visit and targets around 61, so a burst of trivial tasks can’t push the global-queue visit off indefinitely.
If you need to override the auto-tuning — typically only in benchmarking or latency-sensitive edge cases — the runtime builder exposes global_queue_interval, which takes a Duration and disables the adaptive logic:
use std::time::Duration;
use tokio::runtime::Builder;
fn main() -> std::io::Result<()> {
let rt = Builder::new_multi_thread()
.global_queue_interval(Duration::from_micros(500))
.build()?;
rt.block_on(async { /* ... */ });
Ok(())
}The other notable piece is the LIFO slot: the most recently woken task goes into a dedicated slot on the worker (not the back of the local queue), and the worker checks it first. That makes message-passing ping-pong between two tasks — a request handler and the service it calls — take one hop instead of a full queue rotation. The slot is deliberately shallow; the source caps related slot polls per tick (a constant of 3 in the current code) so a chatty pair of tasks can still starve others through it. It’s a good reminder that all of Tokio’s fairness mechanisms are heuristics with explicit trade-offs, none of them absolute guarantees.
Between the budget system (bounds work per poll) and these queue policies (bound wait time between polls), Tokio approximates fair scheduling without ever preempting anything. Every rule has a documented exception, which is exactly why reading the actual source pays off — the numbers change occasionally (the budget value’s own comments predict it will rise as the ecosystem adds yield points), but the structure has been stable for years.
Wrapping up
When a Tokio application slows down under load, the suspects are almost always mechanical: a future that never yields and exhausts its cooperative budget invisibly, an unconstrained() wrapper applied too liberally, or a queue policy interacting badly with a chatty task pair. The toolkit for fixing them is small and public — consume_budget() for cooperative loops, bounded channels for natural backpressure, spawn_blocking for CPU work, and runtime-builder knobs for the rare case where the defaults genuinely don’t fit.
If you take one thing from the internals: Rust’s async model delegates fairness entirely to the runtime and to you. The budget of 128 is Tokio politely asking your futures to share — and now you know how to make sure they do. The cooperative scheduling docs and the Tokio repository are both worth a slow read; the scheduler source in particular is some of the most instructive systems code in the ecosystem.