Inside the Go Scheduler: G, M, P, Work Stealing, and Preemption

Goroutines are famously cheap. You can scatter a hundred thousand of them across a program without thinking twice, and the runtime quietly multiplexes them onto a handful of operating system threads. That convenience has a price: the scheduler that makes it work is invisible until something goes wrong — a tail-latency spike nobody can explain, a CPU-bound loop that seems to freeze an entire service, or a container that gets throttled because the runtime guessed the wrong core count.

This post walks through how the Go scheduler actually works — the G/M/P model, the three places a runnable goroutine can wait, how work stealing balances load, and how preemption keeps one greedy goroutine from starving the rest. Everything here is verifiable in the runtime source, and knowing these mechanics turns scheduler surprises from mysteries into expected behavior.

G, M, and P: the cast

The scheduler tracks three entity types. A G is a goroutine: stack, instruction pointer, and scheduling metadata. An M is an operating system thread, the thing that actually executes machine code. A P is a logical processor — a scheduling context that an M must hold to run Go code. The number of P’s is GOMAXPROCS, which determines how many goroutines run simultaneously.

The P is the crucial piece of indirection. When a goroutine blocks on a network read, its M doesn’t sit idle — the runtime parks the G, and the P becomes available for another goroutine, possibly on another thread. When a goroutine makes a blocking syscall, the situation is different: the M is stuck in kernel space where the Go scheduler can’t reach it, so the runtime detaches the P from that M and hands it to another thread. Threads are cheap to spawn from the runtime’s perspective; Ps are the scarce resource.

One recent change worth knowing: the runtime now derives GOMAXPROCS from cgroup CPU limits when running in a container, instead of always using the host’s core count. The GODEBUG mechanism lets you opt out with containermaxprocs=0 if you manage CPU allocation yourself. If you run on Kubernetes and have ever wondered why a 64-core node served by a 2-core-limited pod behaved erratically, this is why — the old default oversubscribed the quota.

Where runnable goroutines wait

A runnable goroutine sits in exactly one of three places:

• The P’s local run queue — a fixed 256-slot ring buffer. Most goroutines live here; it’s a lock-free structure that the owning M accesses without contention.
• The P’s runnext slot — a single reserved slot for the goroutine that was just made runnable (for example, unblocked by a channel operation). It runs next, before anything in the queue. This gives recently-communicated goroutines priority and keeps producer-consumer chains fast.
• The global run queue — a shared, locked queue used as overflow when a local queue is full and as a handoff point for special cases like network-poller wakeups and time.Sleep returns.

When an M finishes a goroutine, it checks its local queue, then the global queue (one check every 61 ticks, to keep the global queue from starving), then polls the network poller. If all are empty, it goes stealing.

Work stealing, in four rounds

Stealing is deliberately cheap to describe and cheap to run. A thief M picks a random starting P (using a random-start traversal order so thieves don’t pile onto the same victim) and tries to take half of the victim’s local queue — not one goroutine, half. Taking half amortizes the cost of the steal and immediately gives the thief meaningful work. If the victim’s queue is empty, the thief checks its runnext slot last — and even then, if that goroutine is currently running, the thief briefly yields to give the victim a chance to schedule it before snatching it.

The whole search is capped: the runtime tries exactly four rounds of stealing per scheduling pass, checking timers on the final round. If nothing turns up, the M parks. A spinning thread is one that’s out of work and hunting — the runtime caps how many threads spin at once (roughly, one per idle P), because spinning burns CPU but cuts wake-up latency when new work arrives. It’s a classic latency-versus-throughput tradeoff, tuned conservatively.

You rarely need to think about stealing directly. In fact, well-shaped Go programs make it almost irrelevant, because work flows through channels and the runnext slot keeps hand-off chains local:

package main

import (
	"fmt"
	"runtime"
	"sync"
)

func main() {
	jobs := make(chan int, 64)
	var wg sync.WaitGroup

	for w := 0; w < runtime.GOMAXPROCS(0); w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for j := range jobs {
				_ = j // process the job
			}
		}()
	}

	for j := 0; j < 100; j++ {
		jobs <- j
	}
	close(jobs)
	wg.Wait()
	fmt.Println("done")
}

Here the scheduler’s queue mechanics barely matter: blocked workers are woken by channel sends, and whichever P has capacity picks the work up. Stealing mainly rescues programs with imbalanced, long-lived goroutines — where one P accumulates a deep queue while its neighbors idle.

Preemption: the 10-millisecond leash

Go’s scheduler is not an OS scheduler. Historically it was purely cooperative: goroutines yielded at function calls, channel operations, and allocation points. A tight loop with no calls inside could hold its P forever. This was a real production hazard — one careless for {} and the garbage collector stalled waiting for that P to reach a safe point.

Two mechanisms fixed this. The sysmon background thread monitors every running P and, when a goroutine has held it for more than 10 milliseconds (forcePreemptNS in proc.go), requests a preemption. Since Go 1.14, that request is asynchronous: the runtime sends the thread a signal and the goroutine stops at the next instruction boundary, not at the next function call. Tight loops became safe:

package main

import (
	"fmt"
	"runtime"
	"time"
)

func main() {
	fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

	go func() {
		var sum uint64
		for {
			sum++ // no function calls, no allocations — yet still preemptible
		}
		_ = sum
	}()

	for i := 0; i < 5; i++ {
		time.Sleep(100 * time.Millisecond)
		fmt.Println("main loop still responsive:", i)
	}
}

Run this with GODEBUG=schedtrace=1000 and the runtime prints a scheduling snapshot every second — queue depths, spinning threads, thread counts. It’s the fastest way to see the scheduler’s side of the story when a program misbehaves.

Two gaps remain. Goroutines stuck in blocking syscalls or cgo calls can’t receive the preemption signal — the sysmon path explicitly notes that preemption doesn’t work while a goroutine is in a syscall, which is why the P-detachment mechanism matters more there. And a goroutine that’s frequently preempted burns time in context switches; if profiling shows heavy scheduler activity, the fix is usually to batch the work, not to fight the runtime.

Practical takeaways

• Check GOMAXPROCS in containers. Verify what the runtime picked at startup; a mismatch between cgroup quota and scheduler width is a common latency source. Since recent releases the runtime handles this itself, but older deployments may still need explicit tuning.
• Don’t spawn goroutines per request without bounds. The scheduler scales fine to hundreds of thousands of G’s, but each carries a stack (starting at a few KB) and scheduling overhead. Semaphores or worker channels cap the population.
• Long syscalls and cgo calls cost a thread. The P gets handed off so throughput survives, but hundreds of concurrently blocked syscalls mean hundreds of OS threads. That’s an M-level problem, not a P-level one.
• Use GODEBUG=schedtrace=1000 and the execution tracer (go test -trace, or runtime/trace in production) before theorizing. Most “scheduler” problems turn out to be lock contention or GC assists that merely look like scheduling delays.

The Go scheduler’s design — distributed queues with work stealing, a spinning-thread budget, and a hard 10ms time slice — is a masterclass in getting predictable behavior from cooperative user-space threading. You don’t need its internals to write ordinary Go, but when a service misbehaves under load, the difference between guessing and diagnosing is knowing where a runnable goroutine can hide, and what the runtime does about the ones that refuse to share.