Every high-concurrency network server on Linux eventually converges on the same bottleneck: not CPU, not memory, but the question of how a single process discovers, cheaply and quickly, which of its ten thousand sockets have data ready right now. The answer the kernel settled on is epoll, and if you write servers — in any language — you are running on it whether you know it or not. Node’s event loop, Go’s netpoller, Rust’s tokio, Java NIO, Python’s asyncio, Redis, and nginx all sit on epoll (or its cousins kqueue/IOCP on other platforms). Understanding what it actually does explains a lot of otherwise mysterious behavior: why level-triggered versus edge-triggered modes matter, why one slow callback stalls thousands of connections, and why select() fell over at a few thousand descriptors.
This post is the systems-level tour: what problem epoll solves, how the kernel-side data structures work, what the two triggering modes really mean, and how the runtimes you use every day build their event loops on top of it.
The problem with select and poll
Before epoll, the tools for multiplexing I/O were select and poll. Both share the same fatal scaling shape: every call passes the entire set of descriptors you care about from user space into the kernel, the kernel walks all of them checking readiness, and then it returns only a count — forcing user space to scan the whole set again to find which descriptors actually fired. With 10,000 connections where 50 are active per event loop iteration, you pay for inspecting 10,000 descriptors twice to learn about 50.
The cost is O(total descriptors) per call, and it repeats every iteration. Worse, the kernel cannot remember anything between calls — the interest set is re-registered, re-copied, and re-walked every time. select adds one more indignity: the fd_set bitmask is fixed size (historically 1024 descriptors unless you recompile with a different FD_SETSIZE), so busy servers literally could not represent their connection count.
epoll redraws the split of responsibilities. Registration becomes a persistent operation: you tell the kernel once which descriptors to watch, and the kernel maintains that interest list in kernel space across calls. Waiting then returns only the ready list.
The three syscalls
epoll’s API is three calls. First you create an epoll instance — a kernel data structure, not a file you can meaningfully read, but one with a descriptor so it participates in the usual descriptor machinery (including being pollable itself, which is how nested event loops work):
#include <sys/epoll.h>
int epfd = epoll_create1(0);epoll_create1 takes flags (pass EPOLL_CLOEXEC in production so the descriptor doesn’t leak across exec into child processes). Then you register interest in individual descriptors with epoll_ctl:
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET; /* readable, edge-triggered */
ev.data.fd = client_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);EPOLL_CTL_ADD, EPOLL_CTL_MOD, and EPOLL_CTL_DEL manage registrations. Registration is incremental and persistent — adding one new connection costs one syscall touching one descriptor, regardless of how many other connections exist. Finally, the wait:
struct epoll_event events[1024];
int n = epoll_wait(epfd, events, 1024, /*timeout_ms=*/-1);
for (int i = 0; i < n; i++) {
handle_io(events[i].data.fd, events[i].events);
}epoll_wait sleeps until some registered descriptor is ready (or the timeout expires) and fills the array only with ready events. The loop over the returned array touches exactly the active connections. That is the O(active) instead of O(total) win in its entirety — the same shape of difference as O(n²) to O(n log n) in algorithmic terms, except it applies to every iteration of your server’s main loop for its entire uptime.
What the kernel actually maintains
Two structures do the work. The interest list (a red-black tree keyed by descriptor) records every registration, making ADD/MOD/DEL operations O(log n) and lookups by descriptor fast. The ready list is the clever part: it is populated not by epoll_wait scanning anything, but by kernel callbacks installed on each registered file. When a NIC interrupt ultimately results in data arriving on a socket’s receive queue, the protocol layer wakes the socket’s wait queue, the epoll callback fires, and that descriptor’s entry is linked onto the ready list. When epoll_wait runs, it simply checks whether the ready list is non-empty and, if so, returns its contents. No scanning of ten thousand sockets. The work happens where the events occur, not where they’re collected.
This design also explains the famous caveat: if descriptors are shared across multiple epoll instances or duplicated via dup(), a single readiness event may be delivered to more than one waiter, and a registration refers to the file description, not the descriptor number. Forked processes sharing a listening socket’s file description can therefore see thundering-herd wakeups on the same event. It is rarely a problem in practice, but when it bites, this is the mechanism.
Level-triggered vs edge-triggered
By default, epoll is level-triggered: as long as a descriptor remains ready — socket buffer still holds unread data — epoll_wait keeps reporting it on every call. Miss a read? It is reported again next iteration. This is forgiving and is what most runtimes use.
Add EPOLLET and behavior flips to edge-triggered: you are notified exactly once per state transition — when new data arrives — and never again for the same data. If you read only half the available bytes and return to the event loop, nothing re-awakens you. The remainder sits in the buffer and your connection stalls, possibly forever. Edge-triggered code therefore must obey a strict discipline: drain until you get EAGAIN (the kernel’s way of saying “would block — nothing more available”), and keep every descriptor in non-blocking mode, because any blocking read or write now deadlocks a thread with no one to wake it.
/* Edge-triggered discipline: read until EAGAIN, always non-blocking */
ssize_t n;
for (;;) {
n = read(fd, buf, sizeof(buf));
if (n > 0) { process(buf, n); continue; }
if (n == 0) { peer_closed(fd); break; }
if (errno == EAGAIN || errno == EWOULDBLOCK) break; /* drained */
if (errno == EINTR) continue; /* retry */
perror("read"); break;
}Why does anyone accept this complexity? Because edge-triggered mode reduces syscall count: with a burst of five packets arriving on one socket, level-triggered mode may return that descriptor in five successive epoll_wait calls (one wait per drain round if you read only part each time), while edge-triggered wakes you once and lets you drain everything in a tight userspace loop. High-performance servers historically chose ET for exactly this reason. The trade is real but narrower than folklore suggests — modern runtimes mostly find other ways to batch, and forgiving LT semantics are easier to keep correct under refactoring.
How runtimes build on this: the Go netpoller as a case study
The interesting engineering is not the syscalls — it is how a runtime grafts epoll onto its concurrency model. Go is worth studying because its design is the most integrated. The runtime’s netpoller registers every network descriptor with an epoll instance (Linux; kqueue on BSDs, IOCP on Windows) and connects readiness events to the goroutine scheduler. The flow for a blocking-looking read:
conn.Readon a socket with no data doesn’t block the OS thread. The runtime parks the goroutine on the pollDesc (the runtime’s per-descriptor wait structure) and moves on; the OS thread picks up other work.- When data arrives, the kernel’s epoll callback (in the platform-specific file — netpoll_epoll.go on Linux) marks the pollDesc ready and hands the goroutine back to a run queue.
- The scheduler’s
findRunnablepath callsnetpollin two situations: when an idle thread has nothing else to do, and periodically fromsysmon, the runtime’s background monitor, so polling happens even under full CPU load.
The consequence is the defining property of Go servers: you write straightforward, synchronous-looking, blocking-style code, and the runtime converts it into an event-driven state machine under you. There is no callback inversion, no async coloring, and yet the OS thread count stays pinned to core count while goroutines number in the hundreds of thousands. One per-connection thread would need an OS thread each and die at a few thousand connections; goroutines waiting on the netpoller cost a few hundred bytes each and no thread at all.
Other runtimes make the same integration explicit instead of invisible. Node and Python’s asyncio hand you the event loop directly — you register callbacks or await futures that the loop resolves from epoll_wait results. tokio wraps epoll (via mio) in its reactor and drives futures from it. Java NIO exposes SelectionKeys, which map almost one-to-one onto epoll_event entries. The polling substrate is identical; the difference is purely where in the API surface the event loop becomes visible to you.
Where epoll shows its limits
epoll’s per-event costs — syscall boundary crossings and one wakeup per readiness event — become the bottleneck in the most demanding regimes. Two directions of evolution matter if you are pushing extreme throughput:
io_uring replaces the readiness model with a submission/completion queue pair shared between user space and kernel. Instead of being told “socket is readable, come do the read” (two crossings: epoll_wait then read), you submit a read request and collect its completion — batching many operations per crossing. io_uring_setup creates the shared queues. For storage I/O io_uring is already dominant; for networking it is winning where request rates are extreme, though epoll remains the portable, well-understood default and every mainstream runtime still ships it as the primary path.
NAPI busy polling attacks the interrupt side. Normally each packet burst costs an IRQ, a softirq, and a wakeup. Busy polling lets the event loop poll the NIC’s RX queue directly during the wait, trading CPU for latency. The kernel’s NAPI documentation describes both the socket-level SO_BUSY_POLL option and the io_uring NAPI integration. Latency-sensitive services (market data, telecom) use it; most servers rightly keep interrupts.
Practical takeaways
- Trust the substrate, but know its shape. Your runtime’s event loop is epoll plus scheduling policy. When you see symptoms like periodic latency spikes under load, the interesting questions are about the scheduling policy (how many callbacks per tick, are they bounded), not the syscall.
- One slow handler stalls everyone. In a single-threaded event loop, all connections share one iteration path. A handler that blocks (sync DNS, sync file I/O, a long computation) delays every other connection’s events — this is the event-loop version of holding a lock too long, and it is the most common real-world epoll-adjacent bug.
- Level-triggered is the default for a reason. If you write raw epoll (or review code that does), prefer LT unless you have measured a syscall-count problem ET solves. The EAGAIN-draining discipline is correct only when every code path follows it, forever, including error paths.
- Watch the ready-list storm, not just the syscall count. Sharing listening sockets across processes that each run their own epoll can produce wakeup storms under accept load. If you scale acceptors, measure wakeups per connection before and after.
- Kernel features flow slowly into runtimes. io_uring and busy polling are real and production-grade, but adoption lags kernel support by years in mainstream runtimes. Check what your runtime’s reactor actually uses before assuming you have the fastest path.
The deeper lesson is architectural. The systems that scale to absurd connection counts are not the ones that bought more hardware — they are the ones that moved work from “collecting events” to “reacting to events,” and pushed the collection down to the exact point where each event occurs. epoll is that idea, twenty-plus years old, still load-bearing under everything you deploy.