Every high-concurrency server you have ever deployed eventually bottoms out at the same place: a call that says “tell me which of these ten thousand sockets is ready.” The abstraction above it may be an http.Server, a tokio runtime, or a bespoke reactor, but underneath there is an event loop talking to a kernel readiness interface. Most developers treat that interface as a black box. That is usually fine — until you have to debug latency spikes under load, size a thread pool, or decide whether an io_uring rewrite is worth the risk.
This post walks through the three interfaces that dominate production systems today — Linux’s epoll, BSD/macOS kqueue, and io_uring — from the perspective of what each one actually costs per event, and what that cost implies for the runtimes built on top of them. The goal is practical: by the end you should know why Go’s netpoller still uses epoll, what io_uring changes structurally rather than just numerically, and when the migration pays for itself.
The readiness model and its hidden cost
epoll, and its BSD cousin kqueue, are readiness interfaces: the kernel tells you a file descriptor is ready, and your code then performs the actual read or write with a separate syscall. The canonical three-call setup looks like this:
int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);
struct epoll_event events[MAX_EVENTS];
int n = epoll_wait(ep, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
// handle events[i].data.fd -- still need read()/accept() after this
}The per-connection registration is a one-time cost, but the hot path carries two syscalls per I/O operation: one for epoll_wait (amortized across the batch it returns) and one each for the read or write that follows. On a proxy moving a million small events per second, that is a million-plus syscall crossings doing nothing but shuttling control between user space and the kernel.
Two design choices shape everything else about epoll servers:
- Level-triggered vs edge-triggered. Level-triggered mode re-reports a descriptor on every call as long as it stays readable, which is forgiving but can wake your loop repeatedly for the same data. Edge-triggered mode reports only state transitions, which cuts wakeups but obligates you to drain the descriptor until
EAGAIN— miss one drain and the connection stalls forever. Most runtimes choose edge-triggered and eat the complexity. - One-shot semantics.
EPOLLONESHOTdisables a descriptor after one event, which is how runtimes keep a single connection from being processed by two threads concurrently. It also means every event requires a follow-upepoll_ctlto re-arm, adding yet another syscall to the hot path.
kqueue: the same idea, wider scope
If you run anything on BSDs or macOS, the equivalent interface is kqueue, and its notable difference is breadth. epoll watches file descriptors. kqueue watches descriptors, signals, process lifecycle events, filesystem events, and timers through one uniform kevent structure:
int kq = kqueue();
struct kevent changes[2];
EV_SET(&changes[0], listen_fd, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, NULL);
EV_SET(&changes[1], SIGTERM, EVFILT_SIGNAL, EV_ADD, 0, 0, NULL);
kevent(kq, changes, 2, NULL, 0, NULL);
struct kevent events[MAX_EVENTS];
int n = kevent(kq, NULL, 0, events, MAX_EVENTS, NULL);The EV_CLEAR flag is kqueue’s analog of edge-triggering. For cross-platform runtimes this breadth matters less for performance than for API symmetry: the same event loop structure can watch a child process and a TCP socket without special cases. Go’s runtime, for instance, ships a kqueue netpoller (netpoll_kqueue.go) alongside its epoll one, and selects between them at build time.
io_uring: removing the syscall from the hot path
io_uring, merged into Linux in kernel 5.1 and realistically usable from about 5.10, inverts the model. Instead of readiness notifications, it is a completion interface built on two ring buffers shared between user space and kernel: a submission queue (SQ) where you place operation descriptors, and a completion queue (CQ) where the kernel places results. You submit a read; you get the data back as a completion. No readiness check, no follow-up syscall.
The current UAPI defines 63 operation opcodes, and the range is the tell: this is not just an async I/O interface for files. Alongside IORING_OP_READ and IORING_OP_WRITE you will find IORING_OP_ACCEPT, IORING_OP_CONNECT, IORING_OP_SEND, IORING_OP_RECV, IORING_OP_CLOSE, IORING_OP_FUTEX_WAIT, and even IORING_OP_EPOLL_WAIT for compatibility. An entire network server lifecycle — socket, bind, listen, accept, transfer, close — can run through the rings.
Three features do most of the performance work:
- Kernel-side polling (
IORING_SETUP_SQPOLL): a dedicated kernel thread drains the submission queue continuously, so the application never issues a syscall to submit work — it writes SQEs into shared memory. The kernel thread does the syscall-equivalent work on the kernel’s side of the boundary. - Multishot operations: a single
IORING_OP_ACCEPTor a multishot receive stays armed across completions, re-arming itself in the kernel. This collapses the epoll pattern of “event, then syscall, then re-arm” into one persistent operation. - Zero-copy send (
IORING_OP_SEND_ZC): for payloads above roughly 4 KB, the user-to-kernel copy on send is eliminated, which matters enormously for proxy and object-store workloads.
A minimal submit-and-reap loop shows the shape of the API via liburing (current release 2.15):
struct io_uring ring;
io_uring_queue_init(256, &ring, 0);
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_recv(sqe, conn_fd, buf, sizeof(buf), 0);
io_uring_sqe_set_data(sqe, conn);
io_uring_submit(&ring);
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
if (cqe->res > 0) {
// got cqe->res bytes in buf
}
io_uring_cqe_seen(&ring, cqe);One caveat deserves emphasis: IORING_SETUP_SQPOLL and the fancier completion paths interact with a history of security restrictions. Several distributions have shipped kernels with io_uring disabled or restricted after repeated vulnerability reports, and container runtimes have periodically blocked it inside sandboxes. Before betting a product’s hot path on io_uring, verify it works — and keeps working — in your deployment targets’ kernel and seccomp configuration.
What Go and Rust actually do today
Given io_uring’s obvious throughput wins, it is fair to ask why mainstream runtimes have not switched. Go is the clearest case study. The runtime’s netpoller — the machinery that makes goroutines blocking on sockets cheap — implements epoll on Linux and kqueue on the BSDs/macOS, and nothing else. The design reason is semantic: Go’s model is “every goroutine thinks it is blocking.” Readiness interfaces map onto that model naturally (block the goroutine until the descriptor is ready), while a completion interface would require either per-connection buffers pinned in shared memory rings or a goroutine parked per in-flight operation with results routed back by completion. That is a real redesign with GC interaction questions, not a drop-in swap, and for the vast majority of Go services the epoll-based netpoller is nowhere near the bottleneck.
In Rust the situation is opt-in. Tokio’s mainstream socket types sit on epoll/kqueue via mio. The tokio-rs/io-uring crate (around 1,750 stars at the time of writing) provides io_uring bindings, and the separate tokio-uring runtime offers a complete io_uring-backed alternative — but with its own API, deliberately different from standard tokio because completion semantics do not hide cleanly behind the AsyncRead/AsyncWrite traits. Neon’s engineers took a middle path with a crate that bridges io_uring into vanilla tokio. The ecosystem’s fragmentation is itself the lesson: integration is genuinely hard, so it stays niche.
When does the upgrade actually pay?
Micro-benchmarks showing 2–3x throughput for echo servers are real but misleading — echo servers exist to showcase batching. The honest decision framework looks more like this:
- Mixed file and network I/O. This is io_uring’s clearest structural win. With epoll you need one event loop for sockets and a thread pool for blocking file reads; with io_uring both are SQEs in the same ring, which removes an entire class of handoff complexity. Databases, proxies with disk-backed caches, and log shippers benefit most.
- Very high connection counts with small I/O. Once syscall overhead dominates, SQPOLL and multishot operations meaningfully reduce CPU per event. Below tens of thousands of concurrent connections, epoll’s overhead is usually lost in the noise.
- Large-payload egress. Zero-copy send pays directly when messages exceed a few kilobytes — CDN edges, file servers, media proxies.
- Everything else. If your service’s 99th-percentile latency is dominated by upstream calls, database queries, or serialization, no event-loop upgrade will show up in the metrics. Profile first.
There is also an operational tax to price in: io_uring’s fast-moving feature surface means behavior varies across kernel versions in ways epoll (stable since 2003 in its current form) does not. Teams running heterogeneous fleets or older LTS kernels need feature detection and fallback paths, which is exactly the engineering cost that keeps runtime maintainers on epoll.
Wrapping up
epoll and kqueue are readiness interfaces: cheap, stable, and battle-tested, with a per-event syscall cost that only matters at scale. io_uring is a completion interface that removes syscalls from the hot path entirely and unifies file and network I/O in one ring — a genuine architectural shift rather than an incremental optimization. The runtimes you use daily still standardize on the readiness model because it fits their execution semantics, and that will hold until the completion model can be hidden as cleanly.
Practical next steps: check what your runtime actually uses (strace -c on a running Go or Node service will show epoll_wait counts immediately); if you operate proxies or databases on Linux 5.10+ and CPU per event is a measured problem, prototype against liburing before rewriting anything; and verify your container sandbox’s io_uring posture before you depend on it. The kernel’s I/O interface is one of the few layers where understanding the mechanism — not just the API — changes design decisions.