we replaced a tidy little blocking-thread-per-connection model with io_uring expecting throughput to go up and latency to follow. throughput did go up. p99.9 doubled. this post is the bisect.
the short version: SQPOLL plus a saturated CPU plus an unfortunate cgroup pinning interacted to produce a multi-millisecond stall on submission that nothing in our metrics caught for two weeks. the rest of this post is the long version.
customer reports: writes pause for 4-6 seconds, every 90 minutes, give or take. no panics. no GC. compaction looks healthy. metrics show nothing. it took me an afternoon and an embarrassing amount of strace to figure out it was the WAL file rotation racing with the fsync on the manifest.
# the smoking gun strace -f -e trace=fsync,rename,openat -p $(pidof db) 2>&1 \ | awk '/manifest/{print strftime("%T"), $0}'
i implemented raft in 2019. i implemented joint consensus in 2021. it took until last week, reading dragonboat's source, for the membership-change algorithm to actually click for me. this post is the explanation i wish someone had handed me.
short one. if you're using a lot of nested arenas in zig and finding the lifetime accounting annoying, you can wrap an arena in a small helper that exposes both the underlying allocator and a "scoped reset" handle. handy for parser code.