~/writing
Writing
Notes on timing, audio, embedded systems, and debugging. 16 posts.
Putting back the bass that was never recorded
I prepped a music set for a 1500-watt DIY rig and synthesized a missing sub-bass octave into tracks rolled off below 40 Hz. Measure first, fix only what the numbers justify, don't touch anything else.
What is putting those bytes on the wire?
The storage interface on a production cluster was pulling 4 to 12x more than the public interface where client requests arrive. Getting to a defensible number meant distrusting everything, including my own profiler, which was off by 30x.
A Dovecot optimization the compiler had been deleting all along
An IMAP THREAD command was panicking inside Dovecot's array code. The crash was real, but the comment above it described an optimization that hadn't run in any -O2 build ever shipped. ATTR_PURE turned it into dead code, and the same annotation decided which line showed up in the backtrace.
A Ceph cluster I throw away on every reboot
I wanted a real Ceph endpoint (S3, CephFS, RBD) to test app code against, without a k8s cluster or a single byte hitting my SSD. One container, ~30 seconds, all in RAM. The version sweep is where it got interesting.
The $20 stratum 1 that chrony kept benching
I built a GPS-disciplined NTP server on a $20 ESP32 with single-nanosecond internal timing. Then chrony refused to use it. The fix was not in the clock. It was in everything I had assumed about serving the time.
You cannot average percentiles
Finding an IMAP server's real capacity means driving load from many hosts at once. The hard part isn't generating the load. It's combining their latencies into one honest P99 without quietly lying about it.
The frozen clock that nobody noticed
A GPS-disciplined NTP server stopped being disciplined, and every monitor stayed green. The giveaway was a freq_ppb value that repeated byte-for-byte across three seconds, which a live servo never does.
130 frames per second, derived from a wire
I built a Go daemon that drives a music visualizer on a Pi, then found out the frame rate wasn't mine to pick. The LED protocol had already decided it. The real work was a latency budget I had to measure before the lights felt locked to the music.
ts2phc and gpsd cannot share a serial port
One USB serial port, two programs that both need exclusive access. ts2phc wants RMC-only for PTP clock discipline; gpsd wants everything else. I wrote a demuxer that gives each one a synthetic port shaped exactly the way it expects.
The spare cores in a BeagleBone keep better time than its kernel
The standard Linux PPS driver timestamps the GPS pulse in an interrupt handler and pays ~20 µs of jitter for it. The BeagleBone has two 200 MHz real-time cores that Linux never touches. I moved the timestamp into one of those, and the clock offset dropped into the low nanoseconds.
Tidal stopped serving lossless, so I read the token
Upstream plugin started pulling 320 kbps instead of lossless FLAC, and swapping client IDs did nothing. The access token had frozen its entitlements at mint time, long before any request went out.
The game that rebooted the hypervisor
A friend's gaming VM hard-reset its Proxmox host the instant a game launched, with nothing in the logs. The cause was below every layer we kept editing: a power transient at game launch that tripped a PCIe fault and reset the box.
The furnace controller that deadlocked when the WiFi dropped
An RP2040 furnace controller would freeze at random, always when the WiFi dropped. The culprit was a connectivity probe that opened a TCP connection to decide whether it could open one. A timeout would have hidden it; removing the probe fixed it.
The GPS that answers 0xFF when it has nothing to say
Reading a u-blox GPS over I2C frees the UART, but the I2C interface has no concept of idle. When the module has nothing to send, every read comes back as 0xFF filler. This is the small bridge that turns that into a chrony refclock.
One socket for the whole subnet
A host scanner that opened a raw socket and a goroutine per host fell over on anything bigger than a /24. The rewrite uses one socket and two goroutines no matter how many hosts, with an ARP fast path and an unprivileged fallback.
My weather station lies about the sun
A cheap Ecowitt station reports UV and solar radiation that read high and jump around. The repeater that forwards it to Weather Underground smooths and scales those channels first, and never lets a slow upstream API stall the station.