Posted: 2026-09-01
An outage was underway. It went on for several hours, at least.
It is manifesting as SNIProxy failures, mirror.ssvr.net 500s, and unbound (recursive resolver) errors.
The tricky issue is that I couldn't SSH into Freya to diagnose it, so I drove to Spokane. It stems from running out of file descriptors. I am not sure why this happened, but saved some logs to try and have more of an idea.
After a reboot, things seem okay, but I don't know what triggered it. If it was a unique DoS. If it was related to syncing 3 Bitcoin nodes at once to diagnose a kernel issue. Or maybe a weird kernel bug.
I couldn't even login via TTY, so seems very odd.
It's been running a fairly out of date 7.8. I've been having good luck with 7.9 and decided to upgrade at the same time, in case there's a fluke issue going on.
Unfortunately, this means more downtime. I have a very remote camping trip going on and this seems like the safest path forward. I really don't know if it is, but I'm taking the gamble.
Right now the upgrade portion is started at 2026-09-02 02:09 UTC and took about 13 minutes. Then reboot was a few more minutes.
I'll update this more when I can. It's been a crazy day. I'm very sorry about this.
So network was totally down for about 20 minutes, but there were ongoing issues for several+ hours that may or may not have impacted you.
Everything should be back to normal now. Please let me know if not!