Posted: 2026-05-26
On May 7th, I announced an upcoming maintenance on May 22nd from 22:00-23:59 UTC. The maintenance happened and I'm just now writing about it. Unfortunately, it did not all go according to plan and went late.
If you haven't seen it already and you're curious, you can see the details and log written as it happened. There's some details missing to try and speed up the process.
The maintenance was scheduled for about two hours but took a bit over four. Now things were not all down the whole time. SporeStack was readonly for a long time as I sorted through things. But it was close to four hours of mixed connectivity. As such, I'm crediting all existing Slow Servers customers an extra two months. I'm sorry that it was down longer than expected! Or down at all -- generally the expectation is that servers stay online.
Ardent Networks supplied two network drops, from two core routers. The plan was to setup BGP on both of my routers to failover between them. Unfortunately, I've never worked with BGP before. One concern was that BGP tables may not be updated, so there could be a 24 hour partial outage window if the switch wasn't done smoothly.
My other concern involved the complexity of BGP being a technology new to me, that I might screw something up. Even if the hardware setup should be more reliable with no single point of failure, I'm very familiar with how complexity and mismanagement can cause even more downtime than hardware failure, over a long enough timeline.
BGP is also pretty unnecessary because there's only the one provider. If I had multiple ISPs, then BGP is pretty much mandatory. But not much benefit working with one provider. I decided, fairly last minute, to do CARP on both the WAN side and LAN side, and use the single uplink. Unfortunately, this means that on Ardent Network's side, a core router update or failure would bring things down. But it still should let me take a router offline for updates, which is going to be more frequent running OpenBSD than a very specific router OS.
Another last minute design change was not doing dual ethernet switches and bridging on the VM hosts. It could still be worth doing, but unmanaged switch failures are so incredibly rare. It really seems like one of those redundancies that may never come into play.
Unfortunately, CARP did not work perfectly. One of the biggest issues was that one CARP interface kept going live in a dual master setup, causing all kinds of trouble. The WAN CARP was fine, my test CARP was fine, but the internal CARP would end up dual master. I still don't know why this is. Thus, the setup isn't fully redundant yet. Otherwise, it was working pretty well with failover testing. I wasn't seeing any lost packets after rebooting Freya, and going back and forth (until Freya+Thor were both partly in action.) This is something that I need to follow up on.
I also wrote PF rules on the fly, and this took a while. I kept iterating trying to do antispoofing. PF's native antispoof feature is great, but it has some limitations that make it quickly unsuitable. But this alone probably took 30 minutes to an hour.
I never did finish installing sniproxy and some other services on Thor. Nor did I have a mechanism in place for syncing configuration.
When Thor's alarm went off, I was pretty concerned and not sure what to do. I decided to diagnose it rather than pull it, and there was no obvious cause. The alarm was quite random, I think it had been going off before I got there, and at the same time could be hard to reproduce. I eventually found that downclocking the processor (setperf=1) seemed to make it go away, but I'm not 100% confident. It doesn't actually seem to be thermal, so not much makes sense. PS voltages looked normal in the BIOS, spread spectrum or not didn't seem to matter. And no kind of dmesg or other messages to give an indication. This is another thing that I need to follow up on.
I didn't realize that the meter had a maximum of 2 hours that I could load at any one time. I ended up topping off the meter twice, which didn't help.
I should have tested the CARP setup in a lab environment. Part of the reason that I didn't is that I was planning on doing BGP, and I wasn't sure I could replicate that setup very well or easily. I had done basic CARP testing and moved mirror.ssvr.net over on CARP. I think the CARP issue I was bitten by might've happened either way in the live environment, but I am not sure.
I also should have written+tested PF rules in a lab and had them ready to go, or close to it. That would have shaved off a lot of time.
I should have parked in the paid parking lot and not on the street for such a maintenance.
Timing it for Friday of Memorial Day weekend wasn't great, but it's when it worked out. I guess it could've been better to hold off. I just don't like extra trips to the area if I don't have to.
I'm sorry for running late on this, as a combination of poor planning and bad luck. I have some things to follow up on. This does mean that the router infrastructure isn't redundant like planned and not much improved, however some of the leg work has been done and perhaps the fix will be somewhat straightforward. We will see...