Posted: 2026-06-09
Slow Servers had some issues with IPv4 today. I am not completely sure what caused it. It appears to be some type of routing table issue on Freya.
I would estimate the window around 16:00-18:30, but I am not exactly sure. I added ICMPv6+ICMPv4 monitoring of two VPSs on two different hosts to status.slowservers.net, to help catch cases like this (and track durations.)
Freya had messages like this:
arpresolve: 172.17.0.3: route contains no arp information
172.17.0.3 is Loki's private IPv4 address used for routing purposes.
When this was happening, I was unable to ping from Freya. Packets from guests would be sent out, but inbound packets made it into Freya's WAN interface, but did not exit Freya's LAN interface.
I could reach 172.17.0.3 without any issue. It seemed to impact some IPv4 guests but not all at the same time.
Routing configuration looks like this from the Freya side of things:
slowservers on spare
!route add -inet 142.249.45.100/32 172.17.0.99debtestloki on loki
!route add -inet 142.249.45.101/32 172.17.0.3
And on spare.slowservers.net, for example. /etc/hostname.tap1:
inet6 2602:f5ef:0:ff01::/127 inet 172.17.99.1/32 group vmgroup up !route add 2602:f5ef:0:ff01::/64 2602:f5ef:0:ff01::1 !route add -inet 142.249.45.100/32 -llinfo -link -iface tap1
I'm not sure why this is as delicate as it is. If anyone has any ideas, please let me know!
It went down again! Here's some monitoring output.
2026-06-09T19:19:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:19:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:20:03Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:20:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:21:03Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:21:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:22:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:22:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:23:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:23:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:24:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:24:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:25:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:25:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:26:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:26:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:27:04Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:27:05Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:28:03Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:28:04Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:29:03Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:29:04Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:30:03Z DOWN: 142.249.45.101 is unreachable over icmp. 2026-06-09T19:30:04Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:30:06Z DOWN: tor+gemini://slowsvr7rgnkzuoma5byw34n2mkkg6v7g5q6qxqaihh7zikp5wn26fid.onion Tor Error: Protocol error 2026-06-09T19:31:03Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:32:03Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:33:03Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:34:03Z DOWN: 142.249.45.100 is unreachable over icmp. 2026-06-09T19:35:03Z DOWN: 142.249.45.100 is unreachable over icmp.
I made a pretty big change that I'm not 100% confident in, but so far seems to be stable.
I removed the static routes on Freya, and replaced them with this:
slowservers on spare
#!route add -inet 142.249.45.100/32 172.17.0.99 !arp -F -s 142.249.45.100 (MAC address of spare)debtestloki on loki
#!route add -inet 142.249.45.101/32 172.17.0.3 !arp -F -s 142.249.45.101 (MAC address of loki)
The -F flag is given because it won't add the entry if there's already been ARP traffic for that IP. In retrospect, it seems like adding "permanent" might be a good idea.