Home Server Auto-Recovery, Part 2: When the Kernel Is Fine but the Network Is Dead
Part 1 covered making the kernel reboot itself after a crash. That approach has one hole. The most frequent failure on my server is the USB network adapter silently dying - and in that case the kernel stays perfectly healthy, so there is no panic and no reboot. The server keeps running, yet from the outside it looks dead, and no alert can leave the box because the network is the thing that broke. This part is about the watchdog (a monitor that checks periodically and steps in when something is wrong) that covers exactly this case.
Reboot Is the Last Resort - Least Painful First
The idea: the server checks on itself every minute. It pings the router (gateway), and if there is no answer it declares a failure and tries fixes from least painful to most. Stage 1 is a software unplug-replug of the USB adapter (a rebind) - if it works, the outage ends in about 30 seconds with no reboot. Stage 2 reloads the driver (the kernel program that operates the adapter) - about a one-minute outage. Stage 3, a reboot, runs only when both failed.
The safety rails matter as much as the order. For the first 3 minutes after boot the check is suspended - the network may still be coming up. And if the machine booted less than 15 minutes ago, stage 3 is held off; without that guard, a failure that a reboot cannot fix turns into an infinite reboot loop. After recovery the script forces a DHCP renew (the router handing out IPs automatically), because bringing the link up alone can leave the interface attached but address-less. Notifications go out only after recovery - during the outage they could not be delivered anyway.
#!/bin/bash
# Runs every minute. If the gateway stops answering, recover in stages. (Trimmed)
NIC="eth0" # your NIC name (check with: ip link)
DRIVER="r8152" # your NIC driver (check with: ethtool -i eth0)
GW=$(ip route | awk '/^default/{print $3; exit}')
alive() { ping -c3 -W2 "$GW" >/dev/null 2>&1; }
# Grace period: for 3 minutes after boot the network may still be coming up
[ "$(awk '{print int($1)}' /proc/uptime)" -lt 180 ] && exit 0
alive && exit 0
# Stage 1: USB rebind (~30s outage if it works, no reboot)
USBID=$(basename "$(readlink -f "/sys/class/net/$NIC/device")")
echo "$USBID" > "/sys/bus/usb/drivers/$DRIVER/unbind"; sleep 3
echo "$USBID" > "/sys/bus/usb/drivers/$DRIVER/bind"; sleep 15
networkctl reconfigure "$NIC" # force a DHCP renew
alive && exit 0
# Stage 2: reload the driver (~1min outage)
modprobe -r "$DRIVER"; sleep 3; modprobe "$DRIVER"; sleep 20
networkctl reconfigure "$NIC"
alive && exit 0
# Stage 3: reboot - held off if we booted <15 min ago (reboot-loop guard)
[ "$(awk '{print int($1)}' /proc/uptime)" -lt 900 ] && exit 0
systemctl reboot --forceRunning It Every Minute - a systemd Timer
The listing above is trimmed to the skeleton; the full script - with logging, a post-recovery webhook alert, and IP-change detection - is on GitHub. For scheduling, cron works too, but I used a systemd timer: two small files, enable the timer, and you get run history and failures visible through systemctl.
# /etc/systemd/system/net-watchdog.service
[Unit]
Description=Network watchdog
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/net_watchdog.sh
# /etc/systemd/system/net-watchdog.timer
[Unit]
Description=Run network watchdog every minute
[Timer]
OnCalendar=*:*:00
[Install]
WantedBy=timers.target
# Apply
# systemctl daemon-reload && systemctl enable --now net-watchdog.timerTwo Weeks In - Zero Deployments Is Also a Result
Two weeks after installing it, this watchdog has not had to act once. Credit goes partly to the driver mitigations applied alongside it (disabling transmit offloads and USB autosuspend), which made the adapter freeze far rarer in the first place. Like any insurance, never needing it is the best outcome - and on the day it is needed, recovery starts within a minute. Meanwhile, the kernel-death class of failure did strike again in the same period, and the Part 1 auto-reboot caught it.
That completes a division of labor by failure type: if the kernel dies, the Part 1 sysctl settings reboot the machine; if only the network dies, this watchdog revives it in stages. One hole remains - the day the hardware cannot get up at all. The next article covers a free cloud standby (failover) setup that keeps the site reachable even when the server will not power on.