Home Server Auto-Recovery, Part 1: Make the Kernel Reboot Itself After a Crash
On a Sunday evening I noticed my site was down - and by then the server had already been frozen for three and a half hours. Power on, fans spinning, but no screen and no network, so I ended up holding the power button. The same thing happened again before dawn. This series is the record of making a home server come back on its own after that night.
Finding the Cause - the Boot Log Remembers Everything
When a server freezes and gets power-cycled, the first step is finding out when and why it died. On a systemd Linux, journalctl (the built-in command that collects system logs) keeps records per boot, so two commands show the exact time of death and the last messages before it.
# List boots - the end time of each line is the moment it died
journalctl --list-boots
# Last messages of the previous boot (-b -1 = one boot back, -e = jump to the end)
journalctl -b -1 -eIn my case the culprit was a USB network adapter. Its driver (r8152) hit a transmit queue timeout that escalated into a kernel error - and since the onboard port is broken, swapping hardware was not an option. But the logs revealed the real problem: the crash itself took a moment, while most of the 3.5-hour outage was simply the dead server waiting for a human to show up.
The Linux Default Is "Stay Dead Forever"
When the kernel panics (hits an unrecoverable fatal error), Linux by default stops right there and never reboots (kernel.panic=0). In a datacenter someone soon walks over to the console; a server at home just stays dead. One config file changes that default.
# /etc/sysctl.d/99-autoreboot.conf
# Reboot automatically 10 seconds after a panic (default 0 = stay frozen forever)
kernel.panic = 10
# Treat an oops (fatal kernel error) as a panic
kernel.panic_on_oops = 1
# Also turn full freezes (lockups) that leave no logs into a panic
kernel.hardlockup_panic = 1
kernel.softlockup_panic = 1
# Apply: sudo sysctl --system
# Check: sysctl kernel.panic <- 10 means it workedAt first I only added the top two lines. The very next night a different failure arrived: the machine froze solid without writing a single log line (a hard lockup), and I was back at the power button. That is what the bottom two lines cover - the watchdog built into the kernel notices a CPU has been unresponsive for seconds and forces a panic, feeding it into the auto-reboot path above.
July 30, 1:39 AM - the Setting Does Its Job
Then, at 1:39 AM on July 30, the kernel died again. The last log line reads "traps: PANIC: double fault". The boot list shows the server switched itself back on 35 seconds later. I was asleep and only learned about the crash from the morning logs. The same class of incident went from a 3.5-hour outage to 35 seconds.
One thing has to pair with this: an automatic reboot only helps if your services start on boot, so make sure your web or game servers are enabled in systemd. And this setting only covers failures where the kernel dies. On the days the server is fine but the network adapter silently drops off, there is no panic and therefore no reboot. Part 2 covers the watchdog script that handles exactly that - checking itself every minute and recovering in stages.