Why Bots Arrive Hours After You Register a New Domain
Published 2026-07-06 · SameOS Tools
I recently launched a site on a new domain. The address was published nowhere and search indexing was blocked - yet within hours, unfamiliar bots showed up in the access log, probing .env (a file that holds settings such as passwords and connection details), Dockerfile, and config file paths one by one. Classic vulnerability scanners. This article covers how bots discover new sites and what a personal server should prepare.
The Secret You Never Hid: Certificate Transparency Logs
The discovery path is simpler than you might think. The moment an HTTPS certificate is issued, it is recorded in Certificate Transparency logs - a public ledger anyone can subscribe to in real time. Scanners watch this ledger and visit new domains immediately. Assume any site with a certificate is discovered within hours, published or not.
What the scanners want is exposed secrets: .env files, backups, .git folders, setup scripts accidentally left in the web root. You cannot stop the probing; the only real defense is making sure there is nothing to find.
What a Personal Server Should Prepare
First, narrow static file serving to a whitelist (serving only files on an explicit allow list). Many server setups default to "serve any file that exists" - during my own audit I found a data file was downloadable, and switched to serving only the asset folders and a few named root files. Second, add a per-IP rate limit so overheated scans cut themselves off. Third, if you use a CDN (a relay network that answers requests on your server's behalf), enable its bot-blocking feature so much of the noise never reaches your server.
Practical example code
// PHP built-in server router: serve static files from a whitelist only
// (the "serve anything that exists" default is where accidents start)
$real = realpath(__DIR__ . $path); // neutralize ../ path traversal
if ($real !== false && is_file($real) && !str_ends_with($real, '.php')) {
$okRoot = in_array($path, ['/favicon.ico', '/robots.txt'], true)
&& dirname($real) === __DIR__;
$okAssets = str_starts_with($real, __DIR__ . '/assets/');
if ($okRoot || $okAssets) {
return false; // serve statically in these two cases only
}
}
// everything else goes to the router -> 404 (blocks data/log/backup exposure)
// Per-IP rate limit: 429 once the threshold is exceeded in a 5-minute window
$stmt = $db->prepare(
"SELECT COUNT(*) FROM visits
WHERE ip = ? AND ts > datetime('now', '-300 seconds')"
);
$stmt->execute([$clientIp]);
if ((int)$stmt->fetchColumn() > $limit) {
http_response_code(429);
header('Retry-After: 300');
exit('Too Many Requests');
}The key mindset: these scans are not a sign you are under attack - they are the ordinary background noise of the internet. Keep access logs so you can see what came by, and route anomalies to a webhook notification (a dedicated URL that lets a program post messages into your messenger channel) for peace of mind. Hiding private files behind an authenticated router is covered in a separate article.