What Months of Running an Image Collector Taught Me
Published 2026-07-26 · SameOS Tools
I have been running a script that automatically gathers and organizes images for several months. It now holds 150,000 files across 136GB, and getting there took three significant rewrites. The 30-line script I started with looked fine, but within days the same photo had piled up dozens of times under different names, every run re-processed everything from scratch, and a single dropped connection left the whole thing frozen. This is the record of fixing those three things. Whatever you collect - tidying photos scattered across your own server, or pulling images from a public API - the lessons transfer.
Before the code - what are you collecting?
One thing before the technical part. What you collect must be yours, or publicly permitted. Sites publish a robots.txt (a file stating what automated collectors may access) and terms of service, and scraping anything behind a login is a terms violation almost everywhere. Collecting other people's photos raises copyright and likeness issues on top of that. The examples here assume organizing images already on your own server - gathering scattered photos into one place and removing duplicates, which anyone can do safely. If you use a public API, check its rate limits and terms first.
Problem 1 - filenames cannot catch duplicates
This one showed up first. The same photo saved as photo.jpg, photo(1).jpg, and IMG_2451.jpg will never be caught by comparing names - and identical names can hold different content. The answer is judging by content. Read every byte and compute a SHA-256 (a fingerprint that changes completely if a single bit differs), and identical content produces an identical fingerprint regardless of the name. I compute that fingerprint once per file and check new arrivals against the list.
# dedupe.py - find duplicate images by content
import hashlib
from pathlib import Path
EXTS = {".jpg", ".jpeg", ".png", ".webp", ".gif"}
def file_hash(path, chunk=1024 * 1024):
"""SHA-256 of file content, read 1MB at a time so large files are safe."""
h = hashlib.sha256()
with open(path, "rb") as f:
while True:
block = f.read(chunk)
if not block:
break
h.update(block)
return h.hexdigest()
def find_duplicates(folder):
"""Group as {fingerprint: [files]}, returning only groups with 2+."""
index = {}
for p in Path(folder).rglob("*"):
if p.is_file() and p.suffix.lower() in EXTS:
index.setdefault(file_hash(p), []).append(p)
return {h: files for h, files in index.items() if len(files) > 1}
if __name__ == "__main__":
for h, files in find_duplicates("./images").items():
keep, *drop = sorted(files, key=lambda p: len(str(p))) # keep the shortest path
print(f"[keep] {keep}")
for d in drop:
print(f" [dupe] {d}") # review before deleting - never delete inlineNote that last line. My first version deleted on sight, picked the wrong file to keep, and destroyed an original. Since then it only prints the list and leaves deletion to a human. Even with automatic deletion, moving files to a trash folder keeps the mistake reversible.
Problem 2 - stop re-processing everything
This one hurts more as the collection grows. Re-checking everything each run means yesterday's 10,000 files get checked again today, and the runtime climbs forever. The fix is simple: remember what you already handled and skip it. I store the fingerprints from step one in a small database (SQLite, a full database in a single file) and look up new arrivals by fingerprint alone. At 150,000 files, a fresh run still finishes in seconds. A plain text file works too, but lookups slow down past a few tens of thousands of lines.
# seen.py - remember processed files and skip them
import sqlite3
from pathlib import Path
from dedupe import file_hash, EXTS
db = sqlite3.connect("seen.db")
db.execute("CREATE TABLE IF NOT EXISTS seen (hash TEXT PRIMARY KEY, path TEXT, at TEXT)")
db.commit()
def is_new(path):
"""True if this content has not been seen before - and records it."""
h = file_hash(path)
row = db.execute("SELECT 1 FROM seen WHERE hash = ?", (h,)).fetchone()
if row:
return False # handled before - skip
db.execute("INSERT INTO seen (hash, path, at) VALUES (?, ?, datetime('now'))",
(h, str(path)))
db.commit()
return True
new_count = skip_count = 0
for p in Path("./images").rglob("*"):
if p.is_file() and p.suffix.lower() in EXTS:
if is_new(p):
new_count += 1 # do the real work here (copy, convert, ...)
else:
skip_count += 1
print(f"new {new_count} / skipped {skip_count}")Problem 3 - surviving failures
Anything that runs for months will meet failure: a brief network drop, a server that stops answering, one corrupt file. My first script simply died, so I would find it frozen in the morning having done nothing all night. Two things are needed. First, one failure must not stop everything. Second, retry with growing delays. Retrying instantly only burdens the other server and fails identically; waiting 2, then 4, then 8 seconds usually outlasts a temporary problem.
# fetch.py - keep going through failures, backing off between retries
import time
import requests
def fetch(url, tries=3):
"""Wait 2s, 4s, 8s between attempts. Returns None if it never succeeds."""
for i in range(tries):
try:
r = requests.get(url, timeout=15,
headers={"User-Agent": "my-collector/1.0"})
if r.status_code == 200:
return r.content
if r.status_code == 429: # rate limited - wait considerably longer
time.sleep(30)
continue
return None # 404 and friends: retrying is pointless
except requests.RequestException:
pass
time.sleep(2 ** (i + 1)) # 2s -> 4s -> 8s
return None
for url in ["https://example.com/a.jpg", "https://example.com/b.jpg"]:
data = fetch(url)
if data is None:
print(f"failed (skipping): {url}") # one failure, then carry on
continue
print(f"ok: {url} ({len(data):,} bytes)")
time.sleep(1) # spacing between requests - basic courtesyThings only 150,000 files taught me
Scale brought surprises. First, tens of thousands of files in one folder slow down file managers and shell commands alike - splitting into date-based subfolders fixed it outright. Second, format drives storage cost: converting archive-only images to WebP cut size substantially with no perceptible quality loss. Third, do not hold the whole fingerprint index in memory - 150,000 entries in a Python dict is a real memory footprint, and querying the database one row at a time (as above) is safer. Fourth, more important than any of it: back up first. A bug in a deduplication script deletes originals too.
Wrapping up
A collector meant to run for months needs three things: judge duplicates by content rather than name, remember what is already done and skip it, and never stop on a single failure. And before all three comes one question - "am I allowed to collect this?" What is technically possible and what is acceptable are different things, and there is plenty worth building on the right side of that line.