SameOS ~/tools/python-duplicate-image-cleanup.md

한국어 English

🌙☀️

Find Duplicate Images With Python and Print Delete Candidates

Image folders often end up holding the same photo under different names. Instead of trusting filenames, this guide finds identical files with a SHA-256 hash (a short fingerprint of a file's contents - same contents, same value) and prints candidates to review before anything is deleted. Save the code below as a file and run it from a terminal (the command-line window) with python3 script_name.py.

Cleanup flow

• Limit work to image extensions.
• Read files in 1 MB chunks and calculate SHA-256 hashes (low memory use even for large files).
• Remember the first path for each new hash.
• Print later files with the same hash as duplicate candidates.

Practical example code

from pathlib import Path
import hashlib

IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".gif"}

def sha256_file(path: Path) -> str:
    # Read the file in 1 MB chunks and compute a SHA-256 hash (content fingerprint)
    digest = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()

def duplicate_candidates(root: str):
    seen = {}
    for path in Path(root).rglob("*"):
        if not path.is_file() or path.suffix.lower() not in IMAGE_EXTS:
            continue  # check image extensions only
        digest = sha256_file(path)
        if digest in seen:
            print(f"duplicate: {path}")
            print(f"original : {seen[digest]}")
            print(f"hash     : {digest}\n")
        else:
            seen[digest] = path  # remember the first hash as the original

duplicate_candidates("./images")

What to check before deleting

When hashes match, the file bytes match. Still, before automatic deletion, check backups, thumbnail files, and any related database records.