SameOS ~/tools/hash-checker.md

Find Duplicate Images and Files With SHA-256 Hashes

Even when filenames, dates, and extensions differ, files are identical if their byte content is the same. Comparing hash values (a short fingerprint computed from a file's entire contents) spots duplicates reliably no matter what the files are called.

In image folders, the same photo can also exist in different formats such as JPG and WEBP. Those files have different bytes, so their hashes differ. If the same file was saved under another name, the hash stays the same.

Operation Tip

For large folders, narrow candidates by file size first, then compare hashes only among files with the same size. Before deleting anything, confirm that original folders, thumbnail folders, and backups are handled together.

How One Line-Feed Byte Changes a Hash

A trailing line feed is part of a file. These two inputs look similar on screen but produce completely different results.

sameos + LF, 7 bytes

Example: printf 'sameos\n' | sha256sum
Result: adfe9ec4e80b11fe6466e663e37a9037947e177f1007fb168c1d4b6306d7d9e7

Files containing the same seven bytes produce this value even when their names differ.

sameos without LF, 6 bytes

Example: printf 'sameos' | sha256sum
Result: a4d76f82fff5ec89c9a0135a5f0ac56f0b53c3408f3ac1de2db5b0670b4b6408

Removing only the final byte changes the entire digest.

Finding Duplicates From the Command Line

For thousands of files, one command beats the browser. Same content means same hash.

# sha256sum ships with Linux/macOS by default (Windows: step 4)

# 1) hash every file in the folder, print only duplicates
sha256sum * | sort | uniq -w64 -D
#    -w64  → compare only the first 64 chars (ignore filename)
#    -D    → show duplicates only

# 2) reading it — two lines sharing the first 64 chars = same content
# e3b0c44298fc1c14...  photo.jpg
# e3b0c44298fc1c14...  photo_copy.jpg
#    different names, identical content — keep just one

# 3) include every subfolder
find . -type f -exec sha256sum {} + | sort | uniq -w64 -D

# 4) on Windows (PowerShell)
Get-FileHash *.jpg -Algorithm SHA256 | Group-Object Hash | Where-Object Count -gt 1