Case · CLI tool · Rust

dupfind: fast duplicate file finder

A Rust CLI that finds duplicate files. Files are grouped by size, then by a partial hash (the first and last 4 KiB) and only then by a full BLAKE3; walking and hashing are parallel. By default it only shows the result; deleting, hard-linking and moving are enabled with --apply and re-check the hash before acting. Hard links to one file are not counted as duplicates, symlinks are not followed.

  • Rust
  • BLAKE3
  • rayon
  • proptest
  • CLI
  • GitHub Actions
  • Docker
Code on request via Telegram

01 · How it looks

Screenshots

Screenshot of the project "dupfind: fast duplicate file finder" at 1280 pixels wide
Desktop, 1280 px
Screenshot of the project "dupfind: fast duplicate file finder" on a phone screen, 390 pixels wide
Phone, 390 px

02 · About

What it is and why

The task: find duplicate files quickly and safely: delete nothing extra and never mistake a hard link for a copy.

The solution: Rust. Grouping by size, then BLAKE3 of the first and last 4 KiB and only then a full BLAKE3; walking and hashing are parallel (rayon). By default it only shows; --apply enables actions and the hash is recomputed before them.

Limitations

It was not run on Windows by hand: build, clippy and tests pass there in CI, hard-link and symlink tests are disabled.

There is no byte-by-byte compare: reliability rests on BLAKE3, the hash is recomputed before acting.

Permissions and extended attributes of the replaced copy are lost with --hardlink.

Measurements are on demo data, Docker Desktop, a warm cache: not a benchmark.

What is inside

  • Paths with the same device+inode collapse into one physical file and are not counted as duplicates; symlinks are neither opened nor followed, so loops are impossible
  • --delete-newer, --hardlink, --move-to modes, text, json, ndjson formats, a path list from stdin (including NUL), --exit-code for CI
  • A demo directory of 10,000 files (307 MB, a third are copies): about 120 ms on 20 threads (116 to 133 over three runs), about 200 ms on one thread; find + sha256sum + sort + uniq takes about 1130 ms
  • A bug found and fixed: a 256 KiB read buffer was allocated per file and 20 threads ran slower than one (about 520 ms against 270 ms); now one buffer per thread
  • 18 integration tests on temporary directories and proptest: groups are exactly the classes of identical content
  • Release binaries for Windows, Linux and macOS are built by a workflow as artifacts

03 · How to run

Run it locally

git clone <repository on request> dupfind
cd dupfind
cargo build --release
./target/release/dupfind ~/Downloads --exclude '**/node_modules' -m 1K

04 · Code

The repository is private

The source is in a private repository. Message me on Telegram and I will open access.

Request the code