Designed for Datasets of Any Size

This project is designed to work efficiently with both small datasets and extremely large collections, scaling up to datasets containing billions of files.

Its goal is to enable dataset analysis and optimization at any scale. Removing duplicate and redundant data improves the overall quality of a dataset, provides a more accurate representation of the data available for training, and reduces unexpected issues during model training.

The --stream mode prioritizes minimal memory usage, making it suitable for datasets that exceed the available system memory. As a trade-off, some operations—such as duplicate detection—can become significantly slower on very large collections.

When --stream is not used, performance is generally much higher while still maintaining a very small memory footprint. In my tests, processing datasets containing tens of thousands of images required only about 4 MiB of RAM.

In summary:

• Small datasets: Fast processing with low memory usage.
• Large datasets: High performance while maintaining a very small memory footprint.
• Massive datasets: Supported through --stream, at the cost of longer execution times for certain operations.

Install

pip install git+https://github.com/JuanForge/datasetforge.git

Examples commands

1 - duplicates

Cache

datasetforge duplicates \
    --input-cache /data/0.0.0 \
    --phash-live \
    --phash-max-percent 10 \
    --phash-min-percent 2

No cache

datasetforge duplicates \
    --input /datasets/0 /datasets/1 \
    --phash-live \
    --phash-max-percent 10 \
    --phash-min-percent 2

note

For the best performance, I recommend using cache mode whenever possible.

To make the CLI more intuitive, I also renamed --input to --input-cache, as it now explicitly refers to the cache directory (matching the previous behavior). Meanwhile, --input can now be used to specify dataset directories directly, which should be simpler and more intuitive for new users.

---

The duplicates subcommand is used to analyze a cache previously generated by the index subcommand.

• --input-cache specifies the directory containing the generated cache to analyze.
• --phash-live enables live pHash filtering mode. This mode allows the use of pHash-specific filtering options instead of displaying only the top-k results.
• --phash-max-percent 10 sets the maximum allowed pHash difference. Image pairs with a pHash difference greater than 10% will not be displayed.
• --phash-min-percent 2 is optional and defaults to 0. Setting it to 2 hides image pairs with a pHash difference lower than 2%.

With these settings, the command will only display image pairs whose pHash difference is between 2% and 10%.

This range is useful for detecting images that are visually similar but have undergone strong recompression, format conversion, resizing, or quality degradation.

For general duplicate detection, it is recommended to keep --phash-min-percent at its default value (0) or omit it entirely. This allows all pHash differences up to the value defined by --phash-max-percent to be detected, including exact or near-exact duplicates.

Comparison

• n = dataset size (number of indexed images)
• k = --top-k value

Mode	Comparisons	Disk reads	Memory usage
Default	$\frac{n(n-1)}{2}$	O(n)	O(n + k)
--stream	$n(n-1)$	O(n²)	O(k)

--stream is useful for very large datasets where memory is limited, but it can be significantly slower because files must be read multiple times.

pHash Explained Simply

The similarity percentage reported by the tool should be interpreted with care. Perceptual hashing (pHash) does not compare images the way a human does, especially when using different hash sizes.

In practice, using a larger value for --phash-bits (for example, 262144) allows the algorithm to preserve significantly more visual information. As a result, pHash can perform a much finer comparison, often producing substantially higher similarity scores than smaller hash sizes such as 1024 or 4096.

Conversely, a smaller --phash-bits value effectively reduces the amount of visual information available to the algorithm. With fewer features to compare, similarity scores may be noticeably lower.

Therefore, it is completely normal for the same pair of images to receive much higher similarity percentages—sometimes nearly twice as high in the top-k results—when using a larger --phash-bits value. This does not necessarily mean the images have become more similar; rather, the higher-resolution perceptual hash enables a more precise comparison.

Command-Line Arguments

--phash-bits

Controls the resolution of the perceptual hash used during comparison.

• Default: 4096
• Higher values preserve more visual information, allowing finer comparisons.
• Lower values are faster and use less memory but may miss subtle similarities.

Examples:

datasetforge duplicates --phash-bits 4096
datasetforge duplicates --phash-bits 262144

Note Higher values often produce higher similarity percentages because the algorithm has more information available for comparison. This does not mean the images themselves are more similar.

--input ( duplicates )

Specifies one or more dataset directories. Indexes are generated automatically, making this suitable for small datasets where pre-indexing is unnecessary.

WARNING

Warning

Experimental Project

This project is currently under active development and should be considered experimental. Breaking changes, bugs, and unexpected behavior may occur between releases.

At this time, the project has only been tested on Linux. While some features may work on other operating systems, compatibility is not guaranteed until they have been officially tested and validated.

If you encounter any issues, please report them through the Issues page.