The problem
- Problem: Large image datasets mix useful examples with irrelevant scenes and near-duplicates. Reviewing them manually consumes hours of specialist time and labeling budget.
- Goal: Find images that match the task while preserving broad visual coverage and minimizing repeated content in the dataset sent to review, labeling, or training.
- Solution: Use AI Index Embeddings to prepare CLIP representations, AI Search to find relevant images and select a diverse subset, and Similar pairs to inspect remaining repetition.
Workflow at a glance
- Starting point24,999 imagesBDD and KITTI road scenes
- Step 1 · Relevance3,922 imagesPrompt-matched urban truck scenarios
- Step 2 · Diversity1,250 imagesRepresentatives selected by AI Search
- Step 3 · Inspect133 pairsSimilar pairs supplies 92 removal-review candidates
The starting point
Source datasets
BDD100K 10K contributes 10,000 images. KITTI Object Detection contributes another 14,999. Both public datasets are imported from Dataset Ninja; after merging, the indexed dataset contains 24,999 road-scene images.
What is CLIP indexing?
CLIP turns images and text into numerical representations and places related visual and written content close together. AI Index Embeddings stores these representations for reuse by the search and inspection stages. This workflow uses CLIP explicitly throughout:
- AI Search with a text prompt: searches the index with a written request and keeps relevant images.
- AI Search in Diverse mode: groups the matches and selects images closest to the group centers.
- Similar pairs: finds highly similar image pairs within that selection and supplies candidates for removal review.
How the workflow works
| Step | What happens | What the step decides |
|---|---|---|
| 1. Combine | Merge BDD and KITTI into one image dataset | Which images are available to search? |
| 2. AI Index Embeddings | Prepare CLIP representations for the merged pool | How can text and images be compared? |
| 3. AI Search · Text prompt | Rank the indexed dataset by text-to-image similarity | Which images are relevant enough to consider? |
| 4. AI Search · Diverse | Select representatives from visual groups in the matches | Which candidates add variety instead of repetition? |
| 5. Save the result | Write the selected images to the curated dataset | Which dataset moves to review, labeling, or training? |
| 6. Similar pairs | Inspect a separate branch from the diverse selection | Which remaining pairs need a removal decision? |
Run settings: AI Search used the prompt large trucks on an urban road and a confidence
threshold of 0.27. The 10,000-image ceiling did not limit the search because 3,922 images matched.
Diverse sampling retained 1,250 of those matches.
Two-stage curation: relevance first, diversity second
Each stage solves a different problem. The first removes scenes that do not match the task. The second reduces repetition inside that relevant pool while preserving its visual range. We tuned the two filters together over a few short iterations; tests with 2,000 and 3,000 final images retained more near-duplicates, while 1,250 produced the better tradeoff.
1. Filter for relevant scenarios
AI Search compares the written request with every indexed image. The configured query returned 3,922 matches. Because that result stayed below the top-k ceiling, the similarity threshold decided which images moved forward.
- Input24,999 imagesIndexed BDD and KITTI road scenes
- AI SearchPrompt + similarity thresholdRank each image by text-to-image similarity
- Output3,922 imagesRelevant matches passed to step 2
2. Select a diverse subset
From the 3,922 relevant matches, diverse sampling selected a 1,250-image subset, reducing the pool by roughly 3×. It grouped the matches by their CLIP representations and kept the images closest to representative group centers. This favors visual variety but does not guarantee that every near-duplicate is excluded.
- Input3,922 imagesRelevant prompt matches
- Diverse samplingCluster + pick representativesKeep 1,250 images nearest the group centers
- Output1,250 imagesA compact, visually varied candidate set
To test that result, we compared diverse sampling with 20 random samples of the same 1,250-image size and averaged the random results. For every image, the evaluation finds its maximum CLIP similarity to another image in the same set; a score of at least 0.98 counts as a near-duplicate. The diverse set had a lower near-duplicate rate in all 20 comparisons. This measures repetition at the chosen threshold; it does not establish coverage of every useful scene. The results are below.
| Evaluation set | Images with a ≥0.98-similar neighbor | Rate | Role in the comparison |
|---|---|---|---|
| Text-prompt output | 1,264 / 3,922 | 32.23% | Input before diverse sampling |
| Diverse sample | 150 / 1,250 | 12.00% | Final curated set |
| Random baseline, mean of 20 samples | 222 / 1,250 | 17.76% | Fair same-size baseline; range 14.16–20.08% |
The final near-duplicate rate is 5.76 percentage points below the random average. In relative terms, that is a 32.42% reduction. It also falls below the best random result, so diverse sampling won all 20 comparisons.
Inspect the remaining similar pairs
Similar pairs checks the 1,250-image diversity selection using its CLIP embeddings and a cosine similarity threshold of at least 0.98. It produces a report and a separate dataset of candidates for removal review. The report is publicly available in the accompanying live example.
| Similar pairs inspection | Result |
|---|---|
| Images checked | 1,250 |
| Unique pairs at CLIP cosine similarity ≥0.98 | 133 |
| Images with a similar neighbor | 150 (12%) |
| Candidate images for removal review | 92 |
| Pair rows displayed | 133 of 133 |
The 500-row display limit did not truncate this report. No images were automatically excluded: the curated selection still contains 1,250 images. You inspect candidates before deciding whether to exclude any; later exclusions reduce the selection size. Groups can connect transitively: if A resembles B and B resembles C, all three can belong to one group even when A and C fall below the threshold. Group membership therefore does not prove that every candidate can safely be removed.
AI Index Embeddings runs on a schedule. You initiate the AI Search workflow and start Similar pairs separately after diversity selection finishes. The baseline evaluation counts images with a similar neighbor; the report also counts pairs, while its candidate dataset is a further selection for review. These three quantities cannot be substituted for one another. The comparison with 20 random samples is a separate evaluation, not an output of Similar pairs.
Visual evidence
The illustration below shows the sampling logic: keep one representative from a visual group instead of filling the result with several near-duplicates.
Summary
AI Search retained 3,922 relevant matches from 24,999 source images and selected a diverse dataset of 1,250 images, reducing review volume 20×. At the 0.98 CLIP threshold, 150 selected images (12%) have a similar neighbor. That rate is 5.76 percentage points below the random average, a 32.42% relative reduction, and lower than all 20 same-size random samples.
Similar pairs exposes 133 pairs in that selection and supplies 92 candidate images for removal review. The curated dataset remains intact until you decide which candidates to exclude. OvalBee keeps the selection and pair evidence together for that review; these results do not establish that all duplicates are removed or that the selection improves a downstream model.