The problem
- Problem: Videos often contain many duplicate and near-duplicate frames. If too many of them reach training, common scenes become overrepresented, the dataset distribution shifts, and the model has a greater risk of overfitting.
- Goal: Create a smaller set containing only training-relevant frames that are as diverse as possible while still representing the source videos.
- Solution: First sample every frame from the videos. Then index the resulting 2,400-frame pool with CLIP and select 1,000 diverse frames. Inspect the selected set with Similar pairs before deciding whether any repetition warrants further review. A benchmark against 30 equal-size random samples measures the selection’s redundancy and coverage.
Results at a glance
An overview of the comma2k19 dataset
The source is comma2k19, an open-source collection of road-facing driving logs released under the MIT license. For this case study, we used a small subset: two one-minute clips recorded at 20 fps, containing 2,400 frames in total.
We sampled every frame from these clips to create the 2,400-image candidate pool. The fixed camera and highway setting make this a demanding test: many frames look similar even after the initial sampling step.
For a fair comparison, both selection methods use the same 2,400-frame candidate pool and return exactly 1,000 images. The diverse result is compared with 30 random samples generated from seeds 0 through 29.
Step-by-step process
Let’s take a look at what the process involved.
| Step | What happens | What the step decides |
|---|---|---|
| 1. Extract | Sample every frame from both source videos | Which images enter the candidate pool? |
| 2. AI Index Embeddings | Build a normalized CLIP representation for all 2,400 images | How can visual similarity be measured consistently? |
| 3. AI Search | Use Diverse mode to keep 1,000 representatives from the indexed pool | Which frames add coverage instead of another repeated view? |
| 4. Similar pairs | Inspect a parallel report of similar pairs within the selected frames | Where does repetition remain after selection? |
| 5. Hand off | Send the selected dataset to the configured labeling stage | Which images consume annotation capacity? |
Inspecting repetition after selection
Similar pairs checks the 1,000 selected frames using CLIP cosine similarity ≥0.98. Its report contains 5,775 unique pairs and collects 839 candidate images for review. The report displays 500 pair rows; that display limit does not reduce the full pair count.
The inspection runs alongside the labeling handoff. It does not remove frames automatically or make labeling wait for a review decision. Sampling, AI Search in Diverse mode, and Similar pairs need no annotations.
Candidate groups can connect frames through intermediate matches; they do not guarantee that every member is similar to every other member. Review the frames before choosing any exclusions.
The pair report makes residual repetition inspectable. The selection benchmark below separately compares redundancy and omitted-frame coverage against 30 random samples; Similar pairs does not perform that baseline or coverage analysis.
Diverse vs. random sampling
The selection benchmark compares diverse sampling with ordinary random sampling. We created 30 random sets of 1,000 frames and averaged their results. Two frames counted as a near-duplicate pair when their CLIP cosine similarity was 0.98 or higher, meaning they showed almost the same visual content.
The benchmark analysis counted 5,774 near-duplicate pairs, compared with 7,335 pairs in the average random set — 21.3% fewer. It outperformed all 30 random trials. Against the most repetitive individual random set, which contained 8,238 pairs, diverse sampling reduced repetition by 29.9%. In practical terms, the same 1,000-frame labeling set has fewer repeated views and more room for frames that add new visual information.
How to read a similar-pair count
A pair count is not a count of duplicate images or a recommendation to remove every flagged frame. A 1,000-image set contains 499,500 possible pairs; about 5.8k, or 1.16%, crossed the near-duplicate threshold. The same image can belong to many flagged pairs, and visually close frames can still capture changes needed to represent the source broadly. The useful signal is the fair comparison — diverse sampling produced fewer near-duplicate pairs than equal-size random samples while preserving better coverage of the original videos.
The selected set still represents what it leaves out
For model training to benefit from a smaller dataset, the selected subset still needs to represent the source. Otherwise, it can create blind spots or miss entire areas of the source data. We checked this in the selection benchmark by matching every omitted frame to the most visually similar frame among the 1,000 selected images, then comparing those matches with the 30 random baselines. A closer match means the selected set contains a better stand-in for the frame that was left out.
| Which omitted frames are being checked? | Match in the diverse set versus random sampling |
|---|---|
| All omitted frames, on average | 9.4% closer |
| Harder-to-match end of the set | 15.5% closer |
The diverse set represented the source more closely instead of gaining variety by ignoring less common scenes. Across all omitted frames, the nearest kept frame was 9.4% closer than with random sampling. In the harder-to-match part of the pool, the improvement reached 15.5%, showing that even less common visual areas remained better represented.
What the frames look like
One seed-0 random sample included ten neighboring candidates from the same clip. The three below are frames 123–125 from that run and show how visually similar images can still survive random sampling.
The next three images are actual members of the diverse set from different points in that clip. They are an illustration of the delivered dataset, not a substitute for the 30-trial measurements.
Summary
AI Index Embeddings and AI Search reduce the 2,400-frame candidate pool to 1,000 visually diverse frames. Similar pairs makes the remaining repetition inspectable: roughly 5.8k similar pairs and 839 candidate images for review, with no automatic removals.
The selection benchmark against 30 random samples found 21.3% fewer near-duplicate pairs while representing the source more closely, including a 15.5% improvement in the harder-to-match part of the pool.
The selected dataset goes to labeling with its size intact. You can inspect the pair report separately before deciding whether any candidate frames should be excluded.