Before you spend on labeling or training
Before a labeling budget or a training run goes into a dataset, you want to know what is broken in it, whether its splits leak into each other, whether anything in it is anomalous, and what is already fine. This study gets those answers for one public detection dataset in a single run.
The run passes the data through eight checks: dataset statistics, image quality, annotation quality, a near-duplicate search, an outlier search, label diagnostics, a missing-box search, and a trial prelabeling model. Each check keeps its own detailed report. A summary report reads them all together and gives two things: findings ranked by severity, and a next step for each of them.
The dataset
PCB Component Detection was published on Kaggle by Animeshkumar Nayak in 2021 under CC0 and is mirrored on Dataset Ninja, where this run imported it by link. It holds 1,410 images and 11,119 bounding boxes across nine classes: 1,099 training images, 200 validation images, and 111 test images that carry no labels by design.
It has the defects public datasets often carry: a misspelled class, heavy visual repetition, and a test split that cannot be scored. That makes it a useful test. The question is not whether the checks fire, but whether the summary tells you which of them matter.
What the audit found
The summary report opens with a findings board: 16 findings grouped into 10 rows, most severe first.
| Level | Finding | Next step |
|---|---|---|
| Warning | Resestor is a 3-object Resistor spelling that train never sees (Dataset statistics, annotation quality) | Rename the 3 objects to Resistor if that is what they are, before training |
| Warning | Nearly every image has a close look-alike in the set (Near-duplicate search) | Check whether any look-alike group spans train and validation |
| Warning | Ten class pairs pull toward each other, one a naming split (Label diagnostics) | Settle Resestor first, then review the Cap3/Cap4 pair |
| Notice | Two boxes sit at 3px or less on their short side (Dataset statistics) | Delete them if they are stray clicks |
| Notice | Ten images have blown highlights, possibly shiny parts (Image quality) | Open them before deciding whether to exclude any |
| Notice | Twenty-nine images fall far below this dataset’s own sharpness (Image quality) | — |
| Notice | Three labeled-split images carry no boxes at all (Annotation quality) | — |
| Notice | The anomaly threshold sat above every score, so nothing was flagged (Outlier search) | Re-run with a threshold set from the score percentiles |
| Notice | The tried prelabeling model proposes almost nothing on these boards (Prelabeling trial) | Try a detector trained on a few labeled boards |
| Passed | The test split is unlabeled by design (Dataset statistics, annotation quality) | — |
The first three next steps form the report’s do-first list. Each finding in the report comes with the images behind it, so a Notice can be judged by eye before anything is excluded:
The sections below cover the findings where reading the summary changes the decision.
A misspelled class name, caught from three sides
Eight class names are Cap1 through Cap4, Resistor, Transformer, MOSFET and Mov. The
ninth is Resestor: three boxes, all in validation, none in train. Three checks point at it, and
the summary reports them as one finding:
- Annotation quality flags the name as a likely typo of
Resistorand the class as rare. - Label diagnostics compares objects by their embeddings: a numeric fingerprint a vision
model computes for each object, so that parts that look alike get close fingerprints. It finds 90
of 1,120
Resistorobjects closer to the averageResestorfingerprint. That is how one component labeled under two names looks. - The split breakdown puts every
Resestorobject in validation.
By file name, the three images that carry Resestor are copies of one frame, so this is most
likely one component labeled once under the wrong name. A model trained on train never sees
Resestor, so its validation score for that class grades something it was never taught. The misspelling is in the original release: Dataset Ninja lists the
classes as ”… cap4, and resestor.”
Near-duplicate images: 99% have a look-alike. Do they leak into validation?
The near-duplicate search compares image embeddings, not files, and finds a near-identical or exact look-alike for 1,396 of 1,410 images. Read alone, that says the dataset is made of copies, yet only 5 image pairs are exact duplicates.
Photos of circuit boards on a similar backdrop are close by nature, so the summary rates this a Warning, not Critical, and points at what matters: whether look-alikes sit on both sides of the train/validation split. Only that makes a validation score look better than the model is.
For this dataset we answered that question by hand, against the source files, after the run; it is
not part of the pipeline. The original file names repeat one frame with different export suffixes,
for example VID20210601143927-96_jpg.rf.51037a… and VID20210601143927-96_jpg.rf.7ab534….
Grouped by name, the 1,410 images are about 433 distinct frames, most in three copies:
Only 2 of the 200 validation images repeat a frame from train. Test repeats train in 93 of 111 images, but test has no labels, so none of that reaches a metric.
Class pairs that look alike
Label diagnostics also checks which class fingerprint each object sits nearest. Ten directions stand out. The strongest are Cap4 toward Cap3 (257 of 1,067 objects), Cap3 toward Cap4 (201 of 1,198) and Transformer toward MOSFET (300 of 2,547).
Most of this is ordinary visual similarity between parts that look alike up close. Resistor toward Resestor (90 of 1,120) is the naming split above, which is why the summary puts the rename first and the Cap3/Cap4 review second.
Checks that could not answer: “not measured” is not “passed”
Two checks came back empty, and the summary counts neither as a clean result:
- Outlier search: 0 of 1,410 images reached the 0.25 threshold, because the threshold sat above every score. The check could not have flagged an unusual image even if one existed.
- Missing-box search: it compares a trial detector’s confident predictions with the labels. The detector made almost no confident predictions, so there was nothing to compare, and the summary lists the check as not measured.
A reader scanning for green checkmarks would take an empty result as reassurance about exactly the two things nobody checked.
A weak prelabeling model is not a bad dataset
The trial prelabeling model is an off-the-shelf detector, never trained on these boards. It gets the lowest grade, Minimal assist: 11,108 of 11,123 ground-truth objects were never proposed at all. The 11,123 is this evaluation’s own count, a slightly different set from the 11,119 boxes above.
The summary keeps this at Notice and says what it measures: the detector, not the labels. A bad dataset needs relabeling; a weak prelabeling model needs a different model.
What checked out fine
- All 11,119 annotations are bounding boxes, with no other shape types mixed in.
- None of the 1,099 training images is empty. Of the 114 images without boxes, 111 are the unlabeled test split and 3 sit in validation, matching the 114 that Dataset Ninja reports.
What comes next
Each check’s own report holds the detail: the
dataset inspection study reads the quality report
chart by chart. The dataset cleanup study applies
the Resestor rename to this same dataset. The training run
and the two-detector comparison trained on this data
before the audit, typo and look-alikes included.