The decision starts after training
Training a second model creates a choice, not an answer. The difficult part is deciding which checkpoint to ship when both candidates were expensive to produce and every small difference in data preparation, evaluation settings, or metric code can make an unfair comparison look conclusive.
This experiment puts EdgeCrafter ecdet_s and DEIM deimv2_hgnetv2_n through one shared process: the same PCB component data, the same held-out images, and the same evaluation method. EdgeCrafter wins the final comparison, but the useful result is more precise than “model A is better.” The evidence shows where its advantage is meaningful, where the models are effectively tied, and where the data is too thin to support any architectural verdict.
One test for both candidates
Both models train on the same 1,099-image labeled pool. A separate set of 200 images and 1,765 ground-truth components is held out from training, then reused image for image for both evaluations. The classes cover several capacitor types, MOSFETs, varistors, resistors, and transformers — the uneven mix found on real boards rather than a synthetic balanced benchmark.
The workflow carries the experiment from the shared split through two training runs, two sets of predictions, two evaluations, and one comparison. Keeping those steps connected matters: the winner remains traceable to the exact data and settings that produced it, and a third candidate can later enter the same test without rebuilding the benchmark by hand.
| Kept identical | Allowed to differ |
|---|---|
| Training and held-out data | Architecture and its native training implementation |
| 21 training epochs | Learned checkpoint |
| 640x640 input resolution | Predictions produced by that checkpoint |
| Held-out evaluation method | Resulting accuracy metrics |
Training reveals an early gap
EdgeCrafter reaches its best validation mAP of 69.5% at epoch 20. DEIM peaks earlier at 64.7%. The curves are more useful than the final epoch alone: EdgeCrafter’s validation score oscillates throughout training, while DEIM improves sharply in the first few epochs and then settles into a narrower band.
These are training-time validation results, not the final head-to-head scores. Keeping that distinction visible prevents a familiar reporting error: quietly mixing the best number observed during training with a later score measured under a different evaluation run.
The held-out verdict favors EdgeCrafter
On the shared held-out set, EdgeCrafter records 68.9% mAP against DEIM’s 64.3%, a lead of 4.6 percentage points. The gap persists at the stricter AP75 threshold (84.0% versus 79.3%) and in the balance of precision and recall: EdgeCrafter reaches 72.9% F1, compared with 68.7% for DEIM.
| Model | mAP | AP50 | AP75 | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
| EdgeCrafter ecdet_s | 68.9% | 86.8% | 84.0% | 72.4% | 73.4% | 72.9% |
| DEIM deimv2_hgnetv2_n | 64.3% | 86.9% | 79.3% | 68.0% | 69.5% | 68.7% |
DEIM is fractionally ahead on AP50 — less than 0.2 percentage points — but AP50 is the most forgiving localization threshold in this comparison. EdgeCrafter’s advantage grows when boxes must align more tightly, which is why AP75 and overall mAP change the shipping decision even though the AP50 scores look tied.
The images explain what the averages cannot
Aggregate metrics make the decision comparable; prediction examples make it reviewable. These two held-out boards show the ground truth beside both models’ detections. They were rendered from the same image and annotation assets used by the evaluation reports, then saved with the article so the page remains independent of the live Platform instance.
The first example checks whether the aggregate result survives contact with an ordinary board. Both models recover the dominant components, while their box placement and treatment of the crowded center differ enough to justify looking beyond the headline score.
The second board changes the orientation and arrangement while preserving the same evaluation contract. Across both examples, the panels expose details a scalar score cannot. Two examples do not replace the 200-image evaluation; they give a reviewer concrete cases to inspect alongside it.
The average hides class-level evidence
The overall lead is not uniform. EdgeCrafter’s largest class-level mAP advantages appear on Cap2 (73.6% versus 58.6%) and Cap1 (82.7% versus 71.1%). DEIM narrowly leads on Resistor (61.0% versus 60.6%), a weak class for both models rather than a decisive counterexample to the overall result.
Both models score zero on the separate Resestor class. That label has only three ground-truth examples in the held-out set, so its result is a data warning, not evidence that both architectures fundamentally cannot detect the component. A useful comparison preserves that uncomfortable detail instead of averaging it away.
Ship the winner, keep the experiment
For this accuracy-focused test, EdgeCrafter is the checkpoint to carry forward. Its 4.6-point mAP lead survives the stricter localization test and is supported by stronger precision, recall, and F1. DEIM’s near-tie on AP50 and tiny Resistor advantage remain visible, so the decision is not based on a selectively chosen headline metric.
That is still not a complete production decision. The workflow did not measure latency, memory, or operational cost, and those may justify a separate benchmark before deployment. What OvalBee makes repeatable here is the evidence chain: shared data, explicit training settings, held-out evaluations, and a comparison that can be rerun when the team adds another architecture or changes the dataset.