The task
A research team is labeling malaria blood-smear images to train a parasite-detection model. Parasites sit inside red blood cells, and the smallest of them — the ring stage — is easy to miss or to confuse with a stain mark. On data like that, one annotator’s work checked by one reviewer is a single unverified opinion, and a wrong accept is expensive to find later.
This study takes the basic label-and-review loop and changes one thing: who decides. Instead of one reviewer per image, a panel judges every submission, and the expert is called in only when the panel genuinely disagrees.
What changes when a panel decides
One reviewer is a single point of judgment. Whatever they accept is accepted; nobody else sees the image, and a missed object leaves no trace.
A panel shows the same image to several reviewers — you set how many — and each votes independently. Agreement decides: everyone accepting sends the image on as completed, everyone rejecting sends it back for another pass. A split means there is no verdict yet, and that image, and only that one, goes to an expert.
The point is not to review more. It is to spend the specialist’s attention where opinions actually diverge, and to let routine agreement close everything else on its own.
- Existing loopLabel and retryRejected annotations return to the labeling pool
- Routine review3 reviewers hereConsensus Review shows every item to the full panel
- Exception routeSplit votes onlyUnresolved holds items without unanimous agreement
- Expert Review1 final callThe specialist accepts the item or returns it for correction
A deliberately difficult test set
A 60-image subset of the P. Vivax (Malaria) Infected Human Blood Smears dataset (BBBC041), sourced through Dataset Ninja. All seven of the dataset’s classes are present, including difficult — cells its own authors could not confidently classify either. The subset over-represents the rare and hard classes, so the run exercises the cases the graph is built for rather than easy ones.
When the panel does not agree
Every image sent to review goes to the whole panel, and each reviewer votes independently. This run requires unanimity rather than a majority, a deliberate choice given what a wrong accept costs here. Unanimous accept moves an image to the completed dataset; unanimous reject returns it to the labeling pool. Anything else has no verdict, and that is what the expert exists for.
Of the ten images the panel has decided so far, one ended that way — a dense field of ring-stage parasites, the smallest and hardest of the four stages to spot. It sits in the unresolved dataset, waiting for the specialist, who will accept it or send it back.
The flow records each verdict but not the reasoning behind it, so the callout marks what is hard to read in the image, not a stated reason for anyone’s vote.
A missed detection, caught without anyone auditing the work
A panel does not only settle close calls. It also catches submissions that are simply incomplete.
What was submitted for the image below boxed red blood cells and nothing else — 44 objects, all of one class. Four objects were missing: two trophozoite, one schizont and one leukocyte. Nobody was told anything was wrong with it; the reviewers found it by looking.
The image is on its way back to the labeling pool for another pass. Nothing about that needed a specialist, an audit or a spot check — the same routine review that accepts good work rejects incomplete work.
What the extra reviews buy
The panel has decided ten images: seven accepted, two sent back, one disputed. The expert was needed for one of ten — not for the whole batch.
That is the trade in plain terms. Three reviewers mean three review passes per image instead of one, and those passes are cheap and parallel: everyone works their own queue at their own pace. What they buy is that the scarce, expensive attention goes only where opinions actually diverged. Set the panel to two reviewers or five, keep the strict unanimous rule or relax it to a majority — both are settings, and both move where that line falls.
Summary
What you end up with is a training set where every accepted image carries more than one person’s word for it. Everything goes through the panel, and the straightforward cases end there: accepted once the reviewers agree, sent back when the work is incomplete. The expensive opinion is reserved for the cases the panel genuinely argues about — which is the only place it changes the outcome.
How strict that confidence is stays a dial rather than a process. Add a reviewer to raise the bar, relax unanimity to a majority to move faster, and the same graph keeps running: it shows each image to the panel, reads the votes, and routes the result without anyone chasing it. The only thing left to manage is the standard you want to hold the data to.