TL;DR — A high-quality dataset is produced by a team, not by labeling volume. A reliable annotation taskforce has five roles — ML specialist, taxonomy lead, annotators, project manager and reviewers — each responsible for one part of quality, from the first guideline to the final check.
Selecting and preparing the right training data is fundamental to every AI model. Models built for industrial applications usually need even more attention: datasets with tens of thousands of data points and carefully chosen annotated features. Delivering that at consistent quality needs a structured team. Here's who does what.
What does the ML specialist do?
The machine learning engineer should take part in requirements definition and annotation iterations from day one. Their involvement ensures that:
- Annotation guidelines translate correctly into model training requirements
- Technical constraints are reflected in the annotation specification
- Early feedback catches problems before they spread across a large dataset
Why do you need a taxonomy lead?
Most label-quality problems start with a bad taxonomy. The taxonomy lead designs the class hierarchy, the rules for edge cases (occlusions, truncated objects, ambiguous classes) and the written guidelines — together with your team, before production starts. Clear documentation directly affects project success, which is why this role sits between the ML specialist and the annotators.
What makes annotators reliable?
Annotators create the labeled data; team size depends on volume and timeline. Their reliability and understanding of the task determine the final dataset quality — which is why the initial training session matters so much. Annotators who understand how their work will be used make better judgement calls on edge cases, and a shared mental model keeps ambiguous scenarios consistent across the team.
What does the project manager do?
On large annotation projects, the project manager structures the work into tasks and assigns them to the team. Responsibilities include:
- Overseeing training and enforcing quality standards
- Tracking progress against the timeline and spotting bottlenecks
- Assigning and coordinating QA reviews
- Keeping communication open between annotators, the taxonomy lead and the client's technical team
Good project management prevents a very common pattern: quality dropping as the team rushes to hit volume targets.
Why is the reviewer the last line of defence?
Even the best annotators get tired on long, repetitive tasks and may miss or mislabel objects. Reviewers stop those errors before they reach the training set. Reviews can be organised as:
- Cross-checking — annotators review each other's work
- Independent QA specialists — reviewers with domain expertise
- Automated consistency checks — flagging statistical outliers in the labels
How do the roles work together in a project?
- Requirements — the ML specialist and taxonomy lead agree what the model needs and how it maps to classes.
- Guidelines and sample — the taxonomy lead writes guidelines and the team labels a small sample.
- Training — annotators are trained on the guidelines and the sample results.
- Production — the project manager runs batches; reviewers check each one.
- Feedback loop — new edge cases update the guidelines; the ML specialist checks model results on the labeled data.
How is this team set up at BoBox?
Our projects are run by senior annotators and a taxonomy lead rather than a rotating crowd. Two QA passes are included by default, with consensus labeling available on request, and we start with a paid pilot so the team, guidelines and throughput are proven on your data before production. Read more about why details in annotation matter, see our Safety1st case study, or check how pilots and pricing work.
Frequently asked questions
Which roles does a data annotation project need?
At minimum: an ML specialist who knows how the data will be used, a taxonomy lead who writes the annotation guidelines, trained annotators, a project manager, and reviewers who check quality before data reaches training.
Why should the ML engineer be involved in annotation?
Because they know what the model needs. Involving them from requirements onwards makes sure the guidelines translate into useful training data and catches problems before thousands of images are labeled the wrong way.
How do you keep annotation quality consistent at scale?
With written guidelines and a taxonomy agreed before production, an initial training session for annotators, review passes on every batch, inter-annotator agreement checks, and a feedback loop that updates the guidelines when new edge cases appear.
How many review passes does a dataset need?
It depends on how critical the model is, but a single pass rarely catches everything on large datasets. At BoBox, two QA passes are included by default, with consensus labeling by several annotators available for the hardest cases.
Should we build an internal annotation team or outsource?
Internal teams make sense for small, continuous labeling tied closely to product work. For large or specialised datasets, an external team with established roles and QA usually delivers faster and keeps your ML engineers focused on models.



