Welcome to Annotate 101!

After you complete your onboarding requirements, you will be granted access to Labelbox.
Labelbox is the platform where you will do all your labeling work as an Alignerr. More
specifically, you’ll do your work using Labelbox’s “Annotate” product.
What is Annotate?
Annotate contains all the tools and workflows you’ll need to complete your labeling tasks, view
your labeling progress, communicate with your project admin, and understand your progress and
performance.
In this course
In the following sections, we will cover these topics:
How to complete labeling tasks
How labeling tasks are reviewed
How to use the “Issues” feature to ask questions
How Alignerr's performance is monitored
Before we get started, let’s go over some key concepts.
Please read through the key terms and definitions below. Understanding these key terms will
help you grasp the concepts covered in the rest of this training.
Key terms
Editor The labeling interface in Labelbox that you will use to view and label each data row.
Data row A data row represents a file to be labeled. A data row can be an image, a video, a
text file, or a conversation thread. Data rows are presented one by one in the editor for Alignerrs
to label.
Annotation An annotation is a single bounding box, classification, segmentation mask, etc on
a data row. The editor interface contains a preset menu of annotations (i.e., an ontology) that you
will use to label your assigned data rows.
Label A “label” represents all of the individual annotations created by a single person on a
given data row. You can think of a label as a completed labeling task on a data row.
Ontology When you open the editor, you will see a preset menu of annotations that you can
use to complete the labeling task. This menu of annotations is called the “ontology”.
Issues If you see a problem with the labeling task or have a question, click on the “issues” icon
in the editor to start a communication thread with the project admin.
Queue Each project has a queueing system that distributes labeling tasks, reviewing tasks, and
rework tasks to each Alignerrs in the project (based on their role). This queueing system ensures
the tasks are distributed fairly among Alignerrs.

Test 1
Match the terms to their definitions. *
Note: Drag cards from the left column to their corresponding match in the right column.

Annotation
Menu of available annotations you can use when labeling a data row.
Label
Represents a single piece of data to be labeled.
Data row
The sum of all annotations submitted on a data row by a single person.
Queue
A single annotation (e.g., bounding box, classification) created in the editor.
Ontology
A mechanism for asking questions or raising concerns when labeling.
Issues
System that automatically distributes labeling tasks.
Part 3
What happens when you get a labeling assignment?
When you are matched to a labeling project that matches your skill set, we will send you a
notification that includes the project scope and the pay rate. If you agree to the project scope and
pay rate, we will assign you to the labeling project.
Once you are assigned to a project, you will be able to access the project in Labelbox. However,
before you can officially begin your assigned scope of work, you must first pass the Evaluation
stage by completing a few labeling tasks.
Note: After you complete your onboarding tasks and training, you may experience a waiting
period before you are matched to a project. The time it takes for Alignerrs to be assigned to a
labeling project varies based on demand and skill set.

What is the evaluation stage?

In the Evaluation stage, you will be prompted to label a small set of data rows. It is critical to do
your best when labeling the data rows in the Evaluation stage. These labels will be evaluated by
our team to ensure your work meets our quality standards. You will be paid for time spent
labeling tasks in the Evaluation stage.
If your labels pass inspection during the Evaluation stage, you will be promoted to the
Production stage of the project. Once you are promoted to the Production stage, you may
officially start your assigned labeling work in the project!
You can think of the Evaluation stage as a test run before the real work begins.
Remember the rules
As you are labeling, please keep the following rules in mind:
Excellence
Maintain the highest standards of quality in all work. Do not produce or submit subpar work.
Integrity
Do not game or manipulate the assessment exams, calibration projects, or timer system. We can
identify such behavior and have logs of screen activity.
Confidentiality
Do not discuss project details publicly or with unauthorized individuals. This includes refraining
from sharing information on social media, forums, or public platforms.

Part 4
Once a project is in the production stage, Labelbox uses a system called Workflows to distribute
data rows to labelers and reviewers to ensure label quality. All data rows must pass through a
series of workflow steps before the labeling tasks can be considered “Done.”
You can think of workflows as a series of steps in an assembly line to ensure all data rows meet
quality standards.
What are the workflow steps?
The default workflow steps are:
To label This is the initial step in the labeling workflow. The data rows in this step are
waiting to be distributed to Alignerrs so they can be labeled.
In review Data rows in this step have been submitted (or skipped) in the editor. Data rows in
this queue will be distributed to other team members for review.
In rework The data rows in this step were rejected during the review step. Therefore, they
must be relabeled or “reworked.”
Done The data rows in this step have been labeled, reviewed, and approved by a reviewer. This
is the final step in the labeling workflow. The goal of the labeling project is to get all data rows
to the “Done” step.
If you are a labeler, you will only label tasks in the “To label” or “In rework” steps. If you are a
reviewer, you may review tasks in the “In review” step.

Part 5
An easy way to filter your data rows by workflow step is by using the Data Rows tab.
What is the Data Rows tab?
The Data Rows tab is helpful for quickly viewing how many data rows are left in each workflow
step and allows you to quickly filter for the data rows you are looking for. The Data Rows tab
displays key information for each data row such as labeling and review time, the dataset the data
row belongs to, if there are any open issues, and more details dependent on project settings.
The Data Rows tab lets you filter data rows by:
Workflow step (To label, In review, In rework, Done)
Issue
Annotation
You can also use the natural language search or find text filters to find data rows.
Demo
Watch this demo to learn how to navigate the workflow steps and how to use the Data Rows tab
to filter data rows by workflow step.
Play Video
NOTE: This video must be fully watched before moving on from this Task

Part 6
What is the projects list?
When you navigate to the Annotate home page, you will see a list of projects that you have been
assigned to. You can use the filters to narrow down your projects if needed.
Try it out
Click through the interactive tour below to learn how to use the filters in the projects list.
Part 7
Awesome job so far! Now that you understand how to navigate the project dashboard, let's cover
the tools you'll use to complete your labeling tasks.
When you click “Start labeling” from the project dashboard, this will bring you to the labeling
interface, also known as the editor. The labeling editor contains the following main components.
What is in the editor?
Component Definition
Tools panel This is where you’ll find the predetermined ontology for your labeling tasks. Use
the keyboard shortcuts to speed up your labeling.
Annotations/Tasks panel Here you’ll find a record of all annotations created on the data row.
You can toggle their visibility, change the class, or move the annotation to back/front.
Labeling instructions If the project has specific labeling instructions, you can find them
by clicking on the instructions icon in the top navbar. Every project should have only one set of
labeling instructions.
Issues Click on the issues icon to open an issue on a specific location. You can use the issues
feature to raise a concern, ask a question, etc.
Keyboard shortcuts Open the keyboard shortcuts panel to view all available shortcuts to speed
up your labeling efforts.
Watch this demo to understand the fundamental concepts and layout of the labeling interface.
How to use the editor navbar
The editor has a set of tools that you can use during your labeling or review tasks. From the
navbar, you can access:
 Labeling instructions
 Attachments
 Data row details
Click through the interactive tour below to learn how to use the editor navbar.

Part 8
When labeling in the editor, you will have the option to submit or skip. The default selection is to
submit your labeling work. However, you can choose to skip the data row if:
 The data row cannot be properly labeled with the ontology provided.
 There is an issue with the visibility of data row (blurry, not viewable, etc).
 There is some other issue blocking you from being able to label the data row.

If you choose to skip a data row, please open an issue and explain why you are skipping the
data row. The project manager will review your comments and resolve the issue. Instructions on
opening issues in the editor are covered in the next section.
Demo
Watch this demo to understand how to navigate the data rows in your labeling queue.
Part 9
When you are labeling, there may be times when you need to open an issue or leave a comment
on an annotation.
Common reasons for opening an issue are:
 You have a question about the labeling task
 You see a problem with the data row
 The labeling instructions are not clear

Take a tour
Click through this interactive tour to learn how to create issues in the editor.
Part 10
You're making great progress! Now, that you understand how to label data rows in the editor,
let's move on to how labeling tasks get reviewed.
How to review data rows
During your time as an Alignerr, you may be asked to review other Alignerrs’ labeling tasks.
Labelbox provides you with the tools and resources to evaluate label quality, give feedback, and
approve/reject labels all within the Labelbox platform.
In the project overview, you’ll see a section called “Workflow tasks” toward the bottom. This
section contains any data rows that need to be reviewed. Click “Start reviewing” to begin
reviewing other Alignerrs’ labels.
Watch this demo to understand how our review process works.
Play Video
NOTE: This video must be fully watched before moving on from this Task
If you are asked to review data rows, you'll have the option to them to rework.
What is rework?
Rework is a workflow step reserved for data rows that are rejected during the review step.
Data rows in the rework step will automatically be distributed back to Alignerrs for relabeling.
If there are data rows that need to be relabeled, the Rework step in the “Workflow tasks” section
will be activated. Click “Start reworking” to begin relabeling rejected labels.
Labelers (in addition to reviewers) can rework labels on any data rows sent to the Rework task,
regardless of whether they created the label.
How to select a Consensus winner
Consensus is a scoring system that Labelbox uses to calculate how much agreement there is
between multiple submitted labels on a data row. In other words, a Consensus score represents
how much overlap or how similar one labeler’s work is to another labeler’s work on the same
data row. Labelbox uses this scoring system to spot errors or mistakes.
By default, projects that use the Consensus scoring system will assign the Consensus label to the
first label submitted for that data row. If you are promoted to a reviewer role, you may be asked
to select which label is the “winner” out of all of the labels submitted for that data row.
To learn how to select a label as the Consensus “winner”, watch this demo.
Play Video
NOTE: This video must be fully watched before moving on from this Task

Part 11
When you are labeling data rows in the editor, you can open an issue in the editor interface.
Issues are commonly opened if the Alignerr has a question, needs clarification on the
instructions, or would like to point out an issue with the labeling task.
How to respond to/resolve issues
If you are promoted to a reviewer role, you may be asked to respond to and/or resolve issues
created by other Alignerrs in the project.
Watch this demo to learn how to respond to issues.
Play Video
NOTE: This video must be fully watched before moving on from this Task

Part 12
If you are ever curious about how well you are doing in your labeling (or review) tasks,
Labelbox provides all the tools you need to monitor your own performance throughout the
duration of the project.
What is the Performance tab?
The Labelbox team uses a combination of automatically generated metrics and human review to
ensure that Alignerrs are working efficiently and creating quality labels.
However, if you would like to review your performance metrics, you can head over to the
Performance tab.
The Performance tab displays metrics that help you visualize how well you are doing in your
labeling tasks. This tab is organized into three major sections: throughput, efficiency, and
quality. Each component has unique views to help you understand your overall performance.

Throughput
These metrics provide insight into the amount of labeling work produced and help you answer
questions like: “How many data rows did I label?” or “How much time have I spent labeling?”
Efficiency
These metrics help you visualize the time spent per unit of work, per labeled asset, or per review.
These metrics help answer questions such as: “What is the average time spent per label?” or
“What is the average time spent reviewing?”
Quality
These metrics display how your labels compare to other Alignerrs’ work. The Benchmarks
metric compares labels to a pre-selected “gold standard” label. The Consensus metric is a
calculation that compares your label to other Alignerrs’ labels.
Demo
Watch this demo to learn how to use the Performance tab.
Play Video
NOTE: This video must be fully watched before moving on from this Task
Part 13
You made it through your training! One more thing before we wrap up this course.
Guides
Click through these links to deepen your knowledge of the key concepts we covered in this
course:
 Keyboard shortcuts
 Issues & comments
 Performance dashboard
Blog posts
Click through these links to gain a better understanding of Labelbox and Alignerr in general.
Blog post: Inside the data factory: How Labelbox produces the highest quality data at scale
Blog post: Improve multimodal evaluations with enhanced arena-style experience
Product Spotlight: Building the data factory for Gen AI and frontier models

Need Support as you're labeling? Reach out to our team of Machine Learning Support Engineers
here: https://labelbox.atlassian.net/servicedesk/customer/portal/2