Annotation interfaces
The web app lets you annotate a variety of different formats, including plain
text, named entities, categorization, images and A/B comparisons, as well as raw
HTML. Prodigy expects annotation tasks to follow a simple JSON-style format.
This format is also used to communicate tasks between REST API and web
application, and will be used when exporting annotations via the db-out
command. Annotated tasks contain an additional "answer" key mapping to either
"accept", "reject" or "ignore".
text, html | Annotate plain text or HTML. |
ner, ner_manual | Annotate and correct named entities. |
spans, spans_manual | Annotate and correct potentially overlapping spans. |
pos, pos_manual | Annotate and correct part-of-speech tags. |
relations, dep | Annotate syntactic dependencies and semantic relations. |
classification | Annotate labelled text or images. |
image, image_manual | Create bounding boxes and image segments. |
audio, audio_manual | Annotate and correct regions in audio and video files. |
choice | Select one or more answers. |
review | Review and merge existing annotations by multiple annotators. |
diff, compare | Compare two texts with a visual diff. |
text_input | Collect free-form text input. |
blocks | Combine different interfaces and custom content. |
llm_io | Show the LLM prompt/response |
text binary
JSON task format
How can I render formatted text?
The text property of an annotation task will always be rendered as plain text.
To add markup like line breaks or simple formatting, use the html key instead.
Prodigy wants you to explicitly choose to use “HTML mode” to avoid stray HTML
tags (which can influence the model’s predictions) from being rendered or hidden
– for example, if you’re working with raw, uncleaned data. If you’re using one
of Prodigy’s default recipes with a model in the loop, keep in mind that the
text of an annotation task is used to update the model.
ner binary
The ner interface renders one or more entity spans in a text using their start
and end character offsets and an optional label. If you’re creating NER
annotation tasks manually and need to render multiple entities per task, make
sure to provide them in order.
JSON task format
ner_manual manual
The ner_manual interface allows highlighting spans of text based on token
boundaries. Recipes like ner.manual will typically use a model to
tokenize the text and add an additional "tokens" property to the annotation
task, e.g. using the add_tokens
preprocessor. This allows the application to resolve the highlighted spans back
to the respective token boundaries. Selected entities will be added to the
task’s "spans". Individual tokens can also set "disabled": true to make them
unselectable (and make spans including those tokens invalid).
JSON task format
Why does Prodigy add ↵ characters for newlines?
A newline only produces a line break and is otherwise invisible. So if your
input data contains many newlines, they can be easy to miss during annotation.
To your model however, \n is just a unicode character. If you’re labelling
entities and accidentally include newlines in some of them and not others,
you’ll end up with inconsistent data and potentially very confusing and bad
results. To prevent this, Prodigy will show you a ↵ for each newline in the
data (in addition to rendering the newline itself). The tab character \t will
be replaced by ⇥, by the way, for the same reason.
As of v1.9, tokens containing only newlines (or only newlines and whitespace)
are unselectable by default, so you can’t include them in spans you highlight.
To disable this behavior, you can set "allow_newline_highlight": true in your
prodigy.json.
Why can't I add HTML formatting to the manual UI?
When you’re highlighting spans in the manual interface, you’re still annotating
raw text and are creating spans that map to character offsets within that
raw text. If you pass in "<strong>hello</strong>", there’s no
clear solution for how this should be handled. How should it be tokenized, and
what are you really labelling here? The underlying markup or just the text, and
what should the character offsets point to? And how should other markup be
handled, e.g. images or complex, nested tags?
Similarly, if you’re planning on training a model later on, that model will also get to see the raw text, including the markup – so if you are working with raw HTML (like web dumps), you usually always want to see the original raw text that the model will be learning from. Otherwise, the model might be seeing data or markup that you didn’t see during annotation, which is problematic.
Available config settings
Settings can be specified in the "config" returned by a recipe, or in your
prodigy.json.
| Name | Description | Default |
|---|---|---|
hide_newlines | New: 1.9 Don’t add real line breaks to tokens and only use ↵ symbols. Previously called hide_true_newline_tokens. | False |
allow_newline_highlight | New: 1.9 Allow highlighting tokens that only consist of newlines. If disabled, spans including those tokens become unselectable. | False |
honor_token_whitespace | New: 1.10 Reflect whitespace or lack of whitespace following a token in the UI. | True |
ner_manual_highlight_chars | New: 1.10 Optimize UI for single-character selection and "tokens" that describe single characters. | False |
ner_manual_require_click | Don’t auto-add the selection as an entity and require an additional button click to convert the selection to an entity. | False |
label_style | New: 1.10.4 Style of label set. "list" for list of keyboard-accessible buttons or "dropdown". Previously available as ner_manual_label_style. | "list" |
spans binaryNew: 1.11
The spans interface renders one or more potentially overlapping spans in a
text using the token information, the start and end character offsets and a
label. The same span can be included multiple times with different labels. For
the interface that lets you add and modify spans, see spans_manual.
JSON task format
spans_manual manualNew: 1.11
The spans_manual interface allows highlighting spans of text based on token
boundaries with multiple potentially overlapping and nested spans. It’s
primarily used by the spans.manual recipe, which uses a model to tokenize
the text and add an additional "tokens" property to the annotation task, e.g.
using the add_tokens preprocessor. This
allows the application to resolve the highlighted spans back to the respective
token boundaries. Selected entities will be added to the task’s "spans".
Individual tokens can also set "disabled": true to make them unselectable (and
make spans including those tokens invalid).
JSON task format
pos binary
Under the hood, the POS tagging interface uses the same UI component as the
ner interface. It also accepts and produces the same data format. The
default theme ships with several customizable
highlight colors for part-of-speech
tags that make it easier to distinguish them.
JSON task format
pos_manual manual
Under the hood, the manual POS tagging interface uses the same UI component as
the ner_manual interface. It also accepts and produces the same data
format. The default theme ships with several customizable
highlight colors for part-of-speech
tags that make it easier to distinguish them.
JSON task format
relations manualNew: 1.10
The relations interface lets you annotate directional labelled
relationships between tokens and expressions by clicking on the “head” and
selecting the “child”, and optionally also assign spans for joint entity and
dependency annotation. Single expressions can be part of multiple overlapping
relations and you can configure the UI to only show arcs on hover and switch
between a single-line tree view and a display with tokens wrapping across lines.
If you’re in span annotation mode, clicking on a span will select it so you can
click or hit d to delete it.
To add a span, click and drag across the tokens, or hold down shift
and click on the start and end token.
To use the interface most efficiently, you typically want to preprocess your
text to add spans (e.g. named entities, noun phrases) and disable tokens you
know will never be part of a dependency relation you’re interested in (e.g.
punctuation). The rel.manual recipe lets you define those using powerful
token-based patterns, and the built-in task-specific recipes like
coref.manual and dep.correct include suitable predefined
configurations.
JSON task format
Relationships are defined as dictionaries with a "head" and a "child" token
index, indicating the direction of the arrow, the corresponding "head_spans"
and "child_spans" describing the tokens or spans the relation is attached to,
as well as a relation "label". If "spans" are present in the data, they will
be displayed as a merged entity. If relations are added to spans, they will
always refer to the last token as the head and child, respectively. If spans
with existing relations are merged or split, Prodigy will always try to resolve
and reconcile the indices.
The interface lets you toggle the following display settings:
| Switch to dependency annotation mode to assign labelled relations to tokens and spans (default mode). To select a head and its child, click on the tokens in order. To attach a token to itself, double-click on it. To remove a relation, click on the arc or its label. | |
| Switch to span annotation mode to merge, highlight and label spans like entities. Spans are indicated by a dotted border. To select a span, drag across the tokens or from the start token to the end token or hold shift and click the first token and last token. To remove a span, click on it to select it and hit or press d. | |
| All relations | Display all annotated relation arcs. If disabled, the tokens retain their color indicating they’re part of an annotated relation, but the arcs connected to a token are only shown on hover. |
| All labels | Show all labels of “special” tokens, e.g. merged named entities or pattern matches. If disabled, labels are only shown on hover and dotted borders indicate “special” tokens. |
| Wrap | Wrap tokens over multiple lines if needed, instead of displaying them all in one row with a scrollbar. Can be useful for longer texts with few but very long dependencies. |
What does it mean if the dragged box in span mode turns red or green?
When you drag across tokens to select them in span highlighting mode, your selection will either turn green or red. This indicates whether the span you’ve selected is valid and can be added. For instance, since a token can only be part of one entity, a span selection is invalid if it overlaps with another span. If you want to create a larger span, remove the previous span first and then add the new span.
Spans also can’t be added if they include disabled tokens, i.e. tokens with
"disable": true. You can take advantage of this to prevent annotation mistakes
and precprocess your data to automatically disable certain words and word types
that shouldn’t be part of spans or relations – like punctuation or determiners.
See the rel.manual recipe for details.
How can I highlight spans over multiple lines?
If you want to highlight a span across a line break, it doesn’t always work to click and drag, because you may be selecting tokens in between that you’re not interested in. In that case, you can either toggle the wrapping at the top and do your span selection witout line breaks, or hold down shift and click the start and end token of the span you want to highlight. You can also custiomize this shortcut.
How can I attach a dependency like the root to itself?
To denote the syntactic root of a sentence, the ROOT dependency is typically
attached to itself and has the same head and child token. In the relations
interface, you can achieve this by double-clicking or double-tapping on
the token you want to attach to itself. It will then be shown as a
downward-pointing arrow. You can see an example of this in the
dependency parsing docs.
What happens if I remove a span that has a relation attached?
Under the hood, "relations" can refer to individual tokens or spans. If you
remove a span that is also the head or child of a relation, this relation will
be reattached to the last token of the span. Prodigy will always try its
best to reconcile the existing relations – however, it’s recommended to assign
and correct your spans first, and then move on to add relations to them.
Available config settings
Settings can be specified in the "config" returned by a recipe, or in your
prodigy.json. In addition to these config settions, you can also change the
maximum height of the arc annotation space via the
custom theme settings.
| Name | Description | Default |
|---|---|---|
relations_span_labels | Optional list of span labels to assign during annotation. If set, an additonal span highlighting mode is added. | None |
wrap_relations | Wrap text across multiple lines instead of displaying all tokens in one line with scrollbar. Will be used as the default value for the checkbox. | False |
show_relations | Show all relations. Will be used as the default value for the checkbox. | True |
show_relations_labels | Show all labels for spans, instead of only on hover. Will be used as the default value for the checkbox. | True |
hide_relations_arrow | Hide the arrow head. Relations will still be added as "head" and "child", but if the distinction and directionality doesn’t matter to you, you can ignore it and hide the arrows. | False |
label_style | New: 1.10.5 Style of label set. "list" for list of keyboard-accessible buttons or "dropdown". Previously available as ner_manual_label_style. | "list" |
dep binary
Dependencies are specified as a list of "arcs" with a "head" and a "child"
token index, indicating the direction of the arrow. Since the arcs refer to the
token index, the task also requires a "tokens" property. This is usually taken
care of in the recipes using the
add_tokens pre-processor.
JSON task format
classification binary
The classification interface lets you render content (text, text with spans, image or HTML) with a prominent label on top. The annotation decision you typically collect with this interface is whether the label applies to the content or not.
JSON task format (text)
JSON task format (text with spans)
JSON task format (image)
image binary
Images can be provided as file paths, URLs or base64-encoded data URIs –
basically, anything that can be rendered as an image. When using local paths,
keep in mind that modern browsers typically
block images from local paths
for security reasons. So you can either host the images with a
local web server using the ImageServer,
loader place them in an S3 bucket, or use the
fetch_media preprocessor to convert all
images from local paths and URIs to base64-encoded data URIs.
JSON task format (image URL)
JSON task format (base64 data URI)
Image tasks can also define a list of "spans", describing the boxes to draw on
the image. Each span can define a list of "points", consisting of [x, y]
coordinate tuples of the bounding box corners. While bounding boxes are usually
squares, Prodigy also supports polygon shapes. The coordinates should be
measured in pixels, relative to the original image. If a "label" is specified,
it will be rendered with the bounding box. You can also define an optional
"color". If none is set, Prodigy will pick one for you.
JSON task format (with boxes)
image_manual manual
The manual image interface lets you draw bounding boxes and polygon shapes on
the image. It’s optimized for fast object annotation, but not necessarily
high-precision image masking. You can click and drag or click and release to
draw boxes. Polygon shapes can also be closed by double-clicking when adding the
last point, similar to closing a shape in Photoshop or Illustrator. Clicking on
the label will select a shape so you can change the label, delete or resize it.
The interface also supports pre-populating the "spans" to pre-highlight
shapes.
JSON task format
JSON task format (with spans)
The interface lets you toggle the following display settings:
Draw a rectangular shape by clicking and dragging. Will store the x, y, width, height, center, points and label for each rect. | |
Draw a polygon shape by clicking on each point of the shape. To close the shape, double-click or click the first point. Will store the points and label. | |
New: v1.10 Draw a freehand shape by dragging your cursor. Will store the points and label. | |
| All labels | New: v1.10 Display all bounding box labels. If unchecked, labels are only shown on hover. |
Available config settings
Settings can be specified in the "config" returned by a recipe, or in your
prodigy.json.
| Name | Description | Default |
|---|---|---|
darken_image | Darken an image in image detection or segmentation mode for better visibility of the colored bounding boxes. For example, 0.25 will darken the image by 25%. | 0 |
show_bounding_box_center | Mark the center of a bounding box (rectangles only). | False |
image_manual_stroke_width | New: 1.10 Stroke width of the bounding box. Also used to calculate the size of the transform anchors. | 4 |
image_manual_font_size | New: 1.10 Font size of the bounding box label. | 16 |
image_manual_from_center | New: 1.10 Draw and resize rectangles from their center instead of the top left corner. | False |
image_manual_modes | New: 1.10 Annotation modes to support in the UI. Can be used to disable existing modes or change the order of the buttons. | ["rect", "polygon", "freehand"] |
image_manual_show_labels | New: 1.10 Default value of the toggle used to show and hide bounding box labels. | True |
image_manual_spans_key | New: 1.10.4 The JSON key used to store the image annotations. Can be customized if the interface is combined with a different UI using "spans", e.g. ner_manual. | "spans" |
image_manual_legacy | New: 1.10 Switch back to the previous manual interface used in Prodigy v1.9 and below. Will likely be deprecated in the future. | False |
label_style | New: 1.10.5 Style of label set. "list" for list of keyboard-accessible buttons or "dropdown". Previously available as ner_manual_label_style. | "list" |
audio binaryNew: 1.10
The audio interface lets you play and label audio and video files, and it’s
able to load URLs or base64-encoded data URIs. To load audio files from local
paths, you can use the Audio, AudioServer, Video or VideoServer
loaders or the
fetch_media preprocessor to convert all
audios from local paths and URIs to base64-encoded data URIs. Using
blocks, the audio interface can also be combined with other interfaces,
like a text_input for creating audio transcript, or the choice
UI to ask a multiple-choice question about the audio and/or certain audio
segments.
JSON task format (audio URL)
JSON task format (video URL)
JSON task format (base64 data URI)
Audio tasks can also define a list of "audio_spans" describing the regions to
pre-highlight. Each audio span consists of a start and end timestamp (in
seconds), as well as an optional "label" and an optional "color", e.g.
#ffd700 (without transparency). Clicking on a highlighted region will play
only that region and then pause.
JSON task format
audio_manual manualNew: 1.10
The manual audio interface lets you play and label audio and video files,
and it’s able to load URLs or base64-encoded data URIs. To load audio files from
local paths, you can use the Audio, AudioServer, Video or VideoServer
loaders or the
fetch_media preprocessor to convert all
audios from local paths and URIs to base64-encoded data URIs. You can also load
in data with existing "audio_spans" to pre-populate the interface. To create a
new region with the current label, you can click and drag. You can also drag and
drop existing regions to move them, or delete a region by clicking its
× button.
JSON task format
JSON task format
By default, videos are scaled to the full width of the annotation card. But you
can also provide a "width" and "height" value on the annotation task, which
will be used instead if available.
Available config settings
Settings can be specified in the "config" returned by a recipe, or in your
prodigy.json.
| Name | Description | Default |
|---|---|---|
show_audio_cursor | Show a line indicating the current cursor position. | True |
show_audio_cursor_time | Show a timestamp next to the audio cursor line (if enabled). | True |
show_audio_minimap | Show a smaller interactive minimap of the whole waveform below. | True |
show_audio_timeline | Show a timeline below the waveform. | False |
audio_autoplay | Auto-play the audio when a new task loads. | False |
audio_loop | Default value of loop setting (whether to loop audio). | False |
audio_bar_width | Width of a single waveform bar. Set to 0 for a more “traiditional” look. | 3 |
audio_bar_height | Height of the waveform bars. If greater than 1, the bars will be stretched. | 2 |
audio_bar_gap | Spacing between the waveform bars. | 2 |
audio_bar_radius | Corner radius of the waveform bars. | 2 |
audio_max_zoom | Maximum zoom level in pixels per second. | 5000 |
audio_rate | Audio playback speed. Lower number is slower, e.g. 0.5 will play at half the speed, 2.0 twice as fast. | 1.0 |
Here’s an example of a customized waveform with different settings and custom
label colors. The settings can be provided globally in the prodigy.json, or in
the "config" returned by a recipe.
prodigy.json (excerpt)
Advanced customizations New: 1.10.4
As of v1.10.4, Prodigy also exposes the underlying
WaveSurfer instance (the widget powering the
interactive audio player) via window.wavesurfer. This allows you to customize
the playback and implement your own controls using a
custom interface, a HTML block and optional custon
JavaScript. Here’s an example of two buttons to toggle the playback rate:
HTML block
choice manual
The choice interface can render text, entities, images or HTML. Each option is
structured like an individual annotation task and will receive a unique ID. You
can also choose to assign the IDs yourself. A choice task can also contain
other task properties – like a text, a label or spans. This content will
be rendered above the options.
Note that selecting options via keyboard shortcuts like 1 and
2 only works for 9 or less options. To use multiple choice instead of
single choice, you can set"choice_style": "multiple" in your Prodigy or recipe
config. You can also set "choice_auto_accept": true to automatically accept a
single-choice option when selecting it – without having to click the
accept button.
JSON task format
When you annotate, an "accept" key is added to the parent task, containing the
IDs of the selected options, as well as the "answer". This allows the
annotator to also reject or ignore tasks, for example if
they contain errors.
JSON task format (annotated)
An option can also contain a style key with an object of CSS properties in the
camel-cased JavaScript format,
mapping to their values:
JSON task format (with custom CSS)
Available config settings
Settings can be specified in the "config" returned by a recipe, or in your
prodigy.json.
| Name | Description | Default |
|---|---|---|
choice_style | Style of choice interface, “single” or “multiple”. | "single" |
choice_auto_accept | Automatically accept a single-choice option when selecting it – without having to click the accept button. | False |
html
The HTML interface can render any HTML, including embeds. It’s especially useful
for basic formatting and simple markup you can create programmatically. If
you’re looking to build complex or interactive custom interfaces, you probably
want to use the "html_template" config setting and
add custom CSS and JavaScript.
JSON task format
review New: 1.8
The review interface currently supports all annotation modes except
image_manual and compare. It will present one or more versions
of a given task, e.g. annotated in different sessions by different users,
display them with the session information and allow the reviewer to correct or
override the decision. The data requires a "view_id" setting that defines the
annotation UI to render the example with. The versions follow the same format as
regular annotation tasks. They can specify a list of "sessions", as well as a
boolean "default" key, marking that version as the default version to
pre-populate the review UI with (if the interface is a manual interface).
JSON task format (binary annotations)
JSON task format (manual annotations)
diff binary
The diff view is especially useful if you’re dealing with very subtle
comparisons, e.g. when evaluating spelling correction. The config setting
"diff_style" can be set to "chars", "words" or "sentences" to customize
how the diff is displayed. If you prefer the suggested changes, you can hit
accept, which corresponds to the "accept" or "added" object in
the data. In a real-world evaluation setting, you would typically randomize
which text gets shown in green or red, and then store that mapping with the
task, so you can resolve it later.
JSON task format (v1.10+)
JSON task format
Available config settings
Settings can be specified in the "config" returned by a recipe, or in your
prodigy.json.
| Name | Description | Default |
|---|---|---|
diff_style | Style to use for visual diffs. "words", "chars" or "sentences". | "words" |
compare binary
The compare interface, typically used via the compare recipe, supports
various types of content like text, HTML and images. It can receive an optional
input containing the original example, plus two examples to collect feedback
on. This format is normally used to compare two different outputs, and requires
you to feed in data from two different sources. The data is associated via its
IDs. To ensure unbiased annotation, Prodigy randomly decides which example to
return as the assumed correct answer, e.g. "accept". The mapping lets you
resolve those back to the original sources later on (A or B).
JSON task format
text_input manualNew: 1.9
The text input interface lets you collect free-form text input. The text is
added to the annotated task with a given key. You can also customize the
placeholder text, add an optional label to the field and make the input field a
text area with multiple rows. The text input interface is best used as a block
in the blocks interface, for example to ask the annotator to translate
a given text or leave feedback about a manual NER annotation task.
JSON task format
The "field_id" property lets you define a custom key that the collected text
is stored under. You can also pre-populate the field when you stream in the data
to set a default value. This can be very useful for tasks that need the
annotator to rewrite a text or perform other manual corrections.
JSON task format (annotated)
As of v1.10, Prodigy lets you provide an optional list of "field_suggestions"
that will be used to auto-suggest and auto-complete an answer if the user types
a letter or presses the ↓ key.
JSON task format
The text_input interface allows the following task properties:
| Name | Description | Default |
|---|---|---|
field_id | Key that the collected text is stored under. | "user_input" |
field_label | Label to display above the field. | |
field_placeholder | Placeholder to display if the input is empty.. | "Type here..." |
field_rows | Number of rows to display. If greater than 1, the field becomes a resizable textarea. | 1 |
field_autofocus | Autofocus the field when a new task loads. | false |
field_suggestions | New: 1.10 List of texts to auto-suggest when the user types or presses ↓. |
blocks New: 1.9
The blocks interface lets you combine different interfaces via the custom recipe
configuration. A list of "blocks" tells it which interfaces to use in which
order. The incoming data needs to match the format expected by the interfaces
used in the blocks. For instance, to render a ner_manual block, the
data needs to contain "text" and "tokens". For more details and examples,
see the documentation on custom blocks.
Recipe config
Example JSON task
llm_io New: 1.13.2
This interface lets you render the prompt sent to the LLM as well as the obtained response. This can be useful when debugging, but it may also provide an extra bit of context while annotating with an LLM in the loop.
This interface is typically used as a component of a blocks interface
together with an annotation interface that renders the output of the LLM.
Recipe config
Example JSON task for LLM-UI
If you use spacy-llm then you can use the prompt/response pair stored in the
spaCy doc object under the user_data key.
import spacyfrom dotenv import load_dotenv
# Just in case, make sure the environment variables are loadedload_dotenv()# Assemble a spaCy pipeline from the confignlp = assemble("config.cfg")# Use this pipeline as you would normallydoc = nlp("I know of a great pizza recipe with anchovis.")# You'll find the LLM response/prompt pair heredoc.user_dataHowever, be sure that you set the save_io property in your config file.
Otherwise the prompt and response won’t be stored.
Start of a spacy-llm config file