Best Practices in Publishing Sampling-event data

Darwin Core Archive (DwC-A) is an informatics data standard that makes use of the Darwin Core terms to produce a single, self-contained dataset for checklist data. The collection of files in an archive form a self-contained dataset, which can be provided as a single compressed (Zip or GZIP) file. A dataset is composed of a descriptive metadata document and a set of one or more data files. For more information about DwC-A refer to the Darwin Core Archives How-to Guide.

Documenting the provenance and scope of datasets is required in order to publish sampling-event data through the GBIF network. Dataset documentation is referred to as ‘resource metadata’ and enables users to evaluate the fitness-for-use of a dataset. It may describe the sampling methodologies used for its compilation, and the individuals and organizations involved in its creation and management. Metadata is shared in a Darwin Core Archive as an XML document. GBIF provides a metadata profile for sampling-event datasets based on the Ecological Metadata Language. A How-to guide describes all the options for describing a sampling-event dataset using this format. See GBIF Metadata Profile – How-to Guide.

The Darwin Core Archive format provides the structural framework for publishing sampling-event data. Darwin Core Archives consist of a series of one or more text files, in standard comma- or tab-delimited format. The files are logically arranged in a star-like manner with one core file, listing the sampling events (sampling protocol, sample size, location, etc.) surrounded by a number of ‘extensions’, that describe related data types (such as species occurrences, measurements or facts related to the sampling event, etc). Links between core and extension records are made using an event identifier (eventID) data element. In this way, many extension records can exist for each single core event record. This “star-schema” provides a simple relational data model that supports many types of annotations that are common to sampling-event datasets.

An alternative way to encode sampling-event data is listing the species occurrence in the core file, surrounded by a number of extensions, that describe the related data types (such as measurements relating to the species occurrences, etc). Note listing sampling events in the core file is preferable if a plot or site is the main focus of the study.

Be aware that the current DwC-A star schema such as does have limitations. For example, it does not allow to link measurements and facts to both events and occurrences in the same dataset. Discussion on this prototype extension are taking place on GitHub but there is no solution available yet.

Please check the data quality requirements for sampling events. Note that if you generate your Darwin Core Archive manually, you might need to add a recordID field containing unique identifiers. (This field is generated automatically if you are using an IPT.) For more information on the structure of a Darwin Core Archive, please check the TDWG Darwin Core text guide. You can always check your archive with the GBIF data validator.

For ease in understanding, we may use the terms field in this guide to refer to the Darwin Core set of terms in the sampling-event publishing profile to which a users data will be mapped. For example, we will refer to the use of the dwc:scientificName field when referring to the Darwin Core term, scientificName.

The following Darwin Core fields store the sample size of a sampling event:

The value of sampleSizeValue is a number and must have a corresponding sampleSizeUnit. The value of sampleSizeUnit should be restricted to use only SI units/derived units or other non-SI units accepted for use within the SI (e.g. minute, hour, day, litre) as per the Unit of Measurement Vocabulary. Examples are given in Table 1 below.

You may represent a sampling area by using an appropriate WKT shape or latitude/longitude point location. Done correctly, the direction sampling was carried out can also be derived. For example, an ocean trawl line represented using a WKT shape LINESTRING allows the direction of the trawl to be determined based on the standard notation for writing the start and end points.

The following Darwin Core fields are also required to be used as a pair:

Table 2 lists some example values. The value of organismQuantity is a number or enumeration, e.g., “27” for an organismQuantityType “individuals”, “12.5” for an organismQuantityType “%biomass”, or “r” for an organismQuantityType “BraunBlanquetScale”. The value of organismQuantityType (i.e., the entity being measured) is expected to be drawn from a small controlled vocabulary with terms such as “Individuals”, “%Biomass”, “%Biovolume”, “%Species”, “%Coverage”, “BraunBlanquetScale”, “DominScale”. Examples when combined with organismQuantity values: “+” on DominScale; “5” on BraunBlanquetScale; “45” for %Biomass.

Each event is uniquely identified using dwc:eventID and occasionally dwc:parentEventID. Although the type and format of identifier is arbitrary we recommend that publishers choose persistent globally unique identifier. In the absence of a GUID, publishers may reuse the original fieldNumber.

Make sure to reuse existing stable identifiers and do not create a new identifier for an event when one is already declared.

Sampling events can be related to each other (e.g., nested samples) via a common parent identifier. For example, several sub-sampling events within a Whittaker Plot each with their own eventID (e.g., “A1:1”, “A1:2”) would share a common parentEventID (e.g., “A1”) thus enabling them to be linked together easily (see Table 4 and Figure 3).

Further information on the nature of the relationship (e.g. part of a monitoring series) can be described in the project section of the accompanying metadata.

You may also refer to the following FAQ.

Refer to the following FAQ.

You may choose to include supplementary media in order to make the data easier to interpret. For example for vegetation data, it is helpful to include a link to the original scanned relevé sheet when interpreting the data.

The files associated have to be hosted on an external server and linked to the occurrence through the dwc:associatedMedia and dwc:associatedReferences. These files may be images, texts or a combination of both as long as the format type is specified. JPG, PNG, etc. images will be visible as thumbnails while PDFs will appear as clickable links.

A number of fields require the usage of unique identifiers: dwc:occurrenceID, dwc:eventID, dwc:organismID and dwc:locationID.

As mentioned previously, although no particular format is enforced, we advise publishers to use Global Unique IDentifiers (GUIDs). There are a few online services which can provide such identifiers. For example, it is possible to use http://www.geonames.org/ to find (or even generate new) identifiers for dwc:locationID, e.g. http://sws.geonames.org/10793757/ is a GUID for a lake in Greenland.

If your dataset contains sensitive species, there are several ways to handle it:

Although verbatim data and descriptions are not visible on the GBIF.org web interface, they are made available to the community through downloads. When entering verbatim description, make sure to link them to the original event or occurrence. For example, the ID or code given to the original event should be entered into dwc:fieldNumber; the ID or code given to the original occurrence observation should be entered into dwc:recordNumber.

Data produced from a large sampling project should be published as a single dataset if possible. If you must publish multiple datasets, we encourage linking them using a common project identifier in the metadata.

Sampling events provide better documentation and benefit both the scientific community and policy makers (read more). We encourage strongly the republishing of occurrence data as sampling-event data when possible.

In order to do so, you should create a new sampling-event dataset and send an email to GBIF’s Help Desk (helpdesk@gbif.org). In this email, you should provide the UUIDs of both the occurrence dataset and the new dataset. We will then be able to link the first dataset to the newest one before de-indexing it thereby avoiding occurrence duplication and preserving citations.

If your dataset contains continuous monitoring of live individuals, such as bird tracking data, you can use dwc:organismID to store the ID of the individual being tracked. In addition to that, you should represent each individual being tracked as a single event.

If possible, we recommend tracking all the issues related to a given dataset using an issue management system such as the one provided by GitHub.

Ideally, custom scripts and programs used to transform data should be made publicly available on GitHub. Other publishers will benefit from accessing these scripts along with detailed instructions on how to use them.

Some datasets may come from the same research project or may be related in certain context. The current recommendation is to link them using a project identifier.

You can make your dataset easier to interpret by including links to related published works such as journal articles, project notes, thesis, etc,. in the bibliographic citation or in the external link part of the metadata.

Core (Event) table

Extension (Occurrence) table

Explanation

Ephemera danica : A total of 14 individuals from 1.25 square meters were obtained in this sampling event. The derived individuals per sq meter count is 11.2 (14/1.25).

Core (Event) table

Extension (Occurrence) table

Extension (Measuremenr-or-Fact) table

Explanation

Abra ovata : A total of 57 individuals from 0.04 square meters were obtained in sampling event IA1.

Each event can also have measurements or facts associated with it, e.g., environmental measurements like sediment temperature and redox potential (Eh).

Note that this example is based on Dutch Vegetation Database (LVD) previous version republished as sampling-event dataset. The Relevé extension underwent significant changes following the publication of the primer. For more information about LVD and the data model for vegetation sampling-event data see: https://gbif.blogspot.com/2016/07/probably-turbovegs-best-kept-secret.html

Extension (Occurrence) table

Extension (Relevé) table

Explanation

Acer psuedoplatanus : In the 100 sq meters surveyed, the abundance of the species was reported as “r” on the Braun Blanquet scale.

Additional vegetation plot measurements such as vegetation community type (syntaxon) % coverage values that are typical of TurboVeg type databases are captured in a Relevé (vegetation-plot) extension.

Core (Event) table

Extension (Occurrence) table

Explanation

Opisthograptis luteolata : 11 individuals were observed over the sampling period of 16 days. The derived number of individuals per day is 0.68 (11/16).

Core (Event) table

Extension (Occurrence) table

Extension (Measurement-or-Fact) table

Explanation

Pieras rapae : A total of 1 individual from 250 sq metres was obtained in this sampling event. Several environmental measurements (e.g., temperature, wind speed, cloudiness) are included in a measurement-or-facts extension.

Core (Event) table

Extension (Occurrence) table

Explanation

Aulostomus chinensis : A total of 4 individuals from 500 sq metres were obtained in this sampling event.

The Event core elements are mainly drawn from the DwC classes Event, Location and Geological Context (Table 3). The Occurrence extension elements are drawn from the Occurrence, Taxon and Identification classes. For reasons of consistency, the Occurrence extension includes all terms found in the Occurrence core. Thus Event, Location and Geological Context terms are also listed for the Occurrence extension but are actually redundant. Note the IPT hides redundant terms by default when mapping for the user’s convenience.

The "+" symbol indicates proposed new terms not yet ratified.

Best Practices in Publishing Sampling-event data - planned additions and notes for revision