8000 deploy: aefbd3c8df29404216eccece8dc293de8b6d5143 · xarray-contrib/xarray-tutorial@eb67876 · GitHub
[go: up one dir, main page]

Skip to content

Commit eb67876

Browse files
committed
deploy: aefbd3c
1 parent a357237 commit eb67876

File tree

72 files changed

+8661
-126
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

72 files changed

+8661
-126
lines changed

404.html

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -203,8 +203,11 @@
203203
<li class="toctree-l1"><a class="reference internal" href="/intermediate/xarray_ecosystem.html">A Tour of Xarray Customizations</a></li>
204204
<li class="toctree-l1"><a class="reference internal" href="/intermediate/hvplot.html">Interactive plots using hvplot</a></li>
205205
<li class="toctree-l1"><a class="reference internal" href="/intermediate/cmip6-cloud.html">Accessing remote data stored on the cloud</a></li>
206-
<li class="toctree-l1 has-children"><a class="reference internal" href="/data_cleaning/data_cleaning.html">Data Tidying</a><input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/><label class="toctree-toggle" for="toctree-checkbox-7"><i class="fa-solid fa-chevron-down"></i></label><ul>
207-
<li class="toctree-l2"><a class="reference internal" href="/data_cleaning/ice_velocity.html">Re-organize InSAR ice velocity data</a></li>
206+
<li class="toctree-l1 has-children"><a class="reference internal" href="/intermediate/data_cleaning/05.1_intro.html">Data Tidying</a><input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/><label class="toctree-toggle" for="toctree-checkbox-7"><i class="fa-solid fa-chevron-down"></i></label><ul>
207+
<li class="toctree-l2"><a class="reference internal" href="/intermediate/data_cleaning/05.2_examples.html">Examples</a></li>
208+
<li class="toctree-l2"><a class="reference internal" href="/intermediate/data_cleaning/05.3_ice_velocity.html">Re-organize InSAR ice velocity data</a></li>
209+
<li class="toctree-l2"><a class="reference internal" href="/intermediate/data_cleaning/05.4_contributing.html">Contributing</a></li>
210+
<li class="toctree-l2"><a class="reference internal" href="/intermediate/data_cleaning/05.5_scipy_talk.html">Presentations</a></li>
208211
</ul>
209212
</li>
210213
</ul>

CONTRIBUTING.html

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -206,8 +206,11 @@
206206
<li class="toctree-l1"><a class="reference internal" href="intermediate/xarray_ecosystem.html">A Tour of Xarray Customizations</a></li>
207207
<li class="toctree-l1"><a class="reference internal" href="intermediate/hvplot.html">Interactive plots using hvplot</a></li>
208208
<li class="toctree-l1"><a class="reference internal" href="intermediate/cmip6-cloud.html">Accessing remote data stored on the cloud</a></li>
209-
<li class="toctree-l1 has-children"><a class="reference internal" href="data_cleaning/data_cleaning.html">Data Tidying</a><input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/><label class="toctree-toggle" for="toctree-checkbox-7"><i class="fa-solid fa-chevron-down"></i></label><ul>
210-
<li class="toctree-l2"><a class="reference internal" href="data_cleaning/ice_velocity.html">Re-organize InSAR ice velocity data</a></li>
209+
<li class="toctree-l1 has-children"><a class="reference internal" href="intermediate/data_cleaning/05.1_intro.html">Data Tidying</a><input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/><label class="toctree-toggle" for="toctree-checkbox-7"><i class="fa-solid fa-chevron-down"></i></label><ul>
210+
<li class="toctree-l2"><a class="reference internal" href="intermediate/data_cleaning/05.2_examples.html">Examples</a></li>
211+
<li class="toctree-l2"><a class="reference internal" href="intermediate/data_cleaning/05.3_ice_velocity.html">Re-organize InSAR ice velocity data</a></li>
212+
<li class="toctree-l2"><a class="reference internal" href="intermediate/data_cleaning/05.4_contributing.html">Contributing</a></li>
213+
<li class="toctree-l2"><a class="reference internal" href="intermediate/data_cleaning/05.5_scipy_talk.html">Presentations</a></li>
211214
</ul>
212215
</li>
213216
</ul>
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Data Tidying
2+
3+
Array data that are represented by Xarray objects are often multivariate, multi-dimensional, and very complex. Part of the beauty of Xarray is that it is adaptable and scalable to represent a large number of data structures. However, this can also introduce difficulty (especially for learning users) in arriving at a workable structure that will best suit one's analytical needs.
4+
5+
```{seealso}
6+
Look for examples [here](05.2_examples.md)
7+
```
8+
9+
This project is motivated by community sentiment and experiences that often, the hardest part of learning and teaching Xarray is teaching users how best to use Xarray conceptually. We hope to leverage the experiences of Xarray and geospatial data users to arrive at a unifying definition of 'tidy' data in this context and best practices for 'tidying' geospatial raster data represented by Xarray objects.
10+
11+
This page discusses common data ‘tidying’ steps and presents principles to keep in mind when organizing data in Xarray. We also point out helpful extensions to simplify and automate this process for specific dataset types like satellite imagery.
12+
13+
A great first step is familiarizing yourself with the [terminology](https://docs.xarray.dev/en/stable/user-guide/terminology.html) used in the Xarray ecosystem.
14+
15+
## A brief primer on tidy data
16+
17+
Tidy data was developed by Hadley Wickham for tabular datasets in the R programming language. Many resources comprehensively explain this concept and the ecosystem of tools built upon it. Below is a very brief explanation:
18+
19+
**Data tidying** is the process of structuring datasets to facilitate analysis. Wickham writes: "...tidy datasets are all alike, but every messy dataset is messy in its own way. Tidy datasets provide a standardized way to link the structure of a dataset (its physical layout) with its semantics (its meaning)" (Wickham, 2014).
20+
21+
### Tidy data principles for tabular datasets
22+
23+
The concept of [tidy data](https://vita.had.co.nz/papers/tidy-data.pdf) was developed by Hadley Wickham in the R programming language, and is a set of principles to guide facilitating tabular data for analysis.
24+
25+
{attribution="Wickham, 2014"}
26+
27+
> "Tidy datasets are all alike, but every messy dataset is messy in its own way."
28+
29+
Wickham defines three core principles of tidy data for tabular principles. They are:
30+
31+
1. Each variable forms an observation
32+
2. Each observation forms a row
33+
3. Each type of observational unit forms a table
34+
35+
## Imagining a 'tidy data' framework for gridded datasets
36+
37+
### Common use-case: Manipulating individual observations to an x-y-time datacube
38+
39+
Data downloaded or accessed from DAACs and other providers is often (for good reason) separated into temporal observations or spatial subsets. This minimizes the services that must be provided for different datasets and allows the user to access just the material that they need. However, most workflows will involve some sort of spatial and/or temporal investigation of an observable, which will usually require the analyst to arrange individual files into spatial mosaics and/or temporal cubes. In addition to being a source of duplicated effort and work, these steps also introduce decision-points that can be stumbling blocks for newer users. We hope a tidy framework for xarray will streamline the process of preparing data for analysis by providing specific expectations of what 'tidied' datasets look like as well as common patterns and tools to use to arrive at a tidy state.
40+
41+
## Tidy data principles for Xarray data structures
42+
43+
These are guidelines to keep in mind while you are organizing your data. For detailed definitions of the terms mentioned below (and more), check out Xarray's [Terminology page](https://docs.xarray.dev/en/stable/user-guide/terminology.html).
44+
45+
**1. Dimensions**
46+
47+
- Minimize the number of dimensional coordinates
48+
49+
**2. Coordinates**
50+
51+
- Non-dimensional coordinates can be numerous. Each should exist along one or multiple dimensions
52+
53+
**3. Data Variables**
54+
55+
- Data variables should be observables rather than contextual. Each should exist along one or multiple dimensions.
56+
57+
**4. Contextual information (metadata)**
58+
59+
- Metadata should only be stored as an attribute if it is static along the dimensions to which it is applied.
60+
- If metadata is dynamic, it should be stored as a coordinate variable.
61+
- Metadata `attrs` should be added such that dataset is self-describing (following CF-conventions)
62+
63+
**5. Variable, attribute naming**
64+
65+
- **Wherever possible, use cf-conventions for naming**
66+
- Variable names should be descriptive
67+
- Variable names should not contain information that belongs in a dimension or coordinate (ie. information stored in a variable name should be reduced to only the observable the variable describes.
68+
69+
**6. Make us of & work within the framework of other tools**
70+
71+
- Specification systems such as [CF](https://cfconventions.org/) and [STAC](https://stacspec.org/en), and related tools such as [Open Data Cube](https://www.opendatacube.org/), [PySTAC](https://pystac.readthedocs.io/en/stable/), [cf_xarray](https://cf-xarray.readthedocs.io/en/latest/),[stackstac](https://stackstac.readthedocs.io/en/latest/) and more make tidying possible and smoother, especially with large, cloud-optimized datasets.
72+
-
73+
74+
## Other guidelines and rules of thumb
75+
76+
- Avoid storing important data in filenames
77+
- Non-descriptive variable names can create + perpetuate confusion
78+
- Missing coordinate information makes datasets harder to use
79+
- Elements of a dataset's 'shape'/structure can sometimes be embedded in variable names; this will complicate subsequent analysis
80+
81+
## Contributing
82+
83+
We would love your help and engagement on this project! If you have a dataset that you've worked with that felt particularly messy, or one with steps you find yourself thinking back to as you work with new datasets, consider submitting it as an example! If you have input on tidy principles, please feel free to raise an issue.
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Examples
2+
3+
This page contains examples of 'tidying' datasets. If you have an example you'd like to submit, or an example of an anti-pattern, please raise an issue !
4+
5+
## 1. Aquarius
6+
7+
This is an example of tidying a dataset comprised of locally downloaded files. Aquarius is a sea surface salinity dataset produced by NASA and accessed as network Common Data Form (NetCDF) files.
8+
You can find this example [here](https://gist.github.com/dcherian/66269bc2b36c2bc427897590d08472d7). This example focuses on data access steps and organizing data into a workable data cube.
9+
10+
## 2. ASE Ice Velocity
11+
12+
Already integrated into the Xarray tutorial, this examples uses an ice velocity dataset derived from synthetic aperture radar imagery. You can find it [here](05.3_ice_velocity.ipynb). This example focuses on data access steps and organizing data into a workable data cube.
13+
14+
## 3. Harmonized Landsat-Sentinel
15+
16+
This [example](https://nbviewer.org/gist/scottyhq/efd583d66999ce8f6e8bcefa81545b8d) features cloud-optimized data that does not need to be downloaded locally. Here, package such as [`odc-stac`](https://github.com/opendatacube/odc-stac) are used to accomplish much of the initial tidying (assembling an x,y,time cube). However, this example shows that there is frequently additional formatting required to make a dataset analysis ready.

0 commit comments

Comments
 (0)
0