Add example using a Zarr array #191
Conversation
|
As a note, this works reasonably fast using this in-memory dataset. @sofroniewn and I looked at a dataset stored in Zarr on disk, but found it less responsive than desired. There are any number of potential reasons for that. It’s probably worth benchmarking to identify the exact cause. @jni mentioned one cause may be the disk’s read speed, which could be significant here. Other issues may be performance issues in Zarr; however, these presumably would impact the in-memory case as well, but don’t seem to. Besides benchmarking some other things to try would be using Zarr’s If there are still problems loading data quickly, would suggest engaging with the Zarr community. There’s a meta issue ( zarr-developers/zarr-python#382 ), which discusses and links to a number of potential improvements in the data loading process. This involves a few different caches at different stages for different purposes. |
|
More side notes:
Thanks for the links to the community, @jakirkham! |
|
@jni @jakirkham I'm making great progress with the on-disk zarr example. Unsurprisingly it is all about finding the right chunk size and making sure the chunking dimensions align with our slicing dimensions. Any order of the dimensions is fine, but things just have to match up. I havn't had to make any changes to the code apart from adding an optional clim_range argument so as to avoid the It's very exciting! Thanks again @jakirkham for your help - this would have taken A LOT longer for me to figure out without your help |
Description
Creates a large Zarr array that is mostly zero except for a small square of 1's in the middle. Demonstrates how one might combine Dask and Zarr to move through larger data in the viewer.
Type of change
References
How has this been tested?
Final checklist: