8000 Update to v0.2 (nested storage) with FSStore by joshmoore · Pull Request #75 · ome/ome-zarr-py · GitHub
[go: up one dir, main page]

Skip to content

Update to v0.2 (nested storage) with FSStore #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 24, 2021
Merged

Conversation

joshmoore
Copy link
Member

With the improvements to zarr.store.FSStore, all URLs can
be handled uniformly without needing to use requests. A
next step may in fact be to remove the ZarrLocation types.

This change does not yet support v0.2 and v0.1 at the same
time.

With the improvements to zarr.store.FSStore, all URLs can
be handled uniformly without needing to use `requests`. A
next step may in fact be to remove the ZarrLocation types.

This change does not yet support v0.2 and v0.1 at the same
time.
@codecov
Copy link
codecov bot commented Apr 19, 2021

Codecov Report

Merging #75 (ce7b807) into master (c3f641d) will increase coverage by 0.09%.
The diff coverage is 77.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #75      +/-   ##
==========================================
+ Coverage   66.85%   66.95%   +0.09%     
==========================================
  Files          11       11              
  Lines        1056     1053       -3     
==========================================
- Hits          706      705       -1     
+ Misses        350      348       -2     
Impacted Files Coverage Δ
ome_zarr/utils.py 91.56% <0.00%> (ø)
ome_zarr/scale.py 54.28% <25.00%> (-0.62%) ⬇️
ome_zarr/io.py 78.76% <74.32%> (+0.98%) ⬆️
ome_zarr/data.py 98.68% <100.00%> (+0.03%) ⬆️
ome_zarr/reader.py 59.39% <100.00%> (+0.52%) ⬆️
ome_zarr/writer.py 93.75% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3f641d...ce7b807. Read the comment docs.

With the introduction of FSStore, nearly all of the local vs remote
logic is now happening behind the scenes. The additional benefit is
that this should appease the Windows implementation which was still
failing with the previous commit.
@joshmoore
Copy link
Member Author

Note: this requires Zarr 2.7.1. See zarr-developers/zarr-python#718

@property
def root_attrs(self) -> JSONDict:
"""Return the contents of the zattrs file."""
return dict(self.__metadata)

def load(self, subpath: str = "") -> da.core.Array:
def load(self, subpath: str = "", nested: bool = False) -> da.core.Array:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going forward, we expect all future specs will be nested, so would it make sense for the default nested: bool = True?

@will-moore
Copy link
Member

I must be doing something wrong here:

$ which bioformats2raw
/Users/wmoore/Desktop/bioformats2raw-0.3.0-rc3/bin/bioformats2raw
$ bioformats2raw /Users/wmoore/Desktop/images/DVs/438CTR_01_4_R3D_D3D.dv 438CTR_01_4_R3D_D3D_nested
$ ls 438CTR_01_4_R3D_D3D_nested/0/0/0/0/0/0/0
438CTR_01_4_R3D_D3D_nested/0/0/0/0/0/0/0
$ napari 438CTR_01_4_R3D_D3D_nested/0

Opens napari image with correct dimensions, but all planes are black.

Same for omero-cli-zarr with nested branch. Will investigate...

@joshmoore
Copy link
Member Author

Which zarr version? You'll need 2.7.1

@will-moore
Copy link
Member

Ah, thanks - That would have taken me a while to figure out. I had zarr 2.7.0.
All working fine with zarr 2.7.1.

@will-moore
Copy link
Member

The SPW images exported via omero zarr export works in napari:

$ napari https://minio-dev.openmicroscopy.org/idr/idr0001-graml-sysgro/pr59_nested/2551.zarr/A/5/0

But not the plate itself (visible in vizarr with nested PR):
https://deploy-preview-85--vizarr.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/idr0001-graml-sysgro/pr59_nested/2551.zarr

Trying to open a nested plate with this PR (same error with the 'nested' plate above, or an older v0.1 plate):

$ napari https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/422.zarr
10:51:46 ERROR PluginError: Error in plugin 'ome_zarr', hook 'napari_get_reader'
  Cause was: IndexError('list index out of range')
    in file: /Users/wmoore/Desktop/ZARR/ome-zarr-py/ome_zarr/reader.py
    at line: 513
     author: The Open Microscopy Team
    package: ome-zarr
        url: https://github.com/ome/ome-zarr-py
    version: 0.0.18.dev0

This also fails using the current master branch (c3f641d) but with a different error:

$ napari https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/422.zarr
10:50:25 ERROR PluginError: Error in plugin 'ome_zarr', hook 'napari_get_reader'
  Cause was: ArrayNotFoundError("array not found at path %r' ''")
    in file: /opt/anaconda3/envs/napari/lib/python3.9/site-packages/zarr/core.py
    at line: 186
     author: The Open Microscopy Team
    package: ome-zarr
        url: https://github.com/ome/ome-zarr-py
    version: 0.0.18.dev0

And in fact it's not working for any of the older versions of ome-zarr-py that I've tried, so I don't know when it last worked (or what version of napari etc).

@joshmoore
Copy link
Member Author
    def get_numpy_type(self, image_node: Node) -> np.dtype:
        return image_node.data[0].dtype

@joshmoore
8F31 Copy link
Member Author

Filed an issue for the dangling issue. Merging and releasing.

@joshmoore joshmoore merged commit d9d183d into ome:master Apr 24, 2021
@joshmoore joshmoore deleted the nested branch April 24, 2021 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
0