8000 update release notes and tutorial · zarr-developers/zarr-python@cfadc1d · GitHub
[go: up one dir, main page]

Skip to content

Commit cfadc1d

Browse files
committed
update release notes and tutorial
1 parent 88cd394 commit cfadc1d

File tree

2 files changed

+48
-15
lines changed

2 files changed

+48
-15
lines changed

docs/release.rst

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,27 @@ Unreleased
99
Bug fixes
1010
~~~~~~~~~
1111

12+
* Fix bug that made it impossible to create an ``FSStore`` on unlistable filesystems
13+
(e.g. some HTTP servers).
14+
By :user:`Ryan Abernathey <rabernat>`; :issue:`993`.
15+
16+
Enhancements
17+
~~~~~~~~~~~~
18+
19+
* **Create FSStore from an existing fsspec filesystem**. If you have created
20+
an fsspec filesystem outside of Zarr, you can now pass it as a keyword
21+
argument to ``FSStore``.
22+
By :user:`Ryan Abernathey <rabernat>`.
23+
24+
25+
.. _release_2.11.1:
26+
27+
2.11.3
28+
------
29+
30+
Bug fixes
31+
~~~~~~~~~
32+
1233
* Changes the default value of ``write_empty_chunks`` to ``True`` to prevent
1334
unanticipated data losses when the data types do not have a proper default
1435
value when empty chunks are read back in.
@@ -319,7 +340,7 @@ Bug fixes
319340

320341
* FSStore: default to normalize_keys=False
321342
By :user:`Josh Moore <joshmoore>`; :issue:`755`.
322-
* ABSStore: compatibility with ``azure.storage.python>=12``
343+
* ABSStore: compatibility with ``azure.storage.python>=12``
323344
By :user:`Tom Augspurger <tomaugspurger>`; :issue:`618`
324345

325346

@@ -484,7 +505,7 @@ This release will be the last to support Python 3.5, next version of Zarr will b
484505

485506
* `DirectoryStore` now uses `os.scandir`, which should make listing large store
486507
faster, :issue:`563`
487-
508+
488509
* Remove a few remaining Python 2-isms.
489510
By :user:`Poruri Sai Rahul <rahulporuri>`; :issue:`393`.
490511

@@ -504,7 +525,7 @@ This release will be the last to support Python 3.5, next version of Zarr will b
504525
``zarr.errors`` have been replaced by ``ValueError`` subclasses. The corresponding
505526
``err_*`` function have been removed. :issue:`590`, :issue:`614`)
506527

507-
* Improve consistency of terminology regarding arrays and datasets in the
528+
* Improve consistency of terminology regarding arrays and datasets in the
508529
documentation.
509530
By :user:`Josh Moore <joshmoore>`; :issue:`571`.
510531

docs/tutorial.rst

Lines changed: 24 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -758,7 +758,7 @@ databases. The :class:`zarr.storage.RedisStore` class interfaces `Redis <https:/
758758
(an in memory data structure store), and the :class:`zarr.storage.MongoDB` class interfaces
759759
with `MongoDB <https://www.mongodb.com/>`_ (an object oriented NoSQL database). These stores
760760
respectively require the `redis-py <https://redis-py.readthedocs.io>`_ and
761-
`pymongo <https://api.mongodb.com/python/current/>`_ packages to be installed.
761+
`pymongo <https://api.mongodb.com/python/current/>`_ packages to be installed.
762762

763763
For compatibility with the `N5 <https://github.com/saalfeldlab/n5>`_ data format, Zarr also provides
764764
an N5 backend (this is currently an experimental feature). Similar to the zip storage class, an
@@ -897,6 +897,18 @@ The second invocation here will be much faster. Note that the ``storage_options`
897897
have become more complex here, to account for the two parts of the supplied
898898
URL.
899899

900+
It is also possible to initialize the filesytem outside of Zarr and then pass
901+
it through. This requires creating an :class:`zarr.storage.FSStore` object
902+
explicitly. For example::
903+
904+
>>> import s3fs * doctest: +SKIP
905+
>>> fs = s3fs.S3FileSystem(anon=True) # doctest: +SKIP
906+
>>> store = zarr.storage.FSStore('/zarr-demo/store', fs=fs) # doctest: +SKIP
907+
>>> g = zarr.open_group(store) # doctest: +SKIP
908+
909+
This is useful in cases where you want to also use the same fsspec filesystem object
910+
separately from Zarr.
911+
900912
.. _fsspec: https://filesystem-spec.readthedocs.io/en/latest/
901913

902914
.. _supported by fsspec: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
@@ -1306,18 +1318,18 @@ filters (e.g., byte-shuffle) have been applied.
13061318

13071319
Empty chunks
13081320
~~~~~~~~~~~~
1309-
1321+
13101322
As of version 2.11, it is possible to configure how Zarr handles the storage of
13111323
chunks that are "empty" (i.e., every element in the chunk is equal to the array's fill value).
1312-
When creating an array with ``write_empty_chunks=False``,
1324+
When creating an array with ``write_empty_chunks=False``,
13131325
Zarr will check whether a chunk is empty before compression and storage. If a chunk is empty,
1314-
then Zarr does not store it, and instead deletes the chunk from storage
1315-
if the chunk had been previously stored.
1326+
then Zarr does not store it, and instead deletes the chunk from storage
1327+
if the chunk had been previously stored.
13161328

1317-
This optimization prevents storing redundant objects and can speed up reads, but the cost is
1318-
added computation during array writes, since the contents of
1319-
each chunk must be compared to the fill value, and these advantages are contingent on the content of the array.
1320-
If you know that your data will form chunks that are almost always non-empty, then there is no advantage to the optimization described above.
1329+
This optimization prevents storing redundant objects and can speed up reads, but the cost is
1330+
added computation during array writes, since the contents of
1331+
each chunk must be compared to the fill value, and these advantages are contingent on the content of the array.
1332+
If you know that your data will form chunks that are almost always non-empty, then there is no advantage to the optimization described above.
13211333
In this case, creating an array with ``write_empty_chunks=True`` (the default) will instruct Zarr to write every chunk without checking for emptiness.
13221334

13231335
The following example illustrates the effect of the ``write_empty_chunks`` flag on
@@ -1329,7 +1341,7 @@ the time required to write an array with different values.::
13291341
>>> from tempfile import TemporaryDirectory
13301342
>>> def timed_write(write_empty_chunks):
13311343
... """
1332-
... Measure the time required and number of objects created when writing
1344+
... Measure the time required and number of objects created when writing
13331345
... to a Zarr array with random ints or fill value.
13341346
... """
13351347
... chunks = (8192,)
@@ -1368,8 +1380,8 @@ the time required to write an array with different values.::
13681380
Random Data: 0.1359s, 1024 objects stored
13691381
Empty Data: 0.0301s, 0 objects stored
13701382

1371-
In this example, writing random data is slightly slower with ``write_empty_chunks=True``,
1372-
but writing empty data is substantially faster and generates far fewer objects in storage.
1383+
In this example, writing random data is slightly slower with ``write_empty_chunks=True``,
1384+
but writing empty data is substantially faster and generates far fewer objects in storage.
13731385

13741386
.. _tutorial_rechunking:
13751387

0 commit comments

Comments
 (0)
0