8000 Handle legacy nested datasets (with tests) (#850) · benjeffery/zarr-python@d1dc987 · GitHub
[go: up one dir, main page]
Skip to content

Commit d1dc987

Browse files
joshmoorejoshmoore
authored
Handle legacy nested datasets (with tests) (zarr-developers#850)
* Fix zarr-developers#840 * Add legacy tests (See zarr-developers#840) Each fixture (flat & nested) should have a version which does not include any new metadata. * Fix PEP8 issues * Try dropping editable installs see: pypa/pip#10573 * Handle case of store being a dict * Exclude unexpected failure from code coverage * Add 2.10.2 release notes
1 parent 4f8cb35 commit d1dc987

File tree

12 files changed

+128
-20
lines changed

12 files changed

+128
-20
lines changed

‎.github/workflows/minimal.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,5 +24,5 @@ jobs:
2424
shell: "bash -l {0}"
2525
run: |
2626
conda activate minimal
27-
python -m pip install -e .
27+
python -m pip install .
2828
pytest -svx

‎.github/workflows/python-package.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ jobs:
5757
python -m pip install --upgrade pip
5858
python -m pip install -U pip setuptools wheel codecov line_profiler
5959
python -m pip install -rrequirements_dev_minimal.txt numpy${{ matrix.numpy_version}} -rrequirements_dev_optional.txt pymongo redis
60-
python -m pip install -e .
60+
python -m pip install .
6161
python -m pip freeze
6262
- name: Tests
6363
shell: "bash -l {0}"

‎.github/workflows/windows-testing.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ jobs:
3939
python -m pip install --upgrade pip
4040
python -m pip install -U pip setuptools wheel
4141
python -m pip install -r requirements_dev_numpy.txt -r requirements_dev_minimal.txt -r requirements_dev_optional.txt
42-
python -m pip install -e .
42+
python -m pip install .
4343
python -m pip freeze
4444
npm install -g azurite
4545
- name: Run Tests

‎docs/release.rst

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,17 @@ Release notes
66
Unreleased
77
----------
88

9+
.. _release_2.10.2:
10+
11+
2.10.2
12+
------
13+
14+
Bug fixes
15+
~~~~~~~~~
16+
17+
* Fix NestedDirectoryStore datasets without dimension_separator metadata.
18+
By :user:`Josh Moore <joshmoore>`; :issue:`850`.
19+
920
.. _release_2.10.1:
1021

1122
2.10.1

‎fixture/flat/.zarray

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
"id": "blosc",
1111
"shuffle": 1
1212
},
13+
"dimension_separator": ".",
1314
"dtype": "<i8",
1415
"fill_value": 0,
1516
"filters": null,
@@ -19,4 +20,4 @@
1920
2
2021
],
2122
"zarr_format": 2
22-
}
23+
}

‎fixture/flat_legacy/.zarray

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"chunks": [
3+
2,
4+
2
5+
],
6+
"compressor": {
7+
"blocksize": 0,
8+
"clevel": 5,
9+
"cname": "lz4",
10+
"id": "blosc",
11+
"shuffle": 1
12+
},
13+
"dtype": "<i8",
14+
"fill_value": 0,
15+
"filters": null,
16+
"order": "C",
17+
"shape": [
18+
2,
19+
2
20+
],
21+
"zarr_format": 2
22+
}

‎fixture/flat_legacy/0.0

48 Bytes
Binary file not shown.

‎fixture/nested_legacy/.zarray

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"chunks": [
3+
2,
4+
2
5+
],
6+
"compressor": {
7+
"blocksize": 0,
8+
"clevel": 5,
9+
"cname": "lz4",
10+
"id": "blosc",
11+
"shuffle": 1
12+
},
13+
"dtype": "<i8",
14+
"fill_value": 0,
15+
"filters": null,
16+
"order": "C",
17+
"shape": [
18+
2,
19+
2
20+
],
21+
"zarr_format": 2
22+
}

‎fixture/nested_legacy/0/0

48 Bytes
Binary file not shown.

‎zarr/core.py

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,18 @@ def _load_metadata_nosync(self):
195195
self._dtype = meta['dtype']
196196
self._fill_value = meta['fill_value']
197197
self._order = meta['order']
198-
self._dimension_separator = meta.get('dimension_separator', '.')
198+
199+
dimension_separator = meta.get('dimension_separator', None)
200+
if dimension_separator is None:
201+
try:
202+
dimension_separator = self._store._dimension_separator
203+
except (AttributeError, KeyError):
204+
pass
205+
206+
# Fallback for any stores which do not choose a default
207+
if dimension_separator is None:
208+
dimension_separator = "."
209+
self._dimension_separator = dimension_separator
199210

200211
# setup compressor
201212
config = meta['compressor']

0 commit comments

Comments
 (0)
0