8000 avoid race condition during chunk write (#327) · zarr-developers/zarr-python@94f7a8d · GitHub
[go: up one dir, main page]
Skip to content

Commit 94f7a8d

Browse files
sbalmeralimanfoo
sbalmer
authored and
alimanfoo
committed
avoid race condition during chunk write (#327)
* avoid race condition during chunk write When the chunk file is first removed before the new version is moved into place, racing reads may encounter a missing chunk. Using rename() or replace() without remove() avoids the issue on Posix-Systems as the methods are atomic. The fallback of remove() -> rename() is included for Windows pre Python 3.3. Fixes #263 * move feature-detection to init-time so it's not repeated on every write * use pyosreplace to get atomic replace() on Windows * disable coverage count for legacy branches * Use conditional instead of env marker Because the env markers didn't work. Just guessing at this point. * add pyosreplace package to requirements * release notes [ci skip]
1 parent c4427a4 commit 94f7a8d

File tree

5 files changed

+37
-17
lines changed

5 files changed

+37
-17
lines changed

‎docs/release.rst

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,22 +19,28 @@ Enhancements
1919
* Support has been added for structured arrays with sub-array shape and/or nested fields. By
2020
:user:`Tarik Onalan <onalant>`, :issue:`111`, :issue:`296`.
2121

22-
Maintenance
23-
~~~~~~~~~~~
22+
Bug fixes
23+
~~~~~~~~~
24+
25+
* The implementation of the :class:`zarr.storage.DirectoryStore` class has been modified to
26+
ensure that writes are atomic and there are no race conditions where a chunk might appear
27+
transiently missing during a write operation. By :user:`sbalmer <sbalmer>`, :issue:`327`,
28+
:issue:`263`.
2429

2530
* The required version of the `numcodecs <http://numcodecs.rtfd.io>`_ package has been upgraded
2631
to 0.6.2, which has enabled some code simplification and fixes a failing test involving
2732
msgpack encoding. By :user:`John Kirkham <jakirkham>`, :issue:`352`, :issue:`355`,
2833
:issue:`324`.
2934

30-
* CI and test environments have been upgraded to include Python 3.7, drop Python 3.4, and
31-
upgrade all pinned package requirements. :issue:`308`.
32-
3335
* Failing tests related to pickling/unpickling have been fixed. By :user:`Ryan Williams <ryan-williams>`,
3436
:issue:`273`, :issue:`308`.
3537

36-
Acknowledgments
37-
~~~~~~~~~~~~~~~
38+
Maintenance
39+
~~~~~~~~~~~
40+
41+
* CI and test environments have been upgraded to include Python 3.7, drop Python 3.4, and
42+
upgrade all pinned package requirements. :issue:`308`.
43+
3844

3945
.. _release_2.2.0:
4046

‎requirements.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@ fasteners
33
numcodecs
44
numpy
55
pytest
6+
pyosreplace; python_version < '3.3' and sys.platform == 'win32'

‎requirements_dev.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
asciitree==0.3.3
22
fasteners==0.14.1
33
numcodecs==0.6.2
4+
pyosreplace==0.1; python_version < '3.3' and sys.platform == 'win32'

‎setup.py

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# -*- coding: utf-8 -*-
22
from __future__ import absolute_import, print_function, division
33
from setuptools import setup
4+
import sys
45

56

67
DESCRIPTION = 'An implementation of chunked, compressed, ' \
@@ -9,6 +10,16 @@
910
with open('README.rst') as f:
1011
LONG_DESCRIPTION = f.read()
1112

13+
dependencies = [
14+
'asciitree',
15+
'numpy>=1.7',
16+
'fasteners',
17+
'numcodecs>=0.6.2',
18+
]
19+
20+
if sys.version_info < (3, 3) and sys.platform == "win32":
21+
dependencies.append('pyosreplace')
22+
1223
setup(
1324
name='zarr',
1425
description=DESCRIPTION,
@@ -22,12 +33,7 @@
2233
'setuptools>18.0',
2334
'setuptools-scm>1.5.4'
2435
],
25-
install_requires=[
26-
'asciitree',
27-
'numpy>=1.7',
28-
'fasteners',
29-
'numcodecs>=0.6.2',
30-
],
36+
install_requires=dependencies,
3137
package_dir={'': '.'},
3238
packages=['zarr', 'zarr.tests'],
3339
classifiers=[
@@ -42,7 +48,6 @@
4248
'Programming Language :: Python :: 2',
4349
'Programming Language :: Python :: 2.7',
4450
'Programming Language :: Python :: 3',
45-
'Programming Language :: Python :: 3.4',
4651
'Programming Language :: Python :: 3.5',
4752
'Programming Language :: Python :: 3.6',
4853
'Programming Language :: Python :: 3.7',

‎zarr/storage.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,15 @@
5353
from zarr.codecs import Zlib
5454
default_compressor = Zlib()
5555

56+
# Find which function to use for atomic replace
57+
if sys.version_info >= (3, 3):
58+
from os import replace
59+
elif sys.platform == "win32": # pragma: no cover
60+
from osreplace import replace
61+
else: # pragma: no cover
62+
# POSIX rename() is always atomic
63+
from os import rename as replace
64+
5665

5766
def _path_to_prefix(path):
5867
# assume path already normalized
@@ -752,9 +761,7 @@ def __setitem__(self, key, value):
752761
f.write(value)
753762

754763
# move temporary file into place
755-
if os.path.exists(file_path):
756-
os.remove(file_path)
757-
os.rename(temp_path, file_path)
764+
replace(temp_path, file_path)
758765

759766
finally:
760767
# clean up if temp file still exists for whatever reason

0 commit comments

Comments
 (0)
0