10000 Add SQLiteStore (#368) · zarr-developers/zarr-python@43f7fae · GitHub
[go: up one dir, main page]
Skip to content

Commit 43f7fae

Browse files
jakirkhamjakirkham
authored
Add SQLiteStore (#368)
* Create an SQLite-backed mutable mapping Implements a key-value store using SQLite. As this is a builtin module in Python and a common database to use in various languages, this should have high utility and be very portable. Not to mention many databases provide an SQLite language on top regardless of the internal representation. So this can be a great template for users wishing to work with Zarr in their preferred 10000 database. * Test SQLiteStore Try using the `SQLiteStore` everywhere one would use another store and make sure that it behaves correctly. This includes simple key-value store usage, creating hierarchies, and storing arrays. * Export `SQLiteStore` to the top-level namespace * Include some SQLiteStore examples Provide a few examples of how one might use `SQLiteStore` to store arrays or groups. These examples are taken with minor modifications from the `LMDBStore` examples. * Demonstrate the `SQLiteStore` in the tutorial Includes a simple example borrowed from `LMDBStore`'s tutorial example, which shows how to create and use an `SQLiteStore`. * Provide API documentation for `SQLiteStore` * Make a release note for `SQLiteStore` * Use unique extension for `SQLiteStore` files Otherwise we may end up opening a different databases' files and try to use them with SQLite only to run into errors. This caused the doctests to fail previously. Changing the extension as we have done should avoid these conflicts. * Only close SQLite database when requested Instead of opening, committing, and closing the SQLite database for every operation, limit these to user requested operations. Namely commit only when the user calls `flush`. Also close only when the user calls `close`. This should make operations with SQLite much more performant than when we automatically committed and closed after every user operation. * Update docs to show how to close `SQLiteStore` As users need to explicitly close the `SQLiteStore` to commit changes and serialize them to the SQLite database, make sure to point this out in the docs. * Ensure all SQL commands are capitalized Appears some of these commands work without capitalization. However as the docs show commands as capitalized, ensure that we are doing the same thing as well. That way this won't run into issues with different SQL implementations or older versions of SQLite that are less forgiving. Plus this should match closer to what users familiar with SQL expect. * Simplify `SQLiteStore`'s `__delitem__` using `in` Make use of `in` instead of repeating the same logic in `__delitem__`. As we now keep the database open between operations, this is much simpler than duplicating the key check logic. Also makes it a bit easier to understand what is going on. * Drop no longer needed flake8 error suppression This was needed when the `import` of `sqlite3` was only here to ensure that it existed (even though it wasn't used). Now we make use of `sqlite3` where it is being imported. So there is no need to tell flake8 to not worry about the unused import as there isn't one. * Simplify `close` and use `flush` * Flush before pickling `SQLiteStore` Make sure that everything intended to be added to the `SQLiteStore` database has been written to disk before attempting to pickle it. That way we can be sure other processes constructing their own `SQLiteStore` have access to the same data and not some earlier representation. * Special case in-memory SQLite database No need to normalize the path when there isn't one (e.g. `:memory:`). * Drop unneeded empty `return` statement * Update docs/release.rst Fix a typo. Co-Authored-By: jakirkham <jakirkham@gmail.com> * Update docs/release.rst Include author and original issue in changelog entry. Co-Authored-By: jakirkham <jakirkham@gmail.com> * Correct default value for `check_same_thread` The default value for `check_same_thread` was previously set to `False` when in reality we want this check enabled. So set `check_same_thread` to `True`. * Flush after making any mutation to the database As users could change the setting of things like `check_same_thread` or they may try to access the same database from multiple threads or processes, make sure to flush any changes that would mutate the database. * Skip flushing data when pickling `SQLiteStore` As we now always commit after an operation that mutates the data, there is no need to commit before pickling the `SQLiteStore` object. After all the data should already be up-to-date in the database. * Skip using `flush` in `close` As everything should already be flushed to the database whenever the state is mutated, there is no need to perform this before closing. * Implement `update` for `SQLiteStore` While there is a default implementation of `update` for `MutableMapping`s, it means that we perform multiple `__setitem__` operations. However it would be better if we could commit all key-value pairs in one operation and commit them. Hence we implement `update` for this purpose. * Rewrite `__setitem__` to use `update` Simplifies `__setitem__` to an `update` operation with a dictionary that contains only one item. This works the same as before, but cuts out some redundancy, which simplifies the code a bit. * Disable `check_same_thread` by default again As we now make sure to commit after every mutating change to the database, disable `check_same_thread` again as it should be safe. * Force some parameters to defaults As some of these parameters no longer make sense to be user customizable, go ahead and just set their values as part of the `sqlite3.connect` call. This ensures that they are set the way we expect. Also it ensures that if users try to mess with them, an error will be raised due to duplicate keyword arguments. To elaborate on why these parameters are not user configurable any more, `detect_types` only makes sense if one is building their own table with specific types. Instead we build the table for users and have very generic types (i.e. text and blobs), which are not worth checking. As we commit after every modification to the database to make it more friendly for other threads and processes, the `isolation_level` might as well be to auto-commit. Setting it to anything else really has no effect. Finally there is no need for `check_same_thread` to be anything other than `False` as we are guaranteeing everything is committed after mutation, which ensures the database is thread and process safe. * Drop `flush` calls from `SQLiteStore` As we already enable auto-committing, any mutation is automatically written after performed. So there is no need for us to commit afterwards. Besides `commit` is turned into a no-op if auto-committing is enabled. * Drop the `flush` function from `SQLiteStore` As we auto-commit all changes, there is no need for a `flush` operation for the `SQLiteStore`. So go ahead and drop the `flush` function and its documentation. * Implement optimized `clear` for `SQLiteStore` As the default implementation of `clear` deletes each key-value pair, this will be considerably slower than an operation that can remove all of them at once. Here we do exactly that by using SQL's `DROP TABLE`. Unfortunately there is not a truncate table command, but doing a drop followed by a create has the same effect. We combine these two operations using `executescript`. Thus auto-commit won't run until after both have run, which will commit the table with all key-value pairs removed. * Implement optimized `rmdir` for `SQLiteStore` Provides an SQL implementation of `rmdir` that is a bit better optimized for removing anything that matches the specified path as opposed to doing multiple removals. If it is detected that the root directory is being removed, simply fallback to clear, which is optimized for that use case as it uses `DROP TABLE` instead of deleting rows. Otherwise remove any path that begins with the normalized user-provided path as long as it may contain at least one more character after. This stops `rmdir` from removing a key-value pair where the key exactly matches normalized user-provided path (i.e. not a "directory" as it contains data). * Implement optimized `getsize` for `SQLiteStore` Take advantage of SQLite's ability to query and filter tables quickly to implement `getsize` entirely in SQL (with the exception of path normalization to sanitize user input). Measures the `LENGTH` of all blobs in the column and calls `SUM` to get their aggregate size. In the event that there are no matches, use `COALESCE` to replace the `NULL` value returned by `SUM` with `0` instead. * Implement optimized `listdir` for `SQLiteStore` Take advantage of SQLite's ability to query and filter tables quickly to implement `listdir` entirely in SQL (with the exception of path normalization to sanitize user input). Makes use of a nested `SELECT`/`AS` to build a set of partial keys below top-level key. These are then further split to get only the portion directly under the top-level key and not any of their children. * Implement `rename` for `SQLiteStore` Creates an SQL implementation of `rename` for `SQLiteStore`. As there isn't a way to safely change the keys in `SQLiteStore` (since they are `PRIMARY`), simply create a temporary table that copies over the key-value pairs with keys renamed using a nested `SELECT` statement. Then delete all key-value pairs that match the keys to move. Finally copy all key-value pairs from the temporary table into our table and delete the temporary table. Perform all of this as a transaction so only the final result of the rename is visible to others. * Allow users to specify the SQLite table name Instead of just picking an arbitrary table name for users, allow them to pick a name for the table. Let it default to `zarr` though to make it easy to discover where it got stored if someone inspects the SQLite database. * Randomize temporary table name Use a UUID to generate a unique table name for the temporary table to hopefully avoid collisions even if multiple such operations are occurring and/or remnants of older operations stuck around. * Merge `SELECT`s in `rename` Fuses the two `SELECTS` in `SQLiteStore`'s `rename` function into one. * Tidy `rename` SQL code a bit * Fuse away one `SELECT` in `listdir` In `SQLiteStore`'s `listdir`, fuse the `SELECT` performing the ordering with the `SELECT` applying the `DISTINCT` criteria. As these can be combined and often `DISTINCT` already performs ordering, this may be a bit easier to optimize for different SQL engines. * Only use `k` in `SQLiteStore`'s `__contains__` We don't make use of the values only the keys when checking for existence. So drop the `v` column from the `SELECT` statement as it is unused and only grab the `k` column. * Fuse `SELECT`s in `SQLiteStore`'s `__contains__` Simplifies the SQL used in `SQLiteStore`'s `__contains__` method by fusing the two `SELECT` statements into one. Does this by using `COUNT(*)` to determine how many rows are left after the `SELECT`. As the selection checks for an exact match with the key (and keys are `PRIMARY`), there either is exactly `1` or `0`. So this works the same as `SELECT EXISTS`, but with a single `SELECT` instead. * Cast `has` to `bool` in `SQLiteStore.__contains__` SQLite does not have a boolean type and merely represents them with integers like `0` and `1` (much like C, which it is written in). While Python will perform the conversion of the `__contains__` result to `bool` for us, go ahead and perform the conversion explicitly for clarity. * Prefer using single quotes in more places * Wrap SQL table creation text * Adjust wrapping of `SQLiteStore.clear`'s code * Use parameters for SQL in `listdir` Make sure to use parameters to pass in `path` used by `listdir`'s SQL code to avoid problems caused by injection. * Use parameters for SQL in `getsize` Make sure to use parameters to pass in `path` used by `getsize`'s SQL code to avoid problems caused by injection. * Use parameters for SQL in `rmdir` Make sure to use parameters to pass in `path` used by `rmdir`'s SQL code to avoid problems caused by injection. * Adjust formatting of `SQLiteStore.__contains__` Make sure the command is run in the first line and result stored. Then unpack and return what it finds. * Drop `SQLiteStore`'s implementation of `rename` It's difficult to protect against injections, avoid copying, using a single transaction, etc. in an SQL implementation of `rename`. So instead just drop this implementation and allow the default `rename` implementation to be used. * Just name the SQL table "zarr" Instead of allowing the user to customize where the table is stored, just set it to "zarr". This avoids issues with the table name potentially exploiting injection attacks. Besides its unclear this level of flexibility is really needed given Zarr supports Groups and thus can store many Arrays in the same key-value store. * Unwrap some lines to compact the code a bit * Simplify `SQLiteStore.__contains__` code wrapping * Check SQLite Cursor's rowcount for deletion Instead of checking if a particular key exists and then either raising a `KeyError` or deleting it in `SQLiteStore`, go ahead with the deletion and check the value of `rowcount`. As the keys are primary, they must be unique and thus each one only occurs once. Thus if deletion worked, the `rowcount` will be exactly `1` (it cannot be larger). Alternatively if deletion failed, the `rowcount` would be `0`. Thus we can simply check if the `rowcount` is not `1` and raise the `KeyError`. This should improve the performance a bit. * Parenthesize operations to `?` in SQL To make sure that SQL prioritizes the right things, parenthesize some operations with `?` to clarify to the reader and the parser what should be prioritized. This is done particularly when concatenating special string match symbols to user parameters. * Check `rowcount` for values less than `1` * Parenthesize a few other SQL commands with `?` * Use one line for `SQLiteStore.rmdir`'s SQL * Use 1 line for `SQLiteStore.rmdir`'s SQL & params * Update docs/release.rst Co-Authored-By: jakirkham <jakirkham@gmail.com> * `TestSQLiteStore` -> `TestGroupWithSQLiteStore` * Drop `else` in `for`/`else` for clarity * Ensure SQLite is new enough to enable threading Adds a simple check to ensure SQLite is new enough to enable thread-safe sharing of connections before setting `check_same_thread=True`. If SQLite is not new enough, set `check_same_thread=False`. * Add spacing around `=` * Hold a lock for any DML operations in SQLiteStore As there are some concerns about keeping operations on the SQLite database sequential for thread-safety, acquire an internal lock when a DML operation occurs. This should ensure that only one modification can occur at a time regardless of whether the connection uses the serialized threading mode or not. * Raise when pickling an in-memory SQLite database * Test in-memory SQLiteStore's separately Uses all the same tests we use for SQLiteStore's on disk except it special cases the pickling test to ensure the `SQLiteStore` cannot be pickled if it is in-memory. * Drop explicit setting of `sqlite3` defaults Simply use the `Connection`'s default arguments implicitly instead of explicitly setting them in the constructor. * Adjust inheritance of `TestSQLiteStoreInMemory` Make sure to inherit directly from `unittest.TestCase` as well.
1 parent fefce3b commit 43f7fae

File tree

8 files changed

+308
-5
lines changed

8 files changed

+308
-5
lines changed

‎docs/api/storage.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ Storage (``zarr.storage``)
2121
.. automethod:: close
2222
.. automethod:: flush
2323

24+
.. autoclass:: SQLiteStore
25+
26+
.. automethod:: close
27+
2428
.. autoclass:: LRUStoreCache
2529

2630
.. automethod:: invalidate

‎docs/release.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ Enhancements
1919
* Support has been added for structured arrays with sub-array shape and/or nested fields. By
2020
:user:`Tarik Onalan <onalant>`, :issue:`111`, :issue:`296`.
2121

22+
* Adds the SQLite-backed :class:`zarr.storage.SQLiteStore` class enabling an
23+
SQLite database to be used as the backing store for an array or group.
24+
By :user:`John Kirkham <jakirkham>`, :issue:`368`, :issue:`365`.
25+
2226
Bug fixes
2327
~~~~~~~~~
2428

‎docs/tutorial.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -729,6 +729,16 @@ group (requires `lmdb <http://lmdb.readthedocs.io/>`_ to be installed)::
729729
>>> z[:] = 42
730730
>>> store.close()
731731

732+
In Zarr version 2.3 is the :class:`zarr.storage.SQLiteStore` class which
733+
enables the SQLite database to be used for storing an array or group (requires
734+
Python is built with SQLite support)::
735+
736+
>>> store = zarr.SQLiteStore('data/example.sqldb')
737+
>>> root = zarr.group(store=store, overwrite=True)
738+
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
739+
>>> z[:] = 42
740+
>>> store.close()
741+ F438
732742
Distributed/cloud storage
733743
~~~~~~~~~~~~~~~~~~~~~~~~~
734744

‎zarr/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@
77
from zarr.creation import (empty, zeros, ones, full, array, empty_like, zeros_like,
88
ones_like, full_like, open_array, open_like, create)
99
from zarr.storage import (DictStore, DirectoryStore, ZipStore, TempStore,
10-
NestedDirectoryStore, DBMStore, LMDBStore, LRUStoreCache)
10+
NestedDirectoryStore, DBMStore, LMDBStore, SQLiteStore,
11+
LRUStoreCache)
1112
from zarr.hierarchy import group, open_group, Group
1213
from zarr.sync import ThreadSynchronizer, ProcessSynchronizer
1314
from zarr.codecs import *

‎zarr/storage.py

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
from __future__ import absolute_import, print_function, division
1919
from collections import MutableMapping, OrderedDict
2020
import os
21+
import operator
2122
import tempfile
2223
import zipfile
2324
import shutil
@@ -26,6 +27,7 @@
2627
import sys
2728
import json
2829
import multiprocessing
30+
from pickle import PicklingError
2931
from threading import Lock, RLock
3032
import glob
3133
import warnings
@@ -1877,6 +1879,211 @@ def __delitem__(self, key):
18771879
self._invalidate_value(key)
18781880

18791881

1882+
class SQLiteStore(MutableMapping):
1883+
"""Storage class using SQLite.
1884+
1885+
Parameters
1886+
----------
1887+
path : string
1888+
Location of database file.
1889+
**kwargs
1890+
Keyword arguments passed through to the `sqlite3.connect` function.
1891+
1892+
Examples
1893+
--------
1894+
Store a single array::
1895+
1896+
>>> import zarr
1897+
>>> store = zarr.SQLiteStore('data/array.sqldb')
1898+
>>> z = zarr.zeros((10, 10), chunks=(5, 5), store=store, overwrite=True)
1899+
>>> z[...] = 42
1900+
>>> store.close() # don't forget to call this when you're done
1901+
1902+
Store a group::
1903+
1904+
>>> store = zarr.SQLiteStore('data/group.sqldb')
1905+
>>> root = zarr.group(store=store, overwrite=True)
1906+
>>> foo = root.create_group('foo')
1907+
>>> bar = foo.zeros('bar', shape=(10, 10), chunks=(5, 5))
1908+
>>> bar[...] = 42
1909+
>>> store.close() # don't forget to call this when you're done
1910+
"""
1911+
1912+
def __init__(self, path, **kwargs):
1913+
import sqlite3
1914+
1915+
# normalize path
1916+
if path != ':memory:':
1917+
path = os.path.abspath(path)
1918+
1919+
# store properties
1920+
self.path = path
1921+
self.kwargs = kwargs
1922+
1923+
# allow threading if SQLite connections are thread-safe
1924+
#
1925+
# ref: https://www.sqlite.org/releaselog/3_3_1.html
1926+
# ref: https://bugs.python.org/issue27190
1927+
check_same_thread = True
1928+
if sqlite3.sqlite_version_info >= (3, 3, 1):
1929+
check_same_thread = False
1930+
1931+
# keep a lock for serializing mutable operations
1932+
self.lock = Lock()
1933+
1934+
# open database
1935+
self.db = sqlite3.connect(
1936+
self.path,
1937+
detect_types=0,
1938+
isolation_level=None,
1939+
check_same_thread=check_same_thread,
1940+
**self.kwargs
1941+
)
1942+
1943+
# handle keys as `str`s
1944+
self.db.text_factory = str
1945+
1946+
# get a cursor to read/write to the database
1947+
self.cursor = self.db.cursor()
1948+
1949+
# initialize database with our table if missing
1950+
with self.lock:
1951+
self.cursor.execute(
1952+
'CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB)'
1953+
)
1954+
1955+
def __getstate__(self):
1956+
if self.path == ':memory:':
1957+
raise PicklingError('Cannot pickle in-memory SQLite databases')
1958+
return self.path, self.kwargs
1959+
1960+
def __setstate__(self, state):
1961+
path, kwargs = state
1962+
self.__init__(path=path, **kwargs)
1963+
1964+
def close(self):
1965+
"""Closes the underlying database."""
1966+
1967+
# close cursor and db objects
1968+
self.cursor.close()
1969+
self.db.close()
1970+
1971+
def __getitem__(self, key):
1972+
value = self.cursor.execute('SELECT v FROM zarr WHERE (k = ?)', (key,))
1973+
for v, in value:
1974+
return v
1975+
raise KeyError(key)
1976+
1977+
def __setitem__(self, key, value):
1978+
self.update({key: value})
1979+
1980+
def __delitem__(self, key):
1981+
with self.lock:
1982+
self.cursor.execute('DELETE FROM zarr WHERE (k = ?)', (key,))
1983+
if self.cursor.rowcount < 1:
1984+
raise KeyError(key)
1985+
1986+
def __contains__(self, key):
1987+
cs = self.cursor.execute(
1988+
'SELECT COUNT(*) FROM zarr WHERE (k = ?)', (key,)
1989+
)
1990+
for has, in cs:
1991+
has = bool(has)
1992+
return has
1993+
1994+
def items(self):
1995+
kvs = self.cursor.execute('SELECT k, v FROM zarr')
1996+
for k, v in kvs:
1997+
yield k, v
1998+
1999+
def keys(self):
2000+
ks = self.cursor.execute('SELECT k FROM zarr')
2001+
for k, in ks:
2002+
yield k
2003+
2004+
def values(self):
2005+
vs = self.cursor.execute('SELECT v FROM zarr')
2006+
for v, in vs:
2007+
yield v
2008+
2009+
def __iter__(self):
2010+
return self.keys()
2011+
2012+
def __len__(self):
2013+
cs = self.cursor.execute('SELECT COUNT(*) FROM zarr')
2014+
for c, in cs:
2015+
return c
2016+
2017+
def update(self, *args, **kwargs):
2018+
args += (kwargs,)
2019+
2020+
kv_list = []
2021+
for dct in args:
2022+
for k, v in dct.items():
2023+
# Python 2 cannot store `memoryview`s, but it can store
2024+
# `buffer`s. However Python 2 won't return `bytes` then. So we
2025+
# coerce to `bytes`, which are handled correctly. Python 3
2026+
# doesn't have these issues.
2027+
if PY2: # pragma: py3 no cover
2028+
v = ensure_bytes(v)
2029+
else: # pragma: py2 no cover
2030+
v = ensure_contiguous_ndarray(v)
2031+
2032+
# Accumulate key-value pairs for storage
2033+
kv_list.append((k, v))
2034+
2035+
with self.lock:
2036+
self.cursor.executemany('REPLACE INTO zarr VALUES (?, ?)', kv_list)
2037+
2038+
def listdir(self, path=None):
2039+
path = normalize_storage_path(path)
2040+
keys = self.cursor.execute(
2041+
'''
2042+
SELECT DISTINCT SUBSTR(m, 0, INSTR(m, "/")) AS l FROM (
2043+
SELECT LTRIM(SUBSTR(k, LENGTH(?) + 1), "/") || "/" AS m
2044+
FROM zarr WHERE k LIKE (? || "_%")
2045+
) ORDER BY l ASC
2046+
''',
2047+
(path, path)
2048+
)
2049+
keys = list(map(operator.itemgetter(0), keys))
2050+
return keys
2051+
2052+
def getsize(self, path=None):
2053+
path = normalize_storage_path(path)
2054+
size = self.cursor.execute(
2055+
'''
2056+
SELECT COALESCE(SUM(LENGTH(v)), 0) FROM zarr
2057+
WHERE k LIKE (? || "%") AND
2058+
0 == INSTR(LTRIM(SUBSTR(k, LENGTH(?) + 1), "/"), "/")
2059+
''',
2060+
(path, path)
2061+
)
2062+
for s, in size:
2063+
return s
2064+
2065+
def rmdir(self, path=None):
2066+
path = normalize_storage_path(path)
2067+
if path:
2068+
with self.lock:
2069+
self.cursor.execute(
2070+
'DELETE FROM zarr WHERE k LIKE (? || "_%")', (path,)
2071+
)
2072+
else:
2073+
self.clear()
2074+
2075+
def clear(self):
2076+
with self.lock:
2077+
self.cursor.executescript(
2078+
'''
2079+
BEGIN TRANSACTION;
2080+
DROP TABLE zarr;
2081+
CREATE TABLE zarr(k TEXT PRIMARY KEY, v BLOB);
2082+
COMMIT TRANSACTION;
2083+
'''
2084+
)
2085+
2086+
18802087
class ConsolidatedMetadataStore(MutableMapping):
18812088
"""A layer over other storage, where the metadata has been consolidated into
18822089
a single key.

‎zarr/tests/test_core.py

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515

1616

1717
from zarr.storage import (DirectoryStore, init_array, init_group, NestedDirectoryStore,
18-
DBMStore, LMDBStore, atexit_rmtree, atexit_rmglob,
18+
DBMStore, LMDBStore, SQLiteStore, atexit_rmtree, atexit_rmglob,
1919
LRUStoreCache)
2020
from zarr.core import Array
2121
from zarr.errors import PermissionError
@@ -1390,6 +1390,31 @@ def test_nbytes_stored(self):
13901390
pass # not implemented
13911391

13921392

1393+
try:
1394+
import sqlite3
1395+
except ImportError: # pragma: no cover
1396+
sqlite3 = None
1397+
1398+
1399+
@unittest.skipIf(sqlite3 is None, 'python built without sqlite')
1400+
class TestArrayWithSQLiteStore(TestArray):
1401+
1402+
@staticmethod
1403+
def create_array(read_only=False, **kwargs):
1404+
path = mktemp(suffix='.db')
1405+
atexit.register(atexit_rmtree, path)
1406+
store = SQLiteStore(path)
1407+
cache_metadata = kwargs.pop('cache_metadata', True)
1408+
cache_attrs = kwargs.pop('cache_attrs', True)
1409+
kwargs.setdefault('compressor', Zlib(1))
1410+
init_array(store, **kwargs)
1411+
return Array(store, read_only=read_only, cache_metadata=cache_metadata,
1412+
cache_attrs=cache_attrs)
1413+
1414+
def test_nbytes_stored(self):
1415+
pass # not implemented
1416+
1417+
13931418
class TestArrayWithNoCompressor(TestArray):
13941419

13951420
def create_array(self, read_only=False, **kwargs):

‎zarr/tests/test_hierarchy.py

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@
1717

1818
from zarr.storage import (DictStore, DirectoryStore, ZipStore, init_group, init_array,
1919
array_meta_key, group_meta_key, atexit_rmtree,
20-
NestedDirectoryStore, DBMStore, LMDBStore, atexit_rmglob,
21-
LRUStoreCache)
20+
NestedDirectoryStore, DBMStore, LMDBStore, SQLiteStore,
21+
atexit_rmglob, LRUStoreCache)
2222
from zarr.core import Array
2323
from zarr.compat import PY2, text_type
2424
from zarr.hierarchy import Group, group, open_group
@@ -928,6 +928,22 @@ def create_store():
928928
return store, None
929929

930930

931+
try:
932+
import sqlite3
933+
except ImportError: # pragma: no cover
934+
sqlite3 = None
935+
936+
937+
@unittest.skipIf(sqlite3 is None, 'python built without sqlite')
938+
class TestGroupWithSQLiteStore(TestGroup):
939+
940+
def create_store(self):
941+
path = tempfile.mktemp(suffix='.db')
942+
atexit.register(atexit_rmtree, path)
943+
store = SQLiteStore(path)
944+
return store, None
945+
946+
931947
class TestGroupWithChunkStore(TestGroup):
932948

933949
@staticmethod

0 commit comments

Comments
 (0)
0