create_array fails with ParamValidationError #2955

markus-zimmermann · 2025-04-04T15:37:25Z

markus-zimmermann
Apr 4, 2025

When creating an array I get a ParamValidationError when specifying path in open_group but not when specifying it in zarr.storage.FsspecStore.

So far I only observed it when creating arrays, opening groups and reading arrays works just fine. Is this a bug or am I missing something?

My environment:

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
...
s3fs                      2025.3.2           pyhd8ed1ab_0    conda-forge
zarr                      3.0.6              pyhd8ed1ab_0    conda-forge
...

Method A (works)

fs = S3FileSystem(
    anon=False,
    endpoint_url=endpoint_url,
    asynchronous=True,
)
store = zarr.storage.FsspecStore(
    fs,
    path=path,
)
group = zarr.open_group(
    store,
    mode="a",
)
group.create_array(
    name=array_name,
    shape=array_shape,
    chunks=array_chunks,
    dtype=array_dtype,
    overwrite=True,
)

Method B (fails)

fs = S3FileSystem(
    anon=False,
    endpoint_url=endpoint_url,
    asynchronous=True,
)
store = zarr.storage.FsspecStore(
    fs,
)
group = zarr.open_group(
    store,
    path=path,
    mode="a",
)
group.create_array(
    name=array_name,
    shape=array_shape,
    chunks=array_chunks,
    dtype=array_dtype,
    overwrite=True,
)

produces the following error message

---------------------------------------------------------------------------
ParamValidationError                      Traceback (most recent call last)
Cell In[12], line 1
----> 1 zarr_image_file.create_array(
      2     name="images",
      3     shape=(256, 256, 1, 160, 19, 9),
      4     chunks=(256, 256, 1, 1, 1, 1),
      5     dtype=np.complex64,
      6     overwrite=True,
      7 )

File [/opt/conda/lib/python3.12/site-packages/zarr/_compat.py:43](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/_compat.py#line=42), in _deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
     41 extra_args = len(args) - len(all_args)
     42 if extra_args <= 0:
---> 43     return f(*args, **kwargs)
     45 # extra_args > 0
     46 args_msg = [
     47     f"{name}={arg}"
     48     for name, arg in zip(kwonly_args[:extra_args], args[-extra_args:], strict=False)
     49 ]

File [/opt/conda/lib/python3.12/site-packages/zarr/core/group.py:2473](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/group.py#line=2472), in Group.create_array(self, name, shape, dtype, chunks, shards, filters, compressors, compressor, serializer, fill_value, order, attributes, chunk_key_encoding, dimension_names, storage_options, overwrite, config)
   2378 """Create an array within this group.
   2379 
   2380 This method lightly wraps :func:`zarr.core.array.create_array`.
   (...)   2467 AsyncArray
   2468 """
   2469 compressors = _parse_deprecated_compressor(
   2470     compressor, compressors, zarr_format=self.metadata.zarr_format
   2471 )
   2472 return Array(
-> 2473     self._sync(
   2474         self._async_group.create_array(
   2475             name=name,
   2476             shape=shape,
   2477             dtype=dtype,
   2478             chunks=chunks,
   2479             shards=shards,
   2480             fill_value=fill_value,
   2481             attributes=attributes,
   2482             chunk_key_encoding=chunk_key_encoding,
   2483             compressors=compressors,
   2484             serializer=serializer,
   2485             dimension_names=dimension_names,
   2486             order=order,
   2487             filters=filters,
   2488             overwrite=overwrite,
   2489             storage_options=storage_options,
   2490             config=config,
   2491         )
   2492     )
   2493 )

File [/opt/conda/lib/python3.12/site-packages/zarr/core/sync.py:208](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/sync.py#line=207), in SyncMixin._sync(self, coroutine)
    205 def _sync(self, coroutine: Coroutine[Any, Any, T]) -> T:
    206     # TODO: refactor this to to take *args and **kwargs and pass those to the method
    207     # this should allow us to better type the sync wrapper
--> 208     return sync(
    209         coroutine,
    210         timeout=config.get("async.timeout"),
    211     )

File [/opt/conda/lib/python3.12/site-packages/zarr/core/sync.py:163](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/sync.py#line=162), in sync(coro, loop, timeout)
    160 return_result = next(iter(finished)).result()
    162 if isinstance(return_result, BaseException):
--> 163     raise return_result
    164 else:
    165     return return_result

File [/opt/conda/lib/python3.12/site-packages/zarr/core/sync.py:119](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/sync.py#line=118), in _runner(coro)
    114 """
    115 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
    116 exception, the exception will be returned.
    117 """
    118 try:
--> 119     return await coro
    120 except Exception as ex:
    121     return ex

File [/opt/conda/lib/python3.12/site-packages/zarr/core/group.py:1102](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/group.py#line=1101), in AsyncGroup.create_array(self, name, shape, dtype, chunks, shards, filters, compressors, compressor, serializer, fill_value, order, attributes, chunk_key_encoding, dimension_names, storage_options, overwrite, config)
   1007 """Create an array within this group.
   1008 
   1009 This method lightly wraps :func:`zarr.core.array.create_array`.
   (...)   1097 
   1098 """
   1099 compressors = _parse_deprecated_compressor(
   1100     compressor, compressors, zarr_format=self.metadata.zarr_format
   1101 )
-> 1102 return await create_array(
   1103     store=self.store_path,
   1104     name=name,
   1105     shape=shape,
   1106     dtype=dtype,
   1107     chunks=chunks,
   1108     shards=shards,
   1109     filters=filters,
   1110     compressors=compressors,
   1111     serializer=serializer,
   1112     fill_value=fill_value,
   1113     order=order,
   1114     zarr_format=self.metadata.zarr_format,
   1115     attributes=attributes,
   1116     chunk_key_encoding=chunk_key_encoding,
   1117     dimension_names=dimension_names,
   1118     storage_options=storage_options,
   1119     overwrite=overwrite,
   1120     config=config,
   1121 )

File [/opt/conda/lib/python3.12/site-packages/zarr/core/array.py:4146](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/array.py#line=4145), in create_array(store, name, shape, dtype, data, chunks, shards, filters, compressors, serializer, fill_value, order, zarr_format, attributes, chunk_key_encoding, dimension_names, storage_options, overwrite, config, write_data)
   4141 store_path = await make_store_path(store, path=name, mode=mode, storage_options=storage_options)
   4143 data_parsed, shape_parsed, dtype_parsed = _parse_data_params(
   4144     data=data, shape=shape, dtype=dtype
   4145 )
-> 4146 result = await init_array(
   4147     store_path=store_path,
   4148     shape=shape_parsed,
   4149     dtype=dtype_parsed,
   4150     chunks=chunks,
   4151     shards=shards,
   4152     filters=filters,
   4153     compressors=compressors,
   4154     serializer=serializer,
   4155     fill_value=fill_value,
   4156     order=order,
   4157     zarr_format=zarr_format,
   4158     attributes=attributes,
   4159     chunk_key_encoding=chunk_key_encoding,
   4160     dimension_names=dimension_names,
   4161     overwrite=overwrite,
   4162     config=config,
   4163 )
   4165 if write_data is True and data_parsed is not None:
   4166     await result._set_selection(
   4167         BasicIndexer(..., shape=result.shape, chunk_grid=result.metadata.chunk_grid),
   4168         data_parsed,
   4169         prototype=default_buffer_prototype(),
   4170     )

File [/opt/conda/lib/python3.12/site-packages/zarr/core/array.py:4001](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/array.py#line=4000), in init_array(store_path, shape, dtype, chunks, shards, filters, compressors, serializer, fill_value, order, zarr_format, attributes, chunk_key_encoding, dimension_names, overwrite, config)
   3989     meta = AsyncArray._create_metadata_v3(
   3990         shape=shape_parsed,
   3991         dtype=dtype_parsed,
   (...)   3997         attributes=attributes,
   3998     )
   4000 arr = AsyncArray(metadata=meta, store_path=store_path, config=config)
-> 4001 await arr._save_metadata(meta, ensure_parents=True)
   4002 return arr

File [/opt/conda/lib/python3.12/site-packages/zarr/core/array.py:1369](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/core/array.py#line=1368), in AsyncArray._save_metadata(self, metadata, ensure_parents)
   1359     for parent in parents:
   1360         awaitables.extend(
   1361             [
   1362                 (parent.store_path [/](https://jupyter.cloud.inm-4.fz-juelich.de/) key).set_if_not_exists(value)
   (...)   1366             ]
   1367         )
-> 1369 await gather(*awaitables)

File [/opt/conda/lib/python3.12/site-packages/zarr/storage/_common.py:172](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/storage/_common.py#line=171), in StorePath.set_if_not_exists(self, default)
    163 async def set_if_not_exists(self, default: Buffer) -> None:
    164     """
    165     Store a key to ``value`` if the key is not already present.
    166 
   (...)    170         The buffer to store if the key is not already present.
    171     """
--> 172     await self.store.set_if_not_exists(self.path, default)

File [/opt/conda/lib/python3.12/site-packages/zarr/abc/store.py:259](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/abc/store.py#line=258), in Store.set_if_not_exists(self, key, value)
    254 # Note for implementers: the default implementation provided here
    255 # is not safe for concurrent writers. There's a race condition between
    256 # the `exists` check and the `set` where another writer could set some
    257 # value at `key` or delete `key`.
    258 if not await self.exists(key):
--> 259     await self.set(key, value)

File [/opt/conda/lib/python3.12/site-packages/zarr/storage/_fsspec.py:277](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/zarr/storage/_fsspec.py#line=276), in FsspecStore.set(self, key, value, byte_range)
    275 if byte_range:
    276     raise NotImplementedError
--> 277 await self.fs._pipe_file(path, value.to_bytes())

File [/opt/conda/lib/python3.12/site-packages/s3fs/core.py:1185](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/s3fs/core.py#line=1184), in S3FileSystem._pipe_file(self, path, data, chunksize, max_concurrency, mode, **kwargs)
   1183 # 5 GB is the limit for an S3 PUT
   1184 if size < min(5 * 2**30, 2 * chunksize):
-> 1185     out = await self._call_s3(
   1186         "put_object", Bucket=bucket, Key=key, Body=data, **kwargs, **match
   1187     )
   1188     self.invalidate_cache(path)
   1189     return out

File [/opt/conda/lib/python3.12/site-packages/s3fs/core.py:371](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/s3fs/core.py#line=370), in S3FileSystem._call_s3(self, method, *akwarglist, **kwargs)
    369 logger.debug("CALL: %s - %s - %s", method.__name__, akwarglist, kw2)
    370 additional_kwargs = self._get_s3_method_kwargs(method, *akwarglist, **kwargs)
--> 371 return await _error_wrapper(
    372     method, kwargs=additional_kwargs, retries=self.retries
    373 )

File [/opt/conda/lib/python3.12/site-packages/s3fs/core.py:146](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/s3fs/core.py#line=145), in _error_wrapper(func, args, kwargs, retries)
    144         err = e
    145 err = translate_boto_error(err)
--> 146 raise err

File [/opt/conda/lib/python3.12/site-packages/s3fs/core.py:114](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/s3fs/core.py#line=113), in _error_wrapper(func, args, kwargs, retries)
    112 for i in range(retries):
    113     try:
--> 114         return await func(*args, **kwargs)
    115     except S3_RETRYABLE_ERRORS as e:
    116         err = e

File [/opt/conda/lib/python3.12/site-packages/aiobotocore/client.py:369](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/aiobotocore/client.py#line=368), in AioBaseClient._make_api_call(self, operation_name, api_params)
    365 if properties:
    366     # Pass arbitrary endpoint info with the Request
    367     # for use during construction.
    368     request_context['endpoint_properties'] = properties
--> 369 request_dict = await self._convert_to_request_dict(
    370     api_params=api_params,
    371     operation_model=operation_model,
    372     endpoint_url=endpoint_url,
    373     context=request_context,
    374     headers=additional_headers,
    375 )
    376 resolve_checksum_context(request_dict, operation_model, api_params)
    378 service_id = self._service_model.service_id.hyphenize()

File [/opt/conda/lib/python3.12/site-packages/aiobotocore/client.py:440](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/aiobotocore/client.py#line=439), in AioBaseClient._convert_to_request_dict(self, api_params, operation_model, endpoint_url, context, headers, set_user_agent_header)
    431 async def _convert_to_request_dict(
    432     self,
    433     api_params,
   (...)    438     set_user_agent_header=True,
    439 ):
--> 440     request_dict = self._serializer.serialize_to_request(
    441         api_params, operation_model
    442     )
    443     if not self._client_config.inject_host_prefix:
    444         request_dict.pop('host_prefix', None)

File [/opt/conda/lib/python3.12/site-packages/botocore/validate.py:381](https://jupyter.cloud.inm-4.fz-juelich.de/opt/conda/lib/python3.12/site-packages/botocore/validate.py#line=380), in ParamValidationDecorator.serialize_to_request(self, parameters, operation_model)
    377     report = self._param_validator.validate(
    378         parameters, operation_model.input_shape
    379     )
    380     if report.has_errors():
--> 381         raise ParamValidationError(report=report.generate_report())
    382 return self._serializer.serialize_to_request(
    383     parameters, operation_model
    384 )

ParamValidationError: Parameter validation failed:
Invalid length for parameter Key, value: 0, valid min length: 1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

create_array fails with ParamValidationError #2955

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Uh oh!

create_array fails with ParamValidationError #2955

Uh oh!

Uh oh!

markus-zimmermann Apr 4, 2025

Method A (works)

Method B (fails)

Replies: 0 comments

markus-zimmermann
Apr 4, 2025