E525 Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes by Copilot · Pull Request #3163 · DeepLabCut/DeepLabCut · GitHub
[go: up one dir, main page]
Skip to content

Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes #3163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
Copilot wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes #3163

Copilot wants to merge 4 commits into from

Conversation

Copy link
Contributor
Copilot AI commented Dec 26, 2025 •
edited
Loading

Pandas 2.0+ uses StringDtype extension dtypes by default for MultiIndex string columns. PyTables 3.10+ cannot serialize these to HDF5, causing NotImplementedError: Saving a MultiIndex with an extension dtype is not supported when saving labeled data.

Changes

  • Added conversion helper (convert_multiindex_to_hdf_compatible()) that downgrades extension dtype levels to object dtype before HDF5 serialization
  • Updated 30+ to_hdf() callsites across the codebase to apply conversion:
    • Core utilities (auxiliaryfunctions, conversioncode, auxfun_multianimal)
    • Dataset management (trainingsetmanipulation, outlier_frames, stitch, tracklets)
    • GUI tools (tracklet_toolbox)
    • Post-processing (filtering, analyze_skeleton, triangulation)
    • Both TensorFlow and PyTorch backends (evaluate, videos, analyze_images)
    • Model zoo utilities
  • Updated PyTables requirement from unspecified to >=3.10.1 in setup.py and requirements.txt
  • Added test verifying round-trip HDF5 serialization with extension dtypes

Example

def convert_multiindex_to_hdf_compatible(df):
    """Convert MultiIndex extension dtypes to object dtype for HDF5 compatibility."""
    if isinstance(df.columns, pd.MultiIndex):
        new_levels = []
        for level in df.columns.levels:
            if hasattr(level.dtype, 'name') and level.dtype.name == 'string':
                new_levels.append(level.astype('object'))
            else:
                new_levels.append(level)
        df.columns = df.columns.set_levels(new_levels)
    # Same for index MultiIndex...
    return df

# Applied before all to_hdf() calls
df = convert_multiindex_to_hdf_compatible(df)
df.to_hdf(output_path, key="df_with_missing", mode="w")

Backward compatible with older pandas/PyTables versions. No API changes.

Original prompt

This section details on the original issue you should resolve

<issue_title>Pytables version incompatibility</issue_title>
<issue_description>### Is there an existing issue for this?

  • I have searched the existing issues

Operating System

Description: Ubuntu 24.04.3 LTS
Release: 24.04
Codename: noble

DeepLabCut version

3.0.0rc13

What engine are you using?

pytorch

DeepLabCut mode

single animal

Device type

GeForce RTX 4070

Bug description 🐛

Pytables 3.8.0 was installed as described in the installation documents. Attempting to save a layer of labelled points during frame labeling gives the error:

ImportError: Pandas requires version '3.10.1' or newer of 'tables' (version '3.8.0' currently installed).

This dependency conflict cannot be resolved. Upgrading pytables then leads to another error at the same point:

NotImplementedError: Saving a MultiIndex with an extension dtype is not supported.

Steps To Reproduce

I installed deeplabcut as described:

conda create -n DEEPLABCUT python=3.12
conda activate DEEPLABCUT
conda install -c conda-forge pytables==3.8.0
conda install pytorch cudatoolkit=11.3 -c pytorch
pip install --pre deeplabcut[gui]

python -c "import torch; print(torch.cuda.is_available())"
Returns True.

I created a project, extracted frames from a subset of videos using automatic k-means, with frame cropping from config. I then tried to label frames. When I try to save my layer, I get the error.

Relevant log output

With pytables 3.8.0

python -m deeplabcut
Loading DLC 3.0.0rc13...
Starting GUI...
QPixmap::scaled: Pixmap is a null pixmap
_pythonToCppCopy: Cannot copy-convert 0x653e45e97880 (NoneType) to C++.
_pythonToCppCopy: Cannot copy-convert 0x653e45e97880 (NoneType) to C++.
Traceback (most recent call last):
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari_deeplabcut/_widgets.py", line 656, in <lambda>
    lambda: _save_layers_dialog(
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari_deeplabcut/_widgets.py", line 300, in _save_layers_dialog
    self.viewer.layers.save("", selected=True, plugin="napari-deeplabcut")
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/utils/_proxies.py", line 188, in __call__
    return self.create(self.__wrapped__(*args, **kwargs))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/components/layerlist.py", line 476, in save
    return save_layers(path, layers, plugin=plugin, _writer=_writer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/plugins/io.py", line 229, in save_layers
    _written, writer_name = _write_single_layer_with_plugins(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/plugins/io.py", line 442, in _write_single_layer_with_plugins
    written_paths, writer_name = _npe2.write_layers(
                                 ^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/plugins/_npe2.py", line 111, in write_layers
    paths, writer = io_utils.write_get_writer(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/npe2/io_utils.py", line 115, in write_get_writer
    return _write(path, layer_data, plugin_name=plugin_name, return_writer=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/npe2/io_utils.py", line 318, in _write
    res = writer.exec(args=args)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/npe2/manifest/utils.py", line 61, in exec
    return self.get_callable(reg)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari_deeplabcut/_writer.py", line 79, in write_hdf
    df.to_hdf(path, key="k
6880
eypoints", mode="w")
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/core/generic.py", line 2789, in to_hdf
    pytables.to_hdf(
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/io/pytables.py", line 313, in to_hdf
    with HDFStore(
         ^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/io/pytables.py", line 578, in __init__
    tables = import_optional_dependency("tables")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/compat/_optional.py", line 187, in import_optional_dependency
    raise ImportErr...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes DeepLabCut/DeepLabCut#3162

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

Copilot AI and others added 3 commits December 26, 2025 10:14
@MMathisLab
Add helper function to convert MultiIndex extension dtypes for HDF5 c…
f9872fa 
…ompatibility - core files updated

Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>
@MMathisLab
Update all to_hdf calls and set PyTables version requirement to >=3.10.1
afd02bb 
Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>
@MMathisLab
Add test for MultiIndex extension dtype conversion to HDF5 compatibility
d56ae2e 
Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix Pytables version incompatibility issue Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes Dec 26, 2025
Copilot AI requested a review from MMathisLab December 26, 2025 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MMathisLab MMathisLab Awaiting requested review from MMathisLab

At least 1 approving review is required to merge this pull request.

Assignees

@MMathisLab MMathisLab

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MMathisLab
0