8000 Fix ``pyarrow`` ``FileInfo`` import by jrbourbeau · Pull Request #9078 · dask/distributed · GitHub
[go: up one dir, main page]
Skip to content

Fix pyarrow FileInfo import #9078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 4, 2025
Merged

Fix pyarrow FileInfo import #9078

merged 4 commits into from
Jun 4, 2025

Conversation

jrbourbeau
Copy link
Member

Custom serialization for pyarrow.fs.FileInfo was recently added to help with Parquet in Dask DataFrame (xref #9025) but this import isn't quite right

In [1]: import pyarrow

In [2]: pyarrow.fs.FileInfo
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 pyarrow.fs.FileInfo

AttributeError: module 'pyarrow' has no attribute 'fs'

I think this went unnoticed because in the Dask DataFrame case we trigger this correct import first

https://github.com/dask/dask/blob/c8ef947bd38e2b739ad3eccdbcce42e08d8c3f4a/dask/dataframe/dask_expr/io/parquet.py#L21

first which allows the import here to work.

However in the non-Dask DataFrame parquet case, that correct import doesn't happen and folks hit the AttributeError

@jrbourbeau jrbourbeau requested a review from fjetter as a code owner May 15, 2025 17:34
@github-actions GitHub Actions
Copy link
Contributor
github-actions bot commented May 15, 2025 •
edited
Loading

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±0      27 suites  ±0   11h 13m 32s ⏱️ + 1m 14s
 4 115 tests ±0   3 999 ✅  - 5    111 💤 ±0  4 ❌ +4  1 🔥 +1 
51 596 runs  +1  49 307 ✅  - 3  2 284 💤  - 1  4 ❌ +4  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit 729e13a. ± Comparison against base commit bf499a2.

♻️ This comment has been updated with latest results.

@jrbourbeau jrbourbeau merged commit ca6efac into dask:main Jun 4, 2025
55 of 61 checks passed
@jrbourbeau jrbourbeau deleted the pa-fileinfo-import branch June 4, 2025 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fjetter fjetter Awaiting requested review from fjetter fjetter is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jrbourbeau
0