FIX _safe_indexing for pyarrow #31040
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
❌ Linting issuesThis PR is introducing linting issues. Here's a summary of the issues. Note that you can avoid having linting issues by enabling You can see the details of the linting issues under the
|
ogrisel
approved these changes
8000
Mar 21, 2025
There was a problem hiding this comment.
+1 for supporting arrow, and I fine with doing it this way as a pragmatic and incremental improvement pending a potential refactoring to make it more general in the future (to be discussed in a follow-up issue or PR).
However I am worried that this code is untested on our CI. Could you add PyArrow as a soft dependencies on one of the worker CIs (like we do for pandas / polars/ pytorch / array-api-strict...)?
I don't follow. |
|
Should the changelog be in |
Actually, everything is fine: the coverage data is collected on the CI config that runs with Some lines such as the The others could maybe benefit from expanding the existing tests a bit but not big deal wither. Still +1 for merge on my side. |
|
@ogrisel Do we need a 2nd reviewer? |
jeremiedbb
reviewed
Apr 17, 2025
There was a problem hiding this comment.
Hey, given that there seems to be a consensus to use narwhals for our dataframe support tools, do you think that this PR is still relevant ?
We can easily fit it in 1.7 while it's unlikely for narwhals. But do we want to introduce this is we're going to rewrite it soon (ref #31049) ?
Yes, it’s a simple fix for a bug. If we later decide to go with narwhals, there is no cost of having this PR. |
|
@ogrisel @jeremiedbb I still would like to have this PR in 1.7. But I changed my mind about the implementation. I replaced the "dataframe interchange protocol" indexing by proper pyarrow indexing. This is much cleaner. I'm now convinced that the dataframe interchange protocol is not usable for anything else but interchange which we don't really do or need, we need an API for indexing and assigning. Therefore, I defer this topic to #31049. |
jeremiedbb
approved these changes
May 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion c
2A95
annot be applied right now. Please check back later.
Reference Issues/PRs
Partially addresses #25896 (comment).
What does this implement/fix? Explain your changes.
_safe_indexing(.., axis=1)is used in theColumnTransformerand raises an error if apyarrow 8000 .Tableis passed even though it implements the dataframe interchange protocol:results in
which shows that the wrong branch (
elif hasattr(X, "shape"):) is taken.Any other comments?
There is no general solution with
__dataframe__because of data-apis/dataframe-api#85.Therefore a dirtier solution is taken.Therefore, just pyarrow indexing is implemented.
narwhals would be much cleaner, but needs its own dedicated issue for discussion. This PR just fixes a bug with pyarrow.Table passed around.