8000 [SPARK-54179][SQL][FOLLOW-UP] Add Dataframe API support for Tuple sketches by cboumalh · Pull Request #54041 · apache/spark · GitHub
[go: up one dir, main page]
Skip to content

[SPARK-54179][SQL][FOLLOW-UP] Add Dataframe API support for Tuple sketches #54041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
cboumalh wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[SPARK-54179][SQL][FOLLOW-UP] Add Dataframe API support for Tuple sketches #54041

cboumalh wants to merge 3 commits into from
+3,679 −7

Conversation

@cboumalh
Copy link
Contributor
@cboumalh cboumalh commented Jan 28, 2026

What changes were proposed in this pull request?

This PR adds DataFrame API support for Tuple sketch functions that were previously only available in Spark SQL. The following functions are now accessible through the DataFrame API:

  • tuple_sketch_agg_* - Aggregation functions for creating tuple sketches
  • tuple_union_agg_* - Union aggregation functions
  • tuple_intersection_agg_* - Intersection aggregation functions
  • tuple_sketch_estimate_* - Estimation functions for tuple sketches
  • tuple_sketch_summary_* - Summary functions for tuple sketches
  • tuple_sketch_theta_* - Theta extraction functions
  • tuple_union_* - Union operations
  • tuple_intersection_* - Intersection operations
  • tuple_difference_* - Difference operations

This is a follow-up to SPARK-54179 that completes the DataFrame API parity for tuple sketch operations.

Why are the changes needed?

This PR expands the sketch feature's adoption in DataFrame cases.

Does this PR introduce any user-facing change?

yes, it provides the DataFrame API for Tuple sketches.

  * tuple_sketch_agg_*
  * tuple_union_agg_*
  * tuple_intersection_agg_*
  * tuple_sketch_estimate_*
  * tuple_sketch_summary_*
  * tuple_sketch_theta_*
  * tuple_union_*
  * tuple_intersection_*
  * tuple_difference_*

How was this patch tested?

Added tests to DataFrameAggregateSuite.scala

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Sonnet 4.5

@github-actions
Copy link
github-actions bot commented Jan 28, 2026

JIRA Issue Information

=== New Feature SPARK-54179 ===
Summary: Add Native Support for Apache Tuple Sketches
Assignee: Christopher Boumalhab
Status: Resolved
Affected: ["4.2.0"]

This comment was automatically generated by GitHub Actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CONNECT DOCS PYTHON SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cboumalh
0