[SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API #54014
Conversation
JIRA Issue Information=== Sub-task SPARK-55228 === === Sub-task SPARK-55230 === This comment was automatically generated by GitHub Actions |
There was a problem hiding this comment.
Otherwise, LGTM.
cc @zhengruifeng
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Outdated
Show resolved
Hide resolved
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Outdated
Show resolved
Hide resolved
Co-authored-by: Takuya UESHIN <ueshin@happy-camper.st>
|
Add |
| } | ||
|
|
||
| /** @inheritdoc */ | ||
| def zipWithIndex(indexColName: String): DataFrame = { |
There was a problem hiding this comment.
since scala client is supported in this PR, we may want to add some tests in
src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
org/apache/spark/sql/connect/ClientE2ETestSuite.scala
but the tests can be added in a followup PR
There was a problem hiding this comment.
Thanks for working on this
|
@fangchenli linter and doc build failed, ptal |
What changes were proposed in this pull request?
Implement Dataset.zipWithIndex in Scala API
Why are the changes needed?
Align Dataset and RDD api
Does this PR introduce any user-facing change?
Yes, this PR added Dataset.zipWithIndex() and Dataset.zipWithIndex(indexColName: String)
How was this patch tested?
Unittests added
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.5