8000 [SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka to 2.7.0 by dongjoon-hyun · Pull Request #31223 · apache/spark · GitHub
[go: up one dir, main page]
Skip to content

[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka to 2.7.0 #31223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka to 2.7.0 #31223

wants to merge 4 commits into from

Conversation

dongjoon-hyun
Copy link
Member
@dongjoon-hyun dongjoon-hyun commented Jan 18, 2021 •
edited
Loading

What changes were proposed in this pull request?

This PR is the 3rd try to upgrade Scala 2.12.x in order to see the feasibility.

silencer library is updated accordingly. And, Kafka version upgrade is required because it fails like the following.

[info] KafkaDataConsumerSuite:
[info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds)
[info]   java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)

Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11 and 2.12.12. This will bring all the bug fixes.

Does this PR introduce any user-facing change?

Yes, but this is a bug-fixed version.

How was this patch tested?

Pass the CIs.

@github-actions github-actions bot added the BUILD label Jan 18, 2021
@dongjoon-hyun dongjoon-hyun marked this pull request as draft January 18, 2021 02:43
@github-actions github-actions bot added the DOCS label Jan 18, 2021
@SparkQA

This comment has been minimized.

Sign in to view
@SparkQA

This comment has been minimized.

Sign in to view
@dongjoon-hyun
Copy link
Member Author
dongjoon-hyun commented Jan 18, 2021 •
edited
Loading
  1. Java 11 build with Maven fails because the new plugin is not available yet in Google Maven cache. It will be resolved automatically.
  2. Kafka module failure looks suspicious. I'm looking at it.
[info] KafkaDataConsumerSuite:
[info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds)
[info]   java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)

@dongjoon-hyun
Copy link
Member Author

To fix Kafka issue, it seems that we need to upgrade Kafka to 2.7.0 together.

@dongjoon-hyun
Copy link
Member Author

To pass Kafka UI, I'll try to upgrade Kafka version together.
cc @viirya since he has the previous PR for Kafka. (#30939)

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-31168][BUILD] Upgrade Scala to 2.12.13 [SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka to 2.7.0 Jan 18, 2021
@viirya
Copy link
Member
viirya commented Jan 18, 2021 •
edited
Loading

As I said in the previous PRs (#30940 (comment), #30939 (comment)), the Scala version of Kafak must be matched with Spark. Otherwise, there will be [info] java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7 error.

Kafka 2.7.0 uses Scala 2.12.2. So, in order to upgrade to Scala 2.12.13, we need a Kafka version which uses Scala 2.12.13 too. Otherwise, we cannot upgrade to Scala 2.12.13 in Spark side.

@dongjoon-hyun
Copy link
Member Author
dongjoon-hyun 8000 commented Jan 18, 2021

I thought like that, but please see the following. For a single module, it works. Let's wait and see the full CI result.

$ build/sbt "streaming-kafka-0-10/testOnly *.KafkaDataConsumerSuite"
...
[info] KafkaDataConsumerSuite:
[info] - KafkaDataConsumer reuse in case of same groupId and TopicPartition (188 milliseconds)
[info] - new KafkaDataConsumer instance in case of Task retry (31 milliseconds)
[info] - concurrent use of KafkaDataConsumer (2 seconds, 791 milliseconds)
[info] ScalaTest
[info] Run completed in 8 seconds, 517 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[success] Total time: 147 s (02:27), completed Jan 17, 2021 9:41:13 PM

@SparkQA
Copy link
SparkQA commented Jan 18, 2021

Test build #134180 has finished for PR 31223 at commit fc235a7.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun marked this pull request as ready for review January 18, 2021 07:14
jaceklaskowski
jaceklaskowski approved these changes Jan 18, 2021
View reviewed changes
Copy link
Contributor
@jaceklaskowski jaceklaskowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (non-binding)

@SparkQA
Copy link
SparkQA commented Jan 18, 2021

Test build #134184 has finished for PR 31223 at commit 2004207.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Jan 18, 2021
View reviewed changes
Copy link
Member
@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK - if it works

@viirya
Copy link
Member
viirya commented Jan 18, 2021

Interesting, seems it works, NoClassDefFoundError was gone.

viirya
viirya previously approved these changes Jan 18, 2021
View reviewed changes
@viirya viirya dismissed their stale review January 18, 2021 17:33

Java 11 build seems failed?

@dongjoon-hyun
Copy link
Member Author

Thank you, @jaceklaskowski , @srowen , @viirya .

For Java 11 build failure, silencer-plugin_2.12.13.jar is not synced to Google Maven Central cache yet. I guess this can be resolved at next sync.

org.apache.maven.artifact.resolver.ArtifactNotFoundException:
Could not find artifact com.github.ghik:silencer-plugin_2.12.13:jar:1.6.0 in gcs-maven-central-mirror (https://maven-central.storage-download.googleapis.com/maven2/)

@dongjoon-hyun
Copy link
Member Author

My bad. I noticed that we have old silencer-lib version in pom.xml and fixed it, @viirya .

viirya
viirya approved these changes Jan 18, 2021
View reviewed changes
Copy link
Member
@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, LGTM and see if latest Java 11 build can pass. Thanks.

@SparkQA
Copy link
SparkQA commented Jan 18, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38792/

@dongjoon-hyun
Copy link
Member Author

GitHub Actions passed. Merged to master for Apache Spark 3.2.0.
Thank you all!

@dongjoon-hyun dongjoon-hyun deleted the SPARK-31168 branch January 18, 2021 21:46
@SparkQA
Copy link
SparkQA commented Jan 18, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38792/

@SparkQA
Copy link
SparkQA commented Jan 18, 2021

Test build #134207 has finished for PR 31223 at commit 39a4aa7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@Ngone51
Copy link
Member
Ngone51 commented Jan 27, 2021

Hi all, I find there's still a regression of scala as reported in scala/bug#12038. I tested the demo locally and verified that the regression does exist in scala 2.12.13 but not 2.12.10. The regression could break users' environment if they use protobuf-java 3.11.4. So I think we may need to revert this. cc @cloud-fan @HyukjinKwon

@dongjoon-hyun
Copy link
Member Author

Thank you for reporting.

Do you think we can add a test case for that with or after reverting?

@dongjoon-hyun
Copy link
Member Author

BTW, are we going to downgrade Scala 2.13, too? It seems that the scala issue is open for 2.13, too.

@dongjoon-hyun
Copy link
Member Author

To be clear, I'm fine with reverting PR here.

@HyukjinKwon
Copy link
Member

Thank you @dongjoon-hyun. @Ngone51, can we file a JIRA for Scala 2.13 too so we can track? I think we're fine not to downgrade 2.13 for now since 2.13 is not out yet .. Also I agree that it would be great to know if we can have a way to test.

@Ngone51
Copy link
Member
Ngone51 commented Jan 27, 2021

Opened the ticket: https://issues.apache.org/jira/browse/SPARK-34258.

Not sure how we add the test for such case in Spark. I'd take a look at least.

Thanks @dongjoon-hyun @HyukjinKwon

@HyukjinKwon
Copy link
Member
HyukjinKwon commented Jan 27, 2021 •
edited
Loading

Sure, thanks. Reverted for now.

I'd take a look at least.

Sure, it would be great to know. Please share the result after the investigation.

@dongjoon-hyun
Copy link
Member Author

Thank you for the investigation and taking care of that, @Ngone51 and @HyukjinKwon .

@smarter smarter mentioned this pull request Jan 31, 2021
Closed
skestle pushed a commit to skestle/spark that referenced this pull request Feb 3, 2021
@dongjoon-hyun @skestle
[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka …
18e3ed7 
…to 2.7.0

### What changes were proposed in this pull request?

This PR is the 3rd try to upgrade Scala 2.12.x in order to see the feasibility.
- apache#27929 (Upgrade Scala to 2.12.11, wangyum )
- apache#30940 (Upgrade Scala to 2.12.12, viirya )

`silencer` library is updated accordingly. And, Kafka version upgrade is required because it fails like the following.
```
[info] KafkaDataConsumerSuite:
[info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds)
[info]   java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
```

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11 and 2.12.12. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes apache#31223 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@smarter smarter mentioned this pull request Feb 16, 2021
Merged
@dongjoon-hyun dongjoon-hyun mentioned this pull request May 28, 2021
Closed
2 tasks
dongjoon-hyun added a commit that referenced this pull request May 30, 2021
@dongjoon-hyun
[SPARK-31168][BUILD] Upgrade Scala to 2.12.14
6c4b60f 
### What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )
- #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.
- Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:
- [x] `silencer` library is released via ghik/silencer#66
- [x] `genjavadoc` library is released via lightbend/genjavadoc#282

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.14
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #32697 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
wangyum added a commit that referenced this pull request May 26, 2023
@wangyum @dongjoon-hyun @LuciferYang
[CARMEL-6190] Upgrade Scala to 2.12.16 (#1063)
ac84911 
* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14

### What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )
- #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.
- Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:
- [x] `silencer` library is released via ghik/silencer#66
- [x] `genjavadoc` library is released via lightbend/genjavadoc#282

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.14
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #32697 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

(cherry picked from commit 6c4b60f)

* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14

### What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )
- #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.
- Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:
- [x] `silencer` library is released via ghik/silencer#66
- [x] `genjavadoc` library is released via lightbend/genjavadoc#282

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.14
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #32697 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

(cherry picked from commit 6c4b60f)

* [SPARK-36759][BUILD] Upgrade Scala to 2.12.15

### What changes were proposed in this pull request?

This PR aims to upgrade Scala to 2.12.15 to support Java 17/18 better.

### Why are the changes needed?

Scala 2.12.15 improves compatibility with JDK 17 and 18:

https://github.com/scala/scala/releases/tag/v2.12.15

- Avoids IllegalArgumentException in JDK 17+ for lambda deserialization
- Upgrades to ASM 9.2, for JDK 18 support in optimizer

### Does this PR introduce _any_ user-facing change?

Yes, this is a Scala version change.

### How was this patch tested?

Pass the CIs

Closes #33999 from dongjoon-hyun/SPARK-36759.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

(cherry picked from commit 16f1f71)

* [SPARK-36759][BUILD][FOLLOWUP] Update version in scala-2.12 profile and doc

### What changes were proposed in this pull request?

This is a follow-up to fix the leftover during switching the Scala version.

### Why are the changes needed?

This should be consistent.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

This is not tested by UT. We need to check manually. There is no more `2.12.14`.
```
$ git grep 2.12.14
R/pkg/tests/fulltests/test_sparkSQL.R:               c(as.Date("2012-12-14"), as.Date("2013-12-15"), as.Date("2014-12-16")))
data/mllib/ridge-data/lpsa.data:3.5307626,0.987291634724086 -0.36279314978779 -0.922212414640967 0.232904453212813 -0.522940888712441 1.79270085261407 0.342627053981254 1.26288870310799
sql/hive/src/test/resources/data/files/over10k:-3|454|65705|4294967468|62.12|14.32|true|mike white|2013-03-01 09:11:58.703087|40.18|joggying
```

Closes #34020 from dongjoon-hyun/SPARK-36759-2.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

(cherry picked from commit adbea25)

* [SPARK-39414][BUILD] Upgrade Scala to 2.12.16

### What changes were proposed in this pull request?
This PR aims to upgrade Scala to 2.12.16

### Why are the changes needed?
This version bring some bug fix and  start to try to support Java 19

scala/scala@v2.12.15...v2.12.16

- [Upgrade to asm 9.3, for JDK19 support](scala/scala#10000)
- [Fix codegen for MH.invoke etc under JDK 17 -release](scala/scala#9930)
- [Deprecation related SecurityManager on JDK 17 ](scala/scala#9775)

### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.

### How was this patch tested?
Pass Github Actions

Closes #36807 from LuciferYang/SPARK-39414.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

(cherry picked from commit ed875a8)

* fix

* fix

* fix

* fix
�[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/core/src/main/scala/org/apache/spark/scheduler/SpillableTaskResultGetter.scala:36: non-variable type argument org.apache.spark.scheduler.DirectTaskResult[_] in type pattern scala.runtime.NonLocalReturnControl[org.apache.spark.scheduler.DirectTaskResult[_]] is unchecked since it is eliminated by erasure�[0m
�[0m[�[31merror�[0m] �[0m[warn] private[spark] class SpillableTaskResultGetter(sparkEnv: SparkEnv, scheduler: TaskSchedulerImpl)�[0m
�[0m[�[31merror�[0m] �[0m[warn] �[0m

* fix
�[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:287: match may not be exhaustive.�[0m
�[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "^")), _)�[0m
�[0m[�[31merror�[0m] �[0m[warn]   private val pow: Parser[Term] = term ~ "^" ~ "^[1-9]\\d*".r ^^ {�[0m
�[0m[�[31merror�[0m] �[0m[warn] �[0m
�[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:301: match may not be exhaustive.�[0m
�[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "~")), _)�[0m
�[0m[�[31merror�[0m] �[0m[warn]     (label ~ "~" ~ expr) ^^ { case r ~ "~" ~ t => ParsedRFormula(r, t.asTerms.terms) }�[0m
�[0m[�[31merror�[0m] �[0m[warn] �[0m

* fix

Co-authored-by: Dongjoon Hyun <dhyun@apple.com>
Co-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: yangjie01 <yangjie01@baidu.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srowen srowen srowen left review comments

@jaceklaskowski jaceklaskowski jaceklaskowski approved these changes

@viirya viirya viirya approved these changes

Assignees
No one assigned
Labels
BUILD DOCS SQL STRUCTURED STREAMING
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@dongjoon-hyun @SparkQA @viirya @Ngone51 @HyukjinKwon @jaceklaskowski @srowen
0