8000 [WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 by wangyum · Pull Request #27929 · apache/spark · GitHub
[go: up one dir, main page]
Skip to content

[WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 #27929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 #27929

wants to merge 4 commits into from

Conversation

wangyum
Copy link
Member
@wangyum wangyum commented Mar 16, 2020 •
edited by dongjoon-hyun
Loading

What changes were proposed in this pull request?

This PR(SPARK-31168) upgrade Scala to 2.12.11.

Scala 2.12.11 release notes:
Highlights

  • Performance improvements in the collections library
  • Performance improvements in the compiler
  • Improvements to -Yrepl-class-based, an alternative internal REPL encoding that avoids deadlocks
  • A new -Yrepl-use-magic-imports flag that avoids deep class nesting in the REPL, which can lead to deteriorating performance in long sessions
  • Fix some toX methods that could expose the underlying mutability of a ListBuffer-generated collection

JDK 9+ support

  • ASM was upgraded to 7.3.1, allowing the optimizer to run on JDK 13+
  • :javap in the REPL now works on JDK 9+

Why are the changes needed?

Improvement performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Exist test.

@SparkQA
Copy link
SparkQA commented Mar 16, 2020

Test build #119892 has finished for PR 27929 at commit 7811091.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Mar 16, 2020
View reviewed changes
Copy link
Member
@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the important thing will be to make sure the items tested at #27681 work

docs/_config.yml Outdated
@@ -17,7 +17,7 @@ include:
SPARK_VERSION: 3.1.0-SNAPSHOT
SPARK_VERSION_SHORT: 3.1.0
SCALA_BINARY_VERSION: "2.12"
SCALA_VERSION: "2.12.10"
SCALA_VERSION: "2.12.1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo here

@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.11 [WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 Mar 17, 2020
@dongjoon-hyun
Copy link
Member

Retest this please.

@dongjoon-hyun
Copy link
Member

It seems that we need to make a PR to genjavadoc.

@SparkQA
Copy link
SparkQA commented Mar 17, 2020

Test build #119899 has finished for PR 27929 at commit 7811091.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link
SparkQA commented Mar 17, 2020

Test build #119900 has finished for PR 27929 at commit fa21f57.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link
SparkQA commented Mar 18, 2020

Test build #119994 has finished for PR 27929 at commit 678e5d6.

  • This patch fails from timeout after a configured wait of 400m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member
maropu commented Mar 18, 2020

retest this please

@SparkQA
Copy link
SparkQA commented Mar 19, 2020

Test build #120010 has finished for PR 27929 at commit 678e5d6.

  • This patch fails from timeout after a configured wait of 400m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

retest this please

@SparkQA
Copy link
SparkQA commented Mar 19, 2020

Test build #120042 has finished for PR 27929 at commit 678e5d6.

  • This patch fails from timeout after a configured wait of 400m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

retest this please

@maropu
Copy link
Member
maropu commented Mar 19, 2020

(Does this upgrade mak the testing time much longer..?)

@SparkQA
Copy link
SparkQA commented Mar 19, 2020

Test build #120064 has finished for PR 27929 at commit 678e5d6.

  • This patch fails from timeout after a configured wait of 400m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member

Retest this please.

@SparkQA
Copy link
SparkQA commented Mar 20, 2020

Test build #120073 has finished for PR 27929 at commit 678e5d6.

  • This patch fails from timeout after a configured wait of 400m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link
SparkQA commented Mar 20, 2020

Test build #120084 has finished for PR 27929 at commit f1d8e38.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author
wangyum commented Mar 20, 2020

Retest this please.

@SparkQA
Copy link
SparkQA commented Mar 20, 2020

Test build #120105 has finished for PR 27929 at commit f1d8e38.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member
maropu commented Mar 20, 2020

retest this please

@SparkQA
Copy link
SparkQA commented Mar 21, 2020

Test build #120118 has finished for PR 27929 at commit f1d8e38.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author
wangyum commented Mar 21, 2020

It seems 2.12.11 breaks generate Java doc: scala/bug#11917

@wangyum
Copy link
Member Author
wangyum commented Mar 27, 2020

Close it due to this issue: scala/bug#11917

@wangyum wangyum closed this Mar 27, 2020
@dongjoon-hyun
Copy link
Member

Thank you, @wangyum .
I updated SPARK-31168 from Upgrade Scala to 2.12.11 to Upgrade Scala to 2.12.12.

@wangyum wangyum deleted the SPARK-31168 branch March 28, 2020 01:49
@dongjoon-hyun dongjoon-hyun mentioned this pull request Jan 18, 2021
Closed
dongjoon-hyun added a commit that referenced this pull request Jan 18, 2021
@dongjoon-hyun
[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka …
a65e86a 
…to 2.7.0

### What changes were proposed in this pull request?

This PR is the 3rd try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )

`silencer` library is updated accordingly. And, Kafka version upgrade is required because it fails like the following.
```
[info] KafkaDataConsumerSuite:
[info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds)
[info]   java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
```

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11 and 2.12.12. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #31223 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
skestle pushed a commit to skestle/spark that referenced this pull request Feb 3, 2021
@dongjoon-hyun @skestle
[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka …
18e3ed7 
…to 2.7.0

### What changes were proposed in this pull request?

This PR is the 3rd try to upgrade Scala 2.12.x in order to see the feasibility.
- apache#27929 (Upgrade Scala to 2.12.11, wangyum )
- apache#30940 (Upgrade Scala to 2.12.12, viirya )

`silencer` library is updated accordingly. And, Kafka version upgrade is required because it fails like the following.
```
[info] KafkaDataConsumerSuite:
[info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds)
[info]   java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
```

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11 and 2.12.12. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes apache#31223 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@dongjoon-hyun dongjoon-hyun mentioned this pull request May 28, 2021
Closed
2 tasks
dongjoon-hyun added a commit that referenced this pull request May 30, 2021
@dongjoon-hyun
[SPARK-31168][BUILD] Upgrade Scala to 2.12.14
6c4b60f 
### What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )
- #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.
- Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:
- [x] `silencer` library is released via ghik/silencer#66
- [x] `genjavadoc` library is released via lightbend/genjavadoc#282

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.14
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #32697 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
wangyum added a commit that referenced this pull request May 26, 2023
@wangyum @dongjoon-hyun @LuciferYang
[CARMEL-6190] Upgrade Scala to 2.12.16 (#1063)
ac84911 
* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14

### What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )
- #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.
- Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:
- [x] `silencer` library is released via ghik/silencer#66
- [x] `genjavadoc` library is released via lightbend/genjavadoc#282

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.14
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #32697 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

(cherry picked from commit 6c4b60f)

* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14

### What changes were proposed in this pull request?

This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility.
- #27929 (Upgrade Scala to 2.12.11, wangyum )
- #30940 (Upgrade Scala to 2.12.12, viirya )
- #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun )

Note that Scala 2.12.14 has the following fix for Apache Spark community.
- Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13

REQUIREMENTS:
- [x] `silencer` library is released via ghik/silencer#66
- [x] `genjavadoc` library is released via lightbend/genjavadoc#282

### Why are the changes needed?

Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes.
- https://github.com/scala/scala/releases/tag/v2.12.14
- https://github.com/scala/scala/releases/tag/v2.12.13
- https://github.com/scala/scala/releases/tag/v2.12.12
- https://github.com/scala/scala/releases/tag/v2.12.11

### Does this PR introduce _any_ user-facing change?

Yes, but this is a bug-fixed version.

### How was this patch tested?

Pass the CIs.

Closes #32697 from dongjoon-hyun/SPARK-31168.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

(cherry picked from commit 6c4b60f)

* [SPARK-36759][BUILD] Upgrade Scala to 2.12.15

### What changes were proposed in this pull request?

This PR aims to upgrade Scala to 2.12.15 to support Java 17/18 better.

### Why are the changes needed?

Scala 2.12.15 improves compatibility with JDK 17 and 18:

https://github.com/scala/scala/releases/tag/v2.12.15

- Avoids IllegalArgumentException in JDK 17+ for lambda deserialization
- Upgrades to ASM 9.2, for JDK 18 support in optimizer

### Does this PR introduce _any_ user-facing change?

Yes, this is a Scala version change.

### How was this patch tested?

Pass the CIs

Closes #33999 from dongjoon-hyun/SPARK-36759.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

(cherry picked from commit 16f1f71)

* [SPARK-36759][BUILD][FOLLOWUP] Update version in scala-2.12 profile and doc

### What changes were proposed in this pull request?

This is a follow-up to fix the leftover during switching the Scala version.

### Why are the changes needed?

This should be consistent.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

This is not tested by UT. We need to check manually. There is no more `2.12.14`.
```
$ git grep 2.12.14
R/pkg/tests/fulltests/test_sparkSQL.R:               c(as.Date("2012-12-14"), as.Date("2013-12-15"), as.Date("2014-12-16")))
data/mllib/ridge-data/lpsa.data:3.5307626,0.987291634724086 -0.36279314978779 -0.922212414640967 0.232904453212813 -0.522940888712441 1.79270085261407 0.342627053981254 1.26288870310799
sql/hive/src/test/resources/data/files/over10k:-3|454|65705|4294967468|62.12|14.32|true|mike white|2013-03-01 09:11:58.703087|40.18|joggying
```

Closes #34020 from dongjoon-hyun/SPARK-36759-2.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

(cherry picked from commit adbea25)

* [SPARK-39414][BUILD] Upgrade Scala to 2.12.16

### What changes were proposed in this pull request?
This PR aims to upgrade Scala to 2.12.16

### Why are the changes needed?
This version bring some bug fix and  start to try to support Java 19

scala/scala@v2.12.15...v2.12.16

- [Upgrade to asm 9.3, for JDK19 support](scala/scala#10000)
- [Fix codegen for MH.invoke etc under JDK 17 -release](scala/scala#9930)
- [Deprecation related SecurityManager on JDK 17 ](scala/scala#9775)

### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.

### How was this patch tested?
Pass Github Actions

Closes #36807 from LuciferYang/SPARK-39414.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

(cherry picked from commit ed875a8)

* fix

* fix

* fix

* fix
�[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/core/src/main/scala/org/apache/spark/scheduler/SpillableTaskResultGetter.scala:36: non-variable type argument org.apache.spark.scheduler.DirectTaskResult[_] in type pattern scala.runtime.NonLocalReturnControl[org.apache.spark.scheduler.DirectTaskResult[_]] is unchecked since it is eliminated by erasure�[0m
�[0m[�[31merror�[0m] �[0m[warn] private[spark] class SpillableTaskResultGetter(sparkEnv: SparkEnv, scheduler: TaskSchedulerImpl)�[0m
�[0m[�[31merror�[0m] �[0m[warn] �[0m

* fix
�[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:287: match may not be exhaustive.�[0m
�[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "^")), _)�[0m
�[0m[�[31merror�[0m] �[0m[warn]   private val pow: Parser[Term] = term ~ "^" ~ "^[1-9]\\d*".r ^^ {�[0m
�[0m[�[31merror�[0m] �[0m[warn] �[0m
�[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:301: match may not be exhaustive.�[0m
�[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "~")), _)�[0m
�[0m[�[31merror�[0m] �[0m[warn]     (label ~ "~" ~ expr) ^^ { case r ~ "~" ~ t => ParsedRFormula(r, t.asTerms.terms) }�[0m
�[0m[�[31merror�[0m] �[0m[warn] �[0m

* fix

Co-authored-by: Dongjoon Hyun <dhyun@apple.com>
Co-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: yangjie01 <yangjie01@baidu.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srowen srowen srowen left review comments

Assignees
No one assigned
Labels
BUILD
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@wangyum @SparkQA @dongjoon-hyun @maropu @HyukjinKwon @srowen
0