8000 Adapt to REPL changes planned for Scala 2.12.11 by retronym · Pull Request #1 · retronym/spark · GitHub
[go: up one dir, main page]
Skip to content

Adapt to REPL changes planned for Scala 2.12.11 #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Adapt to REPL changes planned for Scala 2.12.11 #1

wants to merge 2 commits into from

Conversation

retronym
Copy link
Owner
@retronym retronym commented Feb 12, 2020 •
edited
Loading

The renovations of -Yrepl-class-based (scala/scala#8712) require opting into the the newly added -Yrepl-use-magic-imports (scala/scala#8576).

@retronym
Add profile to enable testing builds of Scala from scala/scala's CI
391ba6d 
For instance:

```
 mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.11-bin-ee22df6-SNAPSHOT clean install
```
@retronym
Enable -Yrepl-use-magic-imports in compiler versions with that setting
35e8500 
This not only reduces the bloat of the REPL wrapper code, but
is also necessary to keep the Spark REPL test suites passing
after the Scala REPLs implementation of -Yrepl-class-based has
been updated in Scala 2.12.11.

Tested with:

```
./build/mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.10 -pl :spark-repl_2.12 -Dtest=none -DwildcardSuites='org.apache.spark.repl.ReplSuite,org.apache.spark.repl.SingletonReplSuite,org.apache.spark.repl.ExecutorClassLoaderSuite' test
./build/mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.11-bin-ee22df6-SNAPSHOT -pl :spark-repl_2.12 -Dtest=none -DwildcardSuites='org.apache.spark.repl.ReplSuite,org.apache.spark.repl.SingletonReplSuite,org.apache.spark.repl.ExecutorClassLoaderSuite' test
```
@retronym retronym changed the title Add profile to enable testing builds of Scala from scala/scala's CI Adapt to REPL changes planned for Scala 2.12.11 Feb 13, 2020
@retronym
Copy link
Owner Author

Test with:

./build/mvn -P scala-pr-validation-snapshots -Dscala.version=2.12.11-bin-ee22df6-SNAPSHOT -pl :spark-repl_2.12 -Dtest=none -DwildcardSuites='org.apache.spark.repl.ReplSuite,org.apache.spark.repl.SingletonReplSuite,org.apache.spark.repl.ExecutorClassLoaderSuite' test

@retronym retronym mentioned this pull request Feb 13, 2020
Merged
1 task
@dwijnand
Copy link

Failures when using that 2.12.11 build and without the magic imports (cc @lrytz):

SingletonReplSuite:
Spark context available as 'sc' (master = local-cluster[2,1,1024], app id = app-20200213105349-0000).
Spark session available as 'spark'.
- simple foreach with accumulator
- external vars

- external classes *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala>      |      | defined class C

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 12, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$res$1(<console>:24)
  	at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.java:23)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator.foreach(Iterator.scala:943)
  	at scala.collection.Iterator.foreach$(Iterator.scala:943)
  	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
  	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
  	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
  	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
  	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
  	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
  	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
  	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
  	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
  	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
  	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 35 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$res$1(<console>:24)
    at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.java:23)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    at scala.collection.AbstractIterator.to(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
    at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
    at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>      | _result_1581591233942: Int = 1

  scala> (SingletonReplSuite.scala:106)

- external functions
- external functions that access vars
- broadcast vars
- interacting with files
- local-cluster mode
- SPARK-1199 two instances of same class don't type check.
- SPARK-2452 compound statements.

- SPARK-2576 importing implicits *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> import spark.implicits._

  scala> defined class TestCaseClass

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 4 times, most recent failure: Lost task 0.3 in stage 16.0 (TID 44, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$res15$1(<console>:30)
  	at $anonfun$res15$1$adapted(<console>:30)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
  	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
  	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:339)
  	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
  	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
  	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
  	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:384)
    at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3507)
    at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2831)
    at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3497)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:102)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:162)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:89)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3495)
    at org.apache.spark.sql.Dataset.collect(Dataset.scala:2831)
    ... 37 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$res15$1(<console>:30)
    at $anonfun$res15$1$adapted(<console>:30)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:339)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>
  scala>
  scala> res16: Array[TestCaseClass] = Array(TestCaseClass(1))

  scala>      | _result_1581591240142: Int = 1

  scala> (SingletonReplSuite.scala:106)

- Datasets and encoders

- SPARK-2632 importing a method from non serializable class and not using it. *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> defined class TestClass

  scala> t: TestClass = TestClass@1bbf2d84

  scala> import t.testMethod

  scala> defined class TestCaseClass

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 19.0 failed 4 times, most recent failure: Lost task 0.3 in stage 19.0 (TID 55, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$res20$1(<console>:38)
  	at $anonfun$res20$1$adapted(<console>:38)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator.foreach(Iterator.scala:943)
  	at scala.collection.Iterator.foreach$(Iterator.scala:943)
  	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
  	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
  	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
  	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
  	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
  	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
  	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
  	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
  	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
  	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
  	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 39 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$res20$1(<console>:38)
    at $anonfun$res20$1$adapted(<console>:38)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    at scala.collection.AbstractIterator.to(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
    at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
    at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>      | _result_1581591243727: Int = 1

  scala> (SingletonReplSuite.scala:106)

- collecting objects of class defined in repl *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> defined class Foo

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 57, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: TestClass
  Serialization stack:
  	- object not serializable (class: TestClass, value: TestClass@1bbf2d84)
  	- field (class: $iw, name: t, type: class TestClass)
  	- object (class $iw, $iw@623a8812)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@2a0edbaa)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@a82317a)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@ba68685)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@31f4cc52)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@429285ed)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@5e1d4d)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3dea2a3e)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3fd0c411)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@71a6d5b5)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@728ae1fc)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@121b000e)
  	- field (class: $read, name: $iw, type: class $iw)
  	- object (class $read, $read@5cb921d7)
  	- field (class: $iw, name: read, type: class $read)
  	- object (class $iw, $iw@68fdc77c)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@6f6837e3)
  	- field (class: Foo, name: $outer, type: class $iw)
  	- object (class Foo, Foo(1))
  	- element of array (index: 0)
  	- array (class [LFoo;, size 10)
  	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
  	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray(Foo(1), Foo(2), Foo(3), Foo(4), Foo(5), Foo(6), Foo(7), Foo(8), Foo(9), Foo(10)))
  	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
  	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@cf9)
  	- field (class: org.apache.spark.scheduler.ResultTask, name: partition, type: interface org.apache.spark.Partition)
  	- object (class org.apache.spark.scheduler.ResultTask, ResultTask(20, 0))
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 39 elided

  scala>      | _result_1581591244762: Int = 1

  scala> (SingletonReplSuite.scala:106)

- collecting objects of class defined in repl - shuffling *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> defined class Foo

  scala> list: List[(Int, Foo)] = List((1,Foo(1)), (1,Foo(2)))

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 58, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: TestClass
  Serialization stack:
  	- object not serializable (class: TestClass, value: TestClass@1bbf2d84)
  	- field (class: $iw, name: t, type: class TestClass)
  	- object (class $iw, $iw@623a8812)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@2a0edbaa)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@a82317a)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@ba68685)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@31f4cc52)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@429285ed)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@5e1d4d)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3dea2a3e)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3fd0c411)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@71a6d5b5)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@728ae1fc)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@121b000e)
  	- field (class: $read, name: $iw, type: class $iw)
  	- object (class $read, $read@5cb921d7)
  	- field (class: $iw, name: read, type: class $read)
  	- object (class $iw, $iw@253e26ef)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@4d910314)
  	- field (class: Foo, name: $outer, type: class $iw)
  	- object (class Foo, Foo(1))
  	- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
  	- object (class scala.Tuple2, (1,Foo(1)))
  	- element of array (index: 0)
  	- array (class [Lscala.Tuple2;, size 1)
  	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
  	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray((1,Foo(1))))
  	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
  	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@d22)
  	- field (class: org.apache.spark.scheduler.ShuffleMapTask, name: partition, type: interface org.apache.spark.Partition)
  	- object (class org.apache.spark.scheduler.ShuffleMapTask, ShuffleMapTask(21, 0))
    at org.apache.spark.scheduler.DAGScheduler.fai
8000
lJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 39 elided

  scala>      | _result_1581591245289: Int = 1

  scala> (SingletonReplSuite.scala:106)

- replicating blocks of object with class defined in repl *** FAILED ***
  isContain was true Interpreter output contained 'Exception':

  scala> timeout: Int = 60000

  scala> start: Long = 1581591245957

  scala>      |      |      |
  scala>      |      |
  scala> import org.apache.spark.storage.StorageLevel._

  scala> defined class Foo

  scala> ret: org.apache.spark.rdd.RDD[Foo] = ParallelCollectionRDD[43] at parallelize at <console>:39

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 59, not attempting to retry it. Exception during serialization: java.io.NotSerializableException: TestClass
  Serialization stack:
  	- object not serializable (class: TestClass, value: TestClass@1bbf2d84)
  	- field (class: $iw, name: t, type: class TestClass)
  	- object (class $iw, $iw@623a8812)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@2a0edbaa)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@a82317a)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@ba68685)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@31f4cc52)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@429285ed)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@5e1d4d)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3dea2a3e)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@3fd0c411)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@71a6d5b5)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@728ae1fc)
  	- field (class: $iw, name: $iw, type: class $iw)
  	- object (class $iw, $iw@121b000e)
  	- field (class: $read, name: $iw, type: class $iw)
  	- object (class $read, $read@5cb921d7)
  	- field (class: $iw, name: read, type: class $read)
  	- object (class $iw, $iw@2e516148)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@78e16ce)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@86a622b)
  	- field (class: $iw, name: $outer, type: class $iw)
  	- object (class $iw, $iw@6db87f43)
  	- field (class: Foo, name: $outer, type: class $iw)
  	- object (class Foo, Foo(1))
  	- element of array (index: 0)
  	- array (class [LFoo;, size 10)
  	- field (class: scala.collection.mutable.WrappedArray$ofRef, name: array, type: class [Ljava.lang.Object;)
  	- object (class scala.collection.mutable.WrappedArray$ofRef, WrappedArray(Foo(1), Foo(2), Foo(3), Foo(4), Foo(5), Foo(6), Foo(7), Foo(8), Foo(9), Foo(10)))
  	- writeObject data (class: org.apache.spark.rdd.ParallelCollectionPartition)
  	- object (class org.apache.spark.rdd.ParallelCollectionPartition, org.apache.spark.rdd.ParallelCollectionPartition@d74)
  	- field (class: org.apache.spark.scheduler.ResultTask, name: partition, type: interface org.apache.spark.Partition)
  	- object (class org.apache.spark.scheduler.ResultTask, ResultTask(23, 0))
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    ... 41 elided

  scala> res: Int = 0

  scala>      | _result_1581591245815: Int = 1

  scala> (SingletonReplSuite.scala:106)

- should clone and clean line object in ClosureCleaner *** FAILED ***
  isContain was true Interpreter output contained 'AssertionError':

  scala> import org.apache.spark.rdd.RDD

  scala>
  scala> lines: org.apache.spark.rdd.RDD[String] = pom.xml MapPartitionsRDD[45] at textFile at <console>:38

  scala> defined class Data

  scala> dataRDD: org.apache.spark.rdd.RDD[Data] = MapPartitionsRDD[46] at map at <console>:40

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 24.0 failed 4 times, most recent failure: Lost task 0.3 in stage 24.0 (TID 66, grey.mynet, executor 0): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$dataRDD$1(<console>:40)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
  	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
  	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
  	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
  	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
  	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
  	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    ... 41 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$dataRDD$1(<console>:40)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
    at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala> repartitioned: org.apache.spark.rdd.RDD[Data] = MapPartitionsRDD[50] at repartition at <console>:38

  scala> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 25.0 failed 4 times, most recent failure: Lost task 0.3 in stage 25.0 (TID 75, grey.mynet, executor 1): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$dataRDD$1(<console>:40)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
  	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
  	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
  	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
  	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
  	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
  	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
  	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
  	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
  	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
  	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
  	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
  	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.
8000
scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    ... 41 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$dataRDD$1(<console>:40)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
    at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>
  scala>      |      | getCacheSize: (rdd: org.apache.spark.rdd.RDD[_])Long

  scala> cacheSize1: Long = 0

  scala> cacheSize2: Long = 0

  scala>
  scala>
  scala> deviation: Double = NaN

  scala>      | java.lang.AssertionError: assertion failed: deviation too large: NaN, first size: 0, second size: 0
    at scala.Predef$.assert(Predef.scala:223)
    ... 42 elided

  scala>      | _result_1581591246847: Int = 1

  scala> (SingletonReplSuite.scala:106)

- newProductSeqEncoder with REPL defined class

- create encoder in executors *** FAILED ***
  isContain was false Interpreter output did not contain 'r: Foo = Foo(bar)':

  scala> defined class Foo

  scala>
  scala> import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder

  scala>
  scala>      | org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 27.0 failed 4 times, most recent failure: Lost task 1.3 in stage 27.0 (TID 80, grey.mynet, executor 0): java.lang.NoClassDefFoundError: Could not initialize class
  	at $anonfun$r$1(<console>:42)
  	at $anonfun$r$1$adapted(<console>:42)
  	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
  	at scala.collection.Iterator.foreach(Iterator.scala:943)
  	at scala.collection.Iterator.foreach$(Iterator.scala:943)
  	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
  	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
  	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
  	at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
  	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
  	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
  	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
  	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
  	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
  	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
  	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
  	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
  	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
  	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  	at org.apache.spark.scheduler.Task.run(Task.scala:127)
  	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
  	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
  	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:834)

  Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1983)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1970)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1970)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2200)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2149)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2138)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2116)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2137)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2156)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2181)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    ... 42 elided
  Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    at $anonfun$r$1(<console>:42)
    at $anonfun$r$1$adapted(<console>:42)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableOnce.to(TraversableOnce.scala:315)
    at scala.collection.TraversableOnce.to$(TraversableOnce.scala:313)
    at scala.collection.AbstractIterator.to(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:307)
    at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:307)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
    at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:294)
    at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:288)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
    at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1004)
    at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

  scala>      | _result_1581591249369: Int = 1

  scala> (SingletonReplSuite.scala:100)

@dwijnand
Copy link

I wasn't able to extract from that how we can reproduce that in a test in our suite. 😕

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@retronym @dwijnand
0