8000 Some tests on jdk10 by som-snytt · Pull Request #6889 · scala/scala · GitHub
[go: up one dir, main page]

Skip to content

Some tests on jdk10 #6889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 16, 2018
Merged

Some tests on jdk10 #6889

merged 3 commits into from
Jul 16, 2018

Conversation

som-snytt
Copy link
Contributor
@som-snytt som-snytt commented Jul 5, 2018

@scala-jenkins scala-jenkins added this to the 2.13.0-M5 milestone Jul 5, 2018
@som-snytt som-snytt changed the title Tests on jdk10 [ci:last-only] Tests on jdk10 [ci: last-only] Jul 5, 2018
@som-snytt som-snytt closed this Jul 5, 2018
@som-snytt som-snytt added the WIP label Jul 5, 2018
@som-snytt som-snytt reopened this Jul 9, 2018
@som-snytt
Copy link
Contributor Author

@lrytz Maybe you have a quick answer why this check fails on Java 10. (safeToInline is expected false.)

Invocation of java/lang/Class.forName(Ljava/lang/String;)Ljava/lang/Class;@4 in C.m()Ljava/lang/Class;
[error] Test scala.tools.nsc.backend.jvm.opt.CallGraphTest.callerSensitiveNotSafeToInline failed: java.lang.AssertionError: assertion failed: safeToInline, took 0.298 sec
[error]     at scala.Predef$.assert(Predef.scala:276)
[error]     at scala.tools.nsc.backend.jvm.opt.CallGraphTest.checkCallsite(CallGraphTest.scala:50)
[error]     at scala.tools.nsc.backend.jvm.opt.CallGraphTest.callerSensitiveNotSafeToInline(CallGraphTest.scala:148)

@som-snytt
Copy link
Contributor Author
som-snytt commented Jul 9, 2018

assert8 is used in a couple of places to flag change in reflect behavior that I didn't nail down yet. That's just to let the test suite proceed instead of failing early.

partest is tweaked to handle filtering check file on java9+, but I realized that only java8 and !java8 are required in the current regime.

No attempt to do java modules: avoid Resource and jaxb in tests, and one test filters the WARNING which may fail in future. Actually it WARNs once by default, so I didn't discover all the warnings; I see it in the output.

I tested on OpenJDK Java 10 not 9.

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by scala.reflect.package$ (file:/Users/andrew/workspace/scala/build/quick/classes/library/) to method java.lang.Object.clone()
WARNING: Please consider reporting this to the maintainers of scala.reflect.package$
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

There are still errors in other categories under sbt testAll, as well.

@som-snytt
Copy link
Contributor Author

Also taking the opportunity to remove extraneous flags files. It would be nice if t.flags for directory t could be hidden in t/t.flags. Partest will find group flags there, but that feature is unused by any test. Flags enabled for t/T_1.flags are naturally set for that compilation round.

@lrytz
Copy link
Member
lrytz commented Jul 10, 2018

[error] Test scala.tools.nsc.backend.jvm.opt.CallGraphTest.callerSensitiveNotSafeToInline failed: java.lang.AssertionError: assertion failed: safeToInline, took 0.298 sec

It seems the annotation class moved in 10

https://github.com/scala/scala/blob/v2.13.0-M4/src/compiler/scala/tools/nsc/backend/jvm/opt/BytecodeUtils.scala#L123

checks for sun.reflect.CallerSensitive, but it's now jdk.internal.reflect.CallerSensitive. The test should just look for either of the two.

@retronym
Copy link
Member

WARNING: Illegal reflective access by scala.reflect.package$ (file:/Users/andrew/workspace/scala/build/quick/classes/library/) to method java.lang.Object.clone()

We need to tweak the desugaring of structural calls a little bit to avoid this.

Instead of:

  /** Make a java reflection object accessible, if it is not already
   *  and it is possible to do so. If a SecurityException is thrown in the
   *  attempt, it is caught and discarded.
   */
  def ensureAccessible[T <: jAccessibleObject](m: T): T = {
    if (!m.isAccessible) {
      try m setAccessible true
      catch { case _: SecurityException => } // does nothing
    }
    m
  }

Which is called from, e.g.

scala> class Reflect { def foo = ""; def bar = { this.asInstanceOf[{ def foo: String }].foo }}
defined class Reflect

Which compiles to:

scala> :javap -c Reflect#bar
  public java.lang.String bar();
    Code:
       0: aload_0
       1: astore_1
       2: aload_1
       3: invokevirtual #82                 // Method java/lang/Object.getClass:()Ljava/lang/Class;
       6: invokestatic  #84                 // Method reflMethod$Method1:(Ljava/lang/Class;)Ljava/lang/reflect/Method;
       9: aload_1
      10: iconst_0
      11: anewarray     #4                  // class java/lang/Object
      14: invokevirtual #88                 // Method java/lang/reflect/Method.invoke:(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
...

scala> :javap -c Reflect#reflMethod$Method1
  public static java.lang.reflect.Method reflMethod$Method1(java.lang.Class);
    Code:
       0: invokedynamic #34,  0             // InvokeDynamic #0:apply:()Lscala/runtime/StructuralCallSite;
       5: astore_1
       6: aload_1
       7: aload_0
       8: invokevirtual #37                 // Method scala/runtime/StructuralCallSite.find:(Ljava/lang/Class;)Ljava/lang/reflect/Method;
      11: astore_2
      12: aload_2
      13: ifnull        18
      16: aload_2
      17: areturn
      18: getstatic     #43                 // Field scala/runtime/ScalaRunTime$.MODULE$:Lscala/runtime/ScalaRunTime$;
      21: aload_0
      22: ldc           #45                 // String foo
      24: aload_1
      25: invokevirtual #49                 // Method scala/runtime/StructuralCallSite.parameterTypes:()[Ljava/lang/Class;
      28: invokevirtual #55                 // Method java/lang/Class.getMethod:(Ljava/lang/String;[Ljava/lang/Class;)Ljava/lang/reflect/Method;
      31: invokevirtual #59                 // Method scala/runtime/ScalaRunTime$.ensureAccessible:(Ljava/lang/reflect/Method;)Ljava/lang/reflect/Method;
      34: astore_2
      35: aload_1
      36: aload_0
      37: aload_2
      38: invokevirtual #63                 // Method scala/runtime/StructuralCallSite.add:(Ljava/lang/Class;Ljava/lang/reflect/Method;)Ljava/lang/reflect/Method;
      41: pop
      42: aload_2
      43: areturn

When running on JDK9+, we should call m.canAccess(receiver) rather than m.canAccess(receiverClass).

First, we'll need to put some JDK-dependent code in ensureAccessible, and call canAccess via reflection or via a helper method that we compile under JDK9 in our build.

Then, we need to change the compilation of structural calls to pass the receiver, rather than reciever class down.

For source compatibility, we should probably leave the current signature of ensureAccessible alone (it could be used from userland) and create a new method.

@retronym
Copy link
Member

Class.getSimpleName now uses class metadata to find the simple name, rather than the fragile approach of parsing the full classname.

I think that test/files/run/t1167.scala is failing because we're emitting invalid metadata in the InnerClass attribute in Test1$$anonfun$testFunc$1, possibly because of an interaction with DelayedInit.

@som-snytt
Copy link
Contributor Author

The comment in the test is right, the javadoc says getSimpleName should be empty for anonymous classes. I haven't sorted what's expected here.

@som-snytt
Copy link
Contributor Author

@lrytz Thanks! I'd noticed the package change elsewhere, but too lazy to drill down here.

def tr[T](m: => T): String = try {
val r = m
if (r == null) "null"
else r.toString
} catch { case e: InternalError => e.getMessage }

def assertNotAnonymous(c: Class[_]) = {
val an = try {
def assertNotAnonymous(c: Class[_]) = assert8(!isAnonymous(c), s"$c is anonymous")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check also flipped.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this is just documenting the evolving status quo for java reflect interaction.

@som-snytt
Copy link
Contributor Author

Dread scala.reflect.internal.MissingRequirementError: object scala in compiler mirror not found.

	at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:170)
	at scala.reflect.internal.Definitions$DefinitionsClass.ScalaPackage$lzycompute(Definitions.scala:168)
	at scala.reflect.internal.Definitions$DefinitionsClass.ScalaPackage(Definitions.scala:168)
	at scala.reflect.internal.Definitions$DefinitionsClass.ScalaPackageClass$lzycompute(Definitions.scala:169)
	at scala.reflect.internal.Definitions$DefinitionsClass.ScalaPackageClass(Definitions.scala:169)
	at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1438)
	at scala.tools.nsc.Global$Run.<init>(Global.scala:1159)
	at scala.tools.nsc.doc.DocFactory.makeUniverse(DocFactory.scala:44)
	at scala.tools.nsc.scaladoc.HtmlFactoryTest$.createTemplates(HtmlFactoryTest.scala:37)
	at scala.tools.nsc.scaladoc.HtmlFactoryTest$.<init>(HtmlFactoryTest.scala:535)
	at scala.tools.nsc.scaladoc.HtmlFactoryTest$.<clinit>(HtmlFactoryTest.scala)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:374)
	at org.scalacheck.Platform$.loadModule(Platform.scala:69)
	at org.scalacheck.CustomScalaCheckRunner$BaseTask.<init>(CustomScalaCheckRunner.scala:45)
	at org.scalacheck.CustomScalaCheckRunner$$anon$2.<init>(CustomScalaCheckRunner.scala:89)
	at org.scalacheck.CustomScalaCheckRunner.checkPropTask(CustomScalaCheckRunner.scala:89)
	at org.scalacheck.CustomScalaCheckRunner.$anonfun$tasks$2(CustomScalaCheckRunner.scala:35)
	at scala.collection.ArrayOps$.map$extension(ArrayOps.scala:787)
	at org.scalacheck.CustomScalaCheckRunner.tasks(CustomScalaCheckRunner.scala:34)

@retronym
Copy link
Member

From the migration guide:

The application class loader is no longer an instance of URLClassLoader but, rather, of an internal class. It is the default loader for classes in modules that are neither Java SE nor JDK modules.

That probably foils:

def configureClassAndSourcePath(settings: Settings): Settings = {
val ourClassLoader = HtmlFactoryTest.getClass.getClassLoader
Thread.currentThread.getContextClassLoader match {
case loader: URLClassLoader =>
val paths = loader.getURLs.map(u => URLDecoder.decode(u.getPath))
settings.classpath.value = paths mkString java.io.File.pathSeparator
case loader =>
settings.embeddedDefaults(ourClassLoader) // Running in SBT without forking, we have to ask the SBT classloader for the classpath
}
settings
}

@retronym
Copy link
Member
retronym commented Jul 11, 2018

Something like this ought to be cross JDK compatible:

if (Thread.currentThread.getContextClassLoader == ClassLoader.getSystemClassLoader)
  settings.usejavacp.value = true
else
  settings.embeddedDefaults(ourClassLoader) // assume we're under SBT

@som-snytt
Copy link
Contributor Author

Thanks @retronym I dusted off hg and built jdk11, after spending time trying to remember scala.reflect.runtime.ReflectionUtils.show(classloader), which was the hard part. I also got Eclipse Photon, which is looking pretty nice in dark mode; I don't know if that means I'll never update Scala-IDE at the office; I was hanging on to Neon in case I ever needed it, but I failed to persuade the Eclipse guy at the office to even review any Scala code.

@som-snytt
Copy link
Contributor Author

@retronym OK to squash to one commit? Not sure any history worth preserving. I was about to look at osgi test breakage, but maybe one can merge before fixing all the things.

@retronym
Copy link
Member
retronym commented Jul 12, 2018 via email

@som-snytt som-snytt changed the title Tests on jdk10 [ci: last-only] Tests on jdk10 Jul 13, 2018
@som-snytt
Copy link
Contributor Author

Worth mentioning that zulu 10 crashed twice this week trying to sbt.

Copy link
Member
@retronym retronym left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks^10!

@retronym
Copy link
Member

What's left? OSGi reflection tests, I presume? Once testAll is happy, I propose we add openjdk10 to the Travis build matrix.

@som-snytt
Copy link
Contributor Author
som-snytt commented Jul 13, 2018

Yes, I keep running out of time windows. I'll pursue a couple of clean-ups that I forgot while squashing...

Edit: the weekend is coming up. Maybe book this PR and I'll follow up.

@@ -61,3 +62,67 @@ public void Test4$Foo12.name_$eq(java.lang.String)
99
dylan
2
#partest !java8
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wouldn't be brain surgery for partest --update-check to support existing filters, or possibly under a flag --check-jvm.

/** Assert on Java 8, but on later versions, just print if assert would fail. */
def assert8(b: => Boolean, msg: => Any) =
if (ScalaVersion(javaSpecVersion) == version8) assert(b, msg)
else if (!b) println(s"assert not $msg")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still useful or does it hide migration errors?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used, actually.

@som-snytt
Copy link
Contributor Author

First, fix the bane of my existence neg/checksensible.* on the first commit.

assert(callee.safeToInline == safeToInline)
=======
assertEquals("safeToInline", safeToInline, callee.safeToInline)
>>>>>>> 283e089... Fix caller sensitive test per lrytz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops.

Partest accepts version filter for
arbitrary `javaN` or `java9+` or
`!java8`.

Generally, avoid deprecated API.

Strip module name from output.

scalap classpath test for dot has only dot.

The anonymous class is now really incognito.

Ignore module reflection WARNINGs for now.

Fix path setting for scalacheck/scaladoc

The previous code attempted to reconstruct the class path
from the class loader, but the class loader implementations
change in Java 9. The more direct approach is -usejavacp
if no class loader has been created below the app CL, and
to use the test CL as a parent otherwise.
@som-snytt
Copy link
Contributor Author

Then fix the test munged by a git conflict.

Flags are extraneous for language features.

For single-source tests, flags can be inserted
at the top of the file in the form
`// scalac: -Xlint`.
@som-snytt som-snytt removed the WIP label Jul 14, 2018
@som-snytt som-snytt changed the title Tests on jdk10 Some tests on jdk10 Jul 14, 2018
@retronym retronym merged commit 0c15792 into scala:2.13.x Jul 16, 2018
@retronym
Copy link
Member

Consider the progress booked!

@SethTisue
Copy link
Member
SethTisue commented Jul 17, 2018

thank you so much, Som, for this epic PR!!

@som-snytt som-snytt deleted the issue/sd-522 branch August 5, 2018 21:06
def hasCallerSensitiveAnnotation(methodNode: MethodNode): Boolean =
methodNode.visibleAnnotations != null &&
methodNode.visibleAnnotations.stream.filter(ann =>
ann.desc == "Lsun/reflect/CallerSensitive;" || ann.desc == "Ljdk/internal/reflect/CallerSensitive;"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this fix the issue I described in the comment below?

scala/bug#10817 (comment)

If so, it would be nice to backport it to Scala 2.12.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clever to notice. But I thought only Spark was still stuck on 2.12. I'll try to make time to sort that out, next a modern version of Java is on my class path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Libraries typically support multiple Scala versions. Kafka supports Scala 2.11 and 2.12 today and it will support 2.13 soonish.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, that was my Spark joke for the day. Some people use sbt using 2.10. But thanks again, it's fun to have an opportunity to explore the Java 9/10/11 event horizon.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants
0