PC invalidates previous work on new run [ci: last-only] #10867

som-snytt · 2024-09-20T01:10:55Z

The getDocComment command creates a newTyperRun as of #2387. Then checkForMoreWork notices there is a new run after pollForWork and then will demandNewCompilerRun.

However, a second execution of the test would fail because the unit is deemed up-to-date. This commit updates the minRunId which governs whether the unit is stale when the run is created.

I verified that the test would always fail the second time under Linux but occasionally the first time under Windows.

I don't see a race condition, but the code uses currentTimeMillis instead of nanoTime for timeouts, so maybe it's a Windows clock issue. Or maybe a timing issue (what the PC thread completes before the next ask) causes a different code path. (Two guesses.)

The test comment somewhere says "check you can get a comment one time". The original PR was #2156.

My previous fix was to reset all sources at demandNewCompilerRun, which would likely be heavy-handed.

~~Probably the change at demandNewCompilerRun can be refined, as it matters just for doc comments.~~

askParse is tweaked to submit one file at a time, since otherwise the parser callback on the first job will look for work and start the next parse. (In this test, the chain stopped there because the second file was trivial and invoked no callback.)

Fixes scala/scala-dev#840

lrytz

Looks good / safe to me, I don't fully follow (what are the "parser callbacks"?)

lrytz · 2024-09-20T11:32:32Z

src/interactive/scala/tools/nsc/interactive/Global.scala

@@ -301,6 +301,7 @@ class Global(settings: Settings, _reporter: Reporter, projectName: String = "")
  def isOutOfDate: Boolean = outOfDate

  def demandNewCompilerRun() = {
+    if (!lastWasReload) allSources.foreach(getUnit(_).foreach(reset(_)))


would it be equivalent to add a boolean param to demandNewCompilerRun and set it true at the callsite in reload?

som-snytt · 2024-09-20T14:48:17Z

There is signalParseProgress and then signalDone when a tree is typed (TIL).

som-snytt · 2024-09-20T18:17:22Z

BTW I was curious (or annoyed) about the test; it's not my intention to muck with (or break) the PC. I assumed incorrectly that it would be a mechanism problem that required no special knowledge of PC semantics and operations.

som-snytt · 2025-05-07T06:05:19Z

I was following the "outdated symbols in the IDE" comment to collectImplicitImports which leads to the infamous flaky PC test, i.e., back to here. (I'm considering the extra imported test in isQualifyingImplicit, whereby an import would always be shadowed by an entry in the current scope. That had an exists to combat staleness. I was able to make the stale object test fail on the shadowing PR, but I've forgotten the precious test number, a dir for separate compilation; and I've forgotten how I fixed the broken lookup. Oh I left a code comment, how clever.

 // if lookup fails, it's a stale symbol problem, not our problem (pos/t5639)

which is why that doesn't drop on failed lookup, only on successful lookup of the wrong (shadowing) symbol.

(I was recently reminded to look again at the 2.13.15 PR to warn about "phantom object" from the classpath: the class got a module, so the same module defined in source is hidden; the method of detection is to look in the package owner.)

The test in isQualifyingImiplicit looks benign and correct, except for the staleness caveat. (Not sure it's much more efficient than lookup test.)

The change in collectImplicitImports to use import's qual.tpe (the package) instead of the pre (the package object). Actually the new shadowed check says

            // ii.pre or skipPackageObject
            val pre = if (ii.pre.typeSymbol.isPackageObjectClass) ii.pre.typeSymbol.owner.module.info else ii.pre

som-snytt · 2025-05-21T02:16:09Z

I changed that counter to 1000 and ran ok, then back to 2 and it

reload: Base.scala, Derived.scala
Unexpected foo method (method foo) comment: None
Unexpected Derived class comment:None
Unexpected Derived class comment:None

for

!!  7 - presentation/dock                         [output differs][duration 5.99s]

That is the infamous flaky half of the doc test, transplanted to dock.

Edit: just noticed that 1000 is not a stress test, it's a retry! I flipped the boolean so it runs to failure, so I can actually test it.

I'm pretty sure Homer Simpson works with booleans at the power plant. He may even be a "boolean specialist".

som-snytt · 2025-05-21T02:21:02Z

I think the test is just taunting me, trying to draw me out into the open ocean, like a scene from Moby Dick. Maybe there is a reason I called the second test dock, where it is dry and safe.

som-snytt · 2025-05-22T01:32:05Z

Before and after log for the "sequential parsing" change.

[doctest] new typer run
[doctest] added unit for dock/src/p/Base.scala
[doctest] added unit for dock/src/p/Derived.scala
[doctest] responded, delay = 8ms
[doctest] done with work item: reload List(Base.scala, Derived.scala)
[doctest] quitting work item: reload List(Base.scala, Derived.scala), delay = 0ms
[doctest][Starting new presentation compiler type checking pass]
[doctest] parsing: Base.scala
[doctest] some work at node 2 current = 2
[doctest] picked up work item at RangePosition(dock/src/p/Base.scala, 131, 135, 146): getParsedEntered Base.scala, keepLoaded = true, delay = 16ms
[doctest] parsing: Base.scala
[doctest] some work at node 3 current = 3
[doctest] picked up work item at RangePosition(dock/src/p/Base.scala, 131, 135, 146): getParsedEntered Derived.scala, keepLoaded = true, delay = 2ms
[doctest] parsing: Derived.scala
[doctest] responded, delay = 18ms
[doctest] done with work item: getParsedEntered Derived.scala, keepLoaded = true
[doctest] quitting work item: getParsedEntered Derived.scala, keepLoaded = true, delay = 0ms
[doctest] responded, delay = 2ms
[doctest] done with work item: getParsedEntered Base.scala, keepLoaded = true
[doctest] quitting work item: getParsedEntered Base.scala, keepLoaded = true, delay = 0ms
[doctest] type checking: Base.scala
[doctest] Typer signal done, checking for more work...


[doctest][reload: List(Base.scala, Derived.scala)]
[doctest] new typer run
[doctest] added unit for dock/src/p/Base.scala
[doctest] added unit for dock/src/p/Derived.scala
[doctest] responded, delay = 4ms
[doctest] done with work item: reload List(Base.scala, Derived.scala)
[doctest] quitting work item: reload List(Base.scala, Derived.scala), delay = 0ms
[doctest][Starting new presentation compiler type checking pass]
[doctest] parsing: Base.scala
[doctest] some work at node 2 current = 2
[doctest] picked up work item at RangePosition(dock/src/p/Base.scala, 131, 135, 146): getParsedEntered Base.scala, keepLoaded = true, delay = 12ms
[doctest] parsing: Base.scala
[doctest] responded, delay = 12ms
[doctest] done with work item: getParsedEntered Base.scala, keepLoaded = true
[doctest] quitting work item: getParsedEntered Base.scala, keepLoaded = true, delay = 0ms
[doctest] parsing: Derived.scala
[doctest] some work at node 4 current = 4
[doctest] picked up work item at RangePosition(dock/src/p/Derived.scala, 93, 97, 105): getParsedEntered Derived.scala, keepLoaded = true, delay = 2ms
[doctest] parsing: Derived.scala
[doctest] responded, delay = 1ms
[doctest] done with work item: getParsedEntered Derived.scala, keepLoaded = true
[doctest] quitting work item: getParsedEntered Derived.scala, keepLoaded = true, delay = 0ms
[doctest] type checking: Base.scala
[doctest] Typer signal done, checking for more work...

som-snytt · 2025-05-22T05:24:46Z

Never mind. 10K rep did not complete on Windows 10

reload: Base.scala, Derived.scala
reload: Base.scala, Derived.scala
Missing foo method (method foo) comment
done at 1214

I'll check the other two ideas (brittle wait and not using nanoTime).

scala-jenkins added this to the 2.13.16 milestone Sep 20, 2024

lrytz reviewed Sep 20, 2024

View reviewed changes

SethTisue modified the milestones: 2.13.16, 2.13.17 Dec 12, 2024

som-snytt force-pushed the sd/840-flaky-presentation-doc branch from 27a4ae7 to 11a33e4 Compare February 4, 2025 19:43

som-snytt mentioned this pull request Apr 14, 2025

Use type of function literal param for PF synthesis [ci: last-only] #10486

Merged

som-snytt mentioned this pull request May 20, 2025

Use List.fill instead of range [ci: last-only] #11059

Merged

som-snytt added 3 commits May 21, 2025 17:15

Refactor PC thread, pollForWork, name the project

2ec695d

Bump min run ID for staleness after getDocComment

32e2fc5

Use sequential parsing of sources in PC

4bbd250

som-snytt changed the title ~~PC invalidates previous work on new run~~ PC invalidates previous work on new run [ci: last-only] May 22, 2025

som-snytt force-pushed the sd/840-flaky-presentation-doc branch from 11a33e4 to 4bbd250 Compare May 22, 2025 00:50

som-snytt marked this pull request as ready for review May 22, 2025 03:17

som-snytt marked this pull request as draft May 22, 2025 05:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PC invalidates previous work on new run [ci: last-only] #10867

PC invalidates previous work on new run [ci: last-only] #10867

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PC invalidates previous work on new run [ci: last-only] #10867

Are you sure you want to change the base?

PC invalidates previous work on new run [ci: last-only] #10867

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!