8000 Improve performance of the backend by retronym · Pull Request #5820 · scala/scala · GitHub
[go: up one dir, main page]

Skip to content

Improve performance of the backend #5820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8000
Merged
merged 7 commits into from
Apr 21, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Avoid excessive file stats during classfile writing
The existing implementation pessimistically checks that all
parent directories of the about-to-be-written class file are
indeed directories.

This commit bypasses this logic for the common case of writing
to a regular directory on disk, and optimistically assumes that
the parent directory exists. If an exception is thrown during
writing, it attempts to create the parent directory.

This still avoids a compiler crash if a parent directory is
actually a file, which is tested by the existing test,
`run/t5717.scala`.
  • Loading branch information
retronym committed Apr 3, 2017
commit 7a6dc1abbfc9afda27623dd43424c252dcec8088
8000
49 changes: 38 additions & 11 deletions src/compiler/scala/tools/nsc/backend/jvm/BytecodeWriters.scala
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,15 @@
package scala.tools.nsc
package backend.jvm

import java.io.{ DataOutputStream, FileOutputStream, IOException, File => JFile }
import java.io.{DataOutputStream, FileOutputStream, IOException, File => JFile}
import java.nio.file.{FileAlreadyExistsException, Files}
import java.nio.file.attribute.BasicFileAttributes

import scala.tools.nsc.io._
import java.util.jar.Attributes.Name

import scala.language.postfixOps
import scala.reflect.io.PlainNioFile

/** Can't output a file due to the state of the file system. */
class FileConflictException(msg: String, val file: AbstractFile) extends IOException(msg)
Expand All @@ -29,13 +34,25 @@ trait BytecodeWriters {
* @param clsName cls.getName
*/
def getFile(base: AbstractFile, clsName: String, suffix: String): AbstractFile = {
def ensureDirectory(dir: AbstractFile): AbstractFile =
if (dir.isDirectory) dir
else throw new FileConflictException(s"${base.path}/$clsName$suffix: ${dir.path} is not a directory", dir)
var dir = base
val pathParts = clsName.split("[./]").toList
for (part <- pathParts.init) dir = ensureDirectory(dir) subdirectoryNamed part
ensureDirectory(dir) fileNamed pathParts.last + suffix
if (base.file != null) {
fastGetFile(base, clsName, suffix)
} else {
def ensureDirectory(dir: AbstractFile): AbstractFile =
if (dir.isDirectory) dir
else throw new FileConflictException(s"${base.path}/$clsName$suffix: ${dir.path} is not a directory", dir)
var dir = base
val pathParts = clsName.split("[./]").toList
for (part <- pathParts.init) dir = ensureDirectory(dir) subdirectoryNamed part
ensureDirectory(dir) fileNamed pathParts.last + suffix
}
}
private def fastGetFile(base: AbstractFile, clsName: String, suffix: String) = {
val index = clsName.lastIndexOf('/')
val (packageName, simpleName) = if (index > 0) {
(clsName.substring(0, index), clsName.substring(index + 1))
} else ("", clsName)
val directory = base.file.toPath.resolve(packageName)
new PlainNioFile(directory.resolve(simpleName + suffix))
}
def getFile(sym: Symbol, clsName: String, suffix: String): AbstractFile =
getFile(outputDirectory(sym), clsName, suffix)
Expand Down Expand Up @@ -118,10 +135,20 @@ trait BytecodeWriters {
def writeClass(label: String, jclassName: String, jclassBytes: Array[Byte], outfile: AbstractFile) {
assert(outfile != null,
"Precisely this override requires its invoker to hand out a non-null AbstractFile.")
val outstream = new DataOutputStream(outfile.bufferedOutput)
if (outfile.file != null) {
try {
Files.write(outfile.file.toPath, jclassBytes)
} catch {
case _: java.nio.file.NoSuchFileException =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be too slow to check java.nio.file.Files.exists(p.getParent)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found some small but measurable improvement from this approach, as there are typically many classfiles per directory, which makes the optimism pays off. I could split this into two commits so that we could reproduce that measurement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that there is a better solution in #5815, that will only check a given directory once, and will not call createDirectories unless it the first time that the directory is reached

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lrytz each stat call is expensive in windows particually

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the approach of caching the results of file stats, but opted not to do that in this PR because it takes a bit of care to get the lifecycle of that cache sorted out (we need to start fresh on a new compiler run, but the current design of ByteCodeWriters didn't have a hook for that part of the lifecycle.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@retronym it is hooked in this my PR as the OutputDirectories are created new for each run

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have split out an alternative as #5834

will run performance test overnight and post up is successful

Files.createDirectories(outfile.file.toPath.getParent)
Files.write(outfile.file.toPath, jclassBytes)
}
} else {
val outstream = new DataOutputStream(outfile.bufferedOutput)
try outstream.write(jclassBytes, 0, jclassBytes.length)
finally outstream.close()
}

try outstream.write(jclassBytes, 0, jclassBytes.length)
finally outstream.close()
informProgress("wrote '" + label + "' to " + outfile)
}
}
Expand Down
4 changes: 4 additions & 0 deletions src/compiler/scala/tools/nsc/backend/jvm/GenBCode.scala
Original file line number Diff line number Diff line change
Expand Up @@ -410,6 +410,10 @@ abstract class GenBCode extends BCodeSyncAndTry {
catch {
case e: FileConflictException =>
error(s"error writing $jclassName: ${e.getMessage}")
case e: java.nio.file.FileSystemException =>
if (settings.debug)
e.printStackTrace()
error(s"error writing $jclassName: ${e.getClass.getName} ${e.getMessage}")
}
}
}
Expand Down
0