JEP 139: Enhance javac to Improve Build Speed

Summary

Reduce the time required to build the JDK and enable incremental builds by modifying the Java compiler to run on all available cores in a single persistent process, track package and class dependences between builds, automatically generate header files for native methods, and clean up class and header files that are no longer needed.

Goals

The top level goals are:

1. Increase build speed by having javac use all cores and reuse javac in a server process.
2. Simplify work for developers by having javac build incrementally.

This project is part of a larger effort to improve the build infrastructure of the JDK.

The improvements to javac will be internal and not available through the public api of the javac launcher. Instead an internal wrapper program called smart-javac (or sjavac for short) will house the new functionality. Eventually, when the javac wrapper features have stabilized, they can be proposed in a future JEP to be moved to the public api of javac. This would allow all Java developers to take advantage of the improvements.

Non-Goals

This project only concerns the changes needed in javac and the new wrapper. It does not cover the changes needed in the JDK Makefiles to take advantage of these changes; those are described in JEP 138: Autoconf-Based Build System.

This project will not concern changes needed in javac to speed up Javadoc generation.

Success Metrics

All cores should be used when compiling Java sources and there should be a speedup in build performance when using multiple cores.

The workload distributed on the different cores will not be perfectly balanced and it is known that the same work will be recomputed several times. But the changes supported by this JEP will enable us to successively improve the sharing of work between the cores within javac.

An incremental build should recompile only the changed packages and their dependencies.

After an increment build has been done, where classes or native methods were removed, the output directories should be clean; i.e., no classes or C-header files corresponding to the removed sources should remain.

Motivation

Building the complete OpenJDK is unnecessarily slow. This puts an extra burden on developers and build systems. As a result, developers check out and build just a part of the source code, since the product as a whole takes too long to build.

Description

The internal smart-javac wrapper will probably be invoked like this:

$ java -jar sjavac.jar -classpath ... -sourcepath ... -pkg '*' \
       -j all -h headerdir -d outputdir

This will compile all source files found in the sourcepath with package names matching anything ('*') using all (-j) cores. A database file will be created in the (-d) outputdir called .javac_state which will contain all information necessary to do a fast incremental compile, with proper cleanup of disappearing classes and C-headers as well as correct dependency tracking.

The smart-javac wrapper implements multi core support by creating a JavaCompiler instance for each core. The source code to be compiled is then split into packages and randomly distributed to the JavaCompilers. If a randomly selected package has dependencies, then these dependencies will be compiled automatically, but not written to disk, i.e., -Ximplicit:none. If an implicitly compiled dependency is later requested as part of a randomly selected package then the implicit work is not wasted; instead the already-compiled dependency is written to disk.

Since the initial compile does not know which packages a package depends upon, a random distribution of work is the best we can do. Therefore java.lang.Object will be recompiled as many times as there are cores, but only one of the JavaCompilers will be responsible for writing java.lang.Object to disk. Enough packages are independent of each other to make this a workable strategy for making use of multiple cores.

When we improve javac in the future (not part of this JEP), more and more work will be shared between the JavaCompilers and eventually we will reach the state where java.lang.Object is only compiled once.

javah will be automatically run on any class that contains native methods and the generated C-headers will be put in the (-h) headerdir. A new annotation @ForceNativeHeader is used for classes that have final static primitives that need to be exported to JNI, but no native methods.

To avoid restarting javac and losing optimizations done by the JVM, the smart-javac wrapper supports a -server option. This option will spawn a background javac server so that each subsequent smart-javac wrapper invocation that refers to the same portfile will reuse the same server.

The -server arguments are:

• portfile = where to store the TCP port used
• logfile = where to store the output from javac, defaults to portfile+".logfile"
• stdouterrfile = where to store output from the server, defaults to portfile+".stdouterr"
• javac = the path to the javac to launch by the server, with spaces and commas replaced by %20 and %2C.

Example:

-server:portfile=/tmp/jdk.port,javac=/usr/local/bin/java%20\
-jar%20/tmp/openjdk/langtools/dist/lib/bootstrap/sjavac.jar

Since javac cannot currently share state between concurrent compilations, each additional core will consume roughly as much memory as a single invocation of javac. Improved sharing between the cores will be successively introduced after this JEP is completed. The -j option can be used to limit the number of cores and thus the memory usage.

The javac server stays in memory after all compilations have finished. The server will automatically shut down and free memory and other resources after 30 seconds of inactivity.

The server runs as the same user as the normal javac compiler, and thus has the same privileges and possibility to write to the build output directory. Unlike the normal javac compiler, a compilation can be trigged by connecting to the server through a TCP port. Only the command line, and not the source code, is sent over TCP.

A potential security risk is that an attacker could add the compilation of some malicious piece of code, which would appear in the output directory. To alleviate this risk, we will use the several measures:

• Open a new TCP port each time; the port number is stored in the portfile.
• Only allow connections from localhost.
• Require a unique cookie to be presented to the server before any compilation will occur.

The cookie is a 64 bit random integer stored in the portfile. The portfile has typical temporary file permissions, i.e., only the owner is allowed to read from it or write to it.

Alternatives

Instead of making javac properly parallel, we could start several single-threaded compilations of different and independent java packages in parallel. This would not require any changes to javac, but it would be much harder to get the Makefiles correct, and it would not give as much speed improvement.

Testing

The build infrastructure project will test that the output of the old build system and the new build system is identical. This will make sure that the smart-javac wrapper generates identical output for the same source files.

None of the new options are going to be public, so no tests need to be added to the javac test suite, but more-specific tests for the smart-javac wrapper will be added to the langtools test directory.

Risks and Assumptions

Resulting product is incorrect

• Risk: Javac changes causes incorrect bits to be build
• Mitigation plan: Test resulting build properly

The old build makefiles will be available concurrently with the new makefiles to simplify comparing of the old and new bits.

Dependences

This JEP does not depend on any other changes. It forms the basis for JEP 138: Autoconf-Based Build System, which will use this new feature in javac to speed up JDK builds. The new makefiles can use an unmodified javac, but they will not achieve the desired speedup and incremental build support unless this JEP is completed.

Impact

• Compatibility: Low impact. We will not add new arguments to javac.
• Security: Low impact. In the description section, there is a discussion about the security aspects of opening up a server for compilation jobs.
• I18n/L10n: The new smart-javac wrapper features will most likely result in a few new messages; we will not fully translate these since smart-javac is not yet part of any public api.
• Testing: Apart from the build itself, separate test cases for the different smart-javac options should be written.