Java Compiler API

Andrey RedkoSeptember 18th, 2015Last Updated: December 6th, 2023

11 2,156 9 minutes read

This article is part of our Academy Course titled Advanced Java.

This course is designed to help you make the most effective use of Java. It discusses advanced topics, including object creation, concurrency, serialization, reflection and many more. It will guide you through your journey to Java mastery! Check it out here!

1. Introduction
2. Java Compiler API
3. Annotation Processors
4. Element Scanners
5. Java Compiler Tree API
6. What’s next
7. Download

1. Introduction

In this part of the tutorial we are going to take 10000 feet view of the Java Compiler API. This API provides programmatic access to the Java compiler itself and allows developers to compile Java classes from source files on the fly from application code.

More interestingly, we also are going to walk through the Java Compiler Tree API, which provides access to Java syntax parser functionality. By using this API, Java developers have the ability to directly plug into syntax parsing phase and post-analyze Java source code being compiled. It is a very powerful API which is heavily utilized by many static code analysis tools.

The Java Compiler API also supports annotation processing (for more details please refer to part 5 of the tutorial, How and when to use Enums and Annotations, more to come in part 14 of the tutorial, Annotation Processors) and is split between three different packages, shown in the table below.

Package	Description
javax.annotation.processing	Annotation processing.
javax.lang.model	Language model used in annotation processing and Compiler Tree API (including Java language elements, types and utility classes).
javax.tools	Java Compiler API itself.

On the other side, the Java Compiler Tree API is hosted under the com.sun.source package and, following Java standard library naming conventions, is considered to be non-standard (proprietary or internal). In general, these APIs are not very well documented or supported and could change any time. Moreover, they are tied to the particular JDK/JRE version and may limit the portability of the applications which use them.

2. Java Compiler API

Our exploration will start from the Java Compiler API, which is quite well documented and easy to use. The entry point into Java Compiler API is the ToolProvider class, which allows to obtain the Java compiler instance available in the system (the official documentation is a great starting point to get familiarized with the typical usage scenarios). For example:

final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();        
for( final SourceVersion version: compiler.getSourceVersions() ) {
    System.out.println( version );
}

This small code snippet gets the Java compiler instances and prints out on the console a list of supported Java source versions. For Java 7 compiler, the output looks like this:

RELEASE_3
RELEASE_4
RELEASE_5
RELEASE_6
RELEASE_7

It corresponds to more well-known Java version scheme: 1.3, 1.4, 5, 6 and 7. For Java 8 compiler, the list of supported versions looks a bit longer:

RELEASE_3
RELEASE_4
RELEASE_5
RELEASE_6
RELEASE_7
RELEASE_8

Once the instance of Java compiler is available, it could be used to perform different compilation tasks over the set of Java source files. But before that, the set of compilation units and diagnostics collector (to collect all encountered compilation errors) should be prepared. To experiment with, we are going to compile this simple Java class stored in SampleClass.java source file:

public class SampleClass {
    public static void main(String[] args) {
        System.out.println( "SampleClass has been compiled!" );
    }
}

Having this source file created, let us instantiate the diagnostics collector and configure the list of source files (which includes only SampleClass.java) to compile.

final DiagnosticCollector< JavaFileObject > diagnostics = new DiagnosticCollector<>();
final StandardJavaFileManager manager = compiler.getStandardFileManager( 
    diagnostics, null, null );
        
final File file = new File( 
    CompilerExample.class.getResource("/SampleClass.java").toURI() );

final Iterable< ? extends JavaFileObject > sources = 
    manager.getJavaFileObjectsFromFiles( Arrays.asList( file ) );

Once the preparation is done, the last step is basically to invoke Java compiler task, passing the diagnostics collector and list of source files to it, for example:

final CompilationTask task = compiler.getTask( null, manager, diagnostics, 
    null, null, sources );            
task.call();

That is, basically, it. After the compilation task finishes, the SampleClass.class should be available in the target/classes folder. We can run it to make sure compilation has been performed successfully:

java -cp target/classes SampleClass

The following output will be shown on the console, confirming that the source file has been properly compiled into byte code:

SampleClass has been compiled!

In case of any errors encountered during the compilation process, they will become available through the diagnostics collector (by default, any additional compiler output will be printed into System.err too). To illustrate that, let us try to compile the sample Java source file which by intention contains some errors (SampleClassWithErrors.java):

private class SampleClassWithErrors {
    public static void main(String[] args) {
        System.out.println( "SampleClass has been compiled!" );
    }
}

The compilation process should fail and error message (including line number and source file name) could be retrieved from diagnostics collector, for example:

for( final Diagnostic< ? extends JavaFileObject > diagnostic: 
            diagnostics.getDiagnostics() ) {
            
        System.out.format("%s, line %d in %s", 
            diagnostic.getMessage( null ),
            diagnostic.getLineNumber(),
            diagnostic.getSource().getName() );
}

Invoking the compilation task on the SampleClassWithErrors.java source file will print out on the console the following sample error description:

modifier private not allowed here, line 1 in SampleClassWithErrors.java

Last but not least, to properly finish up working with the Java Compiler API, please do not forget to close file manager:

manager.close();

Or even better, always use try-with-resources construct (which has been covered in part 8 of the tutorial, How and when to use Exceptions):

try( final StandardJavaFileManager manager = 
        compiler.getStandardFileManager( diagnostics, null, null ) ) {
    // Implementation here            
}

In a nutshell, those are typical usage scenarios of Java Compiler API. When dealing with more complicated examples, there is a couple of subtle but quite important details which could speed up the compilation process significantly. To read more about that please refer to the official documentation.

3. Annotation Processors

Fortunately, the compilation process is not limited to compilation only. Java Compiler supports annotation processors which could be thought as a compiler plugins. As the name suggests, annotation processors could perform addition processing (usually driven by annotations) over the code being compiled.

In the part 14 of the tutorial, Annotation Processors, we are going to have much more comprehensive coverage and examples of annotation processors. For the moment, please refer to official documentation to get more details.

4. Element Scanners

Sometimes it becomes necessary to perform shallow analysis across all language elements (classes, methods/constructors, fields, parameters, variables, …) during the compilation process. Specifically for that, the Java Compiler API provides the concept of element scanners. The element scanners are built around visitor pattern and basically require the implementation of the single scanner (and visitor). To simplify the implementation, a set of base classes is kindly provided.

The example we are going to develop is simple enough to show off the basic concepts of element scanners usage and is going to count all classes, methods and fields across all compilation units. The basic scanner / visitor implementation extends ElementScanner7 class and overrides only the methods it is interested in:

public class CountClassesMethodsFieldsScanner extends ElementScanner7< Void, Void > {
    private int numberOfClasses;
    private int numberOfMethods;
    private int numberOfFields;
    
    public Void visitType( final TypeElement type, final Void p ) {
        ++numberOfClasses;
        return super.visitType( type, p );
    }

    public Void visitExecutable( final ExecutableElement executable, final Void p ) {
        ++numberOfMethods;
        return super.visitExecutable( executable, p );
    }

    public Void visitVariable( final VariableElement variable, final Void p ) {
        if ( variable.getEnclosingElement().getKind() == ElementKind.CLASS ) {
            ++numberOfFields;
        }
        
        return super.visitVariable( variable, p );
    }
}

Quick note on the element scanners: the family of ElementScannerX classes corresponds to the particular Java version. For instance, ElementScanner8 corresponds to Java 8, ElementScanner7 corresponds to Java 7, ElementScanner6 corresponds to Java 6, and so forth. All those classes do have a family of visitXxx methods which include:

Method	Description
visitPackage	Visits a package element.
visitType	Visits a type element.
visitVariable	Visits a variable element.
visitExecutable	Visits an executable element.
visitTypeParameter	Visits a type parameter element.

One of the ways to invoke the scanner (and visitors) during the compilation process is by using the annotation processor. Let us define one by extending AbstractProcessor class (please notice that annotation processors are also tight to particular Java version, in our case Java 7):

@SupportedSourceVersion( SourceVersion.RELEASE_7 )
@SupportedAnnotationTypes( "*" )
public class CountElementsProcessor extends AbstractProcessor {
    private final CountClassesMethodsFieldsScanner scanner;
    
    public CountElementsProcessor( final CountClassesMethodsFieldsScanner scanner ) {
        this.scanner = scanner;
    }
    
    public boolean process( final Set< ? extends TypeElement > types, 
            final RoundEnvironment environment ) {

        if( !environment.processingOver() ) {
            for( final Element element: environment.getRootElements() ) {
                scanner.scan( element );
            }
        }
        
        return true;
    }
}

Basically, the annotation processor just delegates all the hard work to the scanner implementation we have defined before (in the part 14 of the tutorial, Annotation Processors, we are going to have much more comprehensive coverage and examples of annotation processors).

The SampleClassToParse.java file is the example which we are going to compile and count all classes, methods/constructors and fields in:

public class SampleClassToParse {
    private String str;
    
    private static class InnerClass {     
        private int number;
        
        public void method() {
            int i = 0;
            
            try {
                // Some implementation here
            } catch( final Throwable ex ) {
                // Some implementation here
            }
        }
    }
    
    public static void main( String[] args ) {
        System.out.println( "SampleClassToParse has been compiled!" );
    }
}

The compilation procedure looks exactly like we have seen in the Java Compiler API section. The only difference is that compilation task should be configured with annotation processor instance(s). To illustrate that, let us take a look on the code snippet below:

final CountClassesMethodsFieldsScanner scanner = new CountClassesMethodsFieldsScanner();
final CountElementsProcessor processor = new CountElementsProcessor( scanner );

final CompilationTask task = compiler.getTask( null, manager, diagnostics, 
    null, null, sources );
task.setProcessors( Arrays.asList( processor ) );
task.call();

System.out.format( "Classes %d, methods/constructors %d, fields %d",
    scanner.getNumberOfClasses(),
    scanner.getNumberOfMethods(), 
    scanner.getNumberOfFields() );

Executing the compilation task against SampleClassToParse.java source file will output the following message in the console:

Classes 2, methods/constructors 4, fields 2

It makes sense: there are two classes declared, SampleClassToParse and InnerClass. SampleClassToParse class has default constructor (defined implicitly), method main and field str. In turn, InnerClass class also has default constructor (defined implicitly), method method and field number.

This example is very naive but its goal is not to demonstrate something fancy but rather to introduce the foundational concepts (the part 14 of the tutorial, Annotation Processors, will include more complete examples).

5. Java Compiler Tree API

Element scanners are quite useful but the details they provide access to are quite limited. Once in a while it becomes necessary to parse Java source files into abstract syntax trees (or AST) and perform more deep analysis. Java Compiler Tree API is the tool we need to make it happen. Java Compiler Tree API works closely with Java Compiler API and uses javax.lang.model package.

The usage of Java Compiler Tree API is very similar to the element scanners from section Element Scanners and is built following the same patterns. Let us reuse the sample source file SampleClassToParse.java from Element Scanners section and count how many empty try/catch blocks are present across all compilation units. To do that, we have to define tree path scanner (and visitor), similarly to element scanner (and visitor) by extending TreePathScanner base class.

public class EmptyTryBlockScanner extends TreePathScanner< Object, Trees > {
    private int numberOfEmptyTryBlocks;
    
    @Override
    public Object visitTry(final TryTree tree, Trees trees) {
        if( tree.getBlock().getStatements().isEmpty() ){
            ++numberOfEmptyTryBlocks;
        }
        
        return super.visitTry( tree, trees );
    }
    
    public int getNumberOfEmptyTryBlocks() {
        return numberOfEmptyTryBlocks;
    }
}

The number of visitXxx methods is significantly richer (around 50 methods) comparing to element scanners and covers all Java language syntax constructs. As with element scanners, one of the ways to invoke tree path scanners is also by defining dedicate annotation processor, for example:

@SupportedSourceVersion( SourceVersion.RELEASE_7 )
@SupportedAnnotationTypes( "*" )
public class EmptyTryBlockProcessor extends AbstractProcessor {
    private final EmptyTryBlockScanner scanner;
    private Trees trees;
    
    public EmptyTryBlockProcessor( final EmptyTryBlockScanner scanner ) {
        this.scanner = scanner;
    }
    
    @Override
    public synchronized void init( final ProcessingEnvironment processingEnvironment ) {
        super.init( processingEnvironment );
        trees = Trees.instance( processingEnvironment );
    }
    
    public boolean process( final Set< ? extends TypeElement > types, 
            final RoundEnvironment environment ) {

        if( !environment.processingOver() ) {
            for( final Element element: environment.getRootElements() ) {
                scanner.scan( trees.getPath( element ), trees );
            }
        }
        
        return true;
    }
}

The initialization procedure became a little bit more complex as we have to obtain the instance of Trees class and convert each element into tree path representation. At this moment, the compilation steps should look very familiar and be clear enough. To make it a little bit more interesting, let us run it against all source files we have experimenting with so far: SampleClassToParse.java and SampleClass.java.

final EmptyTryBlockScanner scanner = new EmptyTryBlockScanner();
final EmptyTryBlockProcessor processor = new EmptyTryBlockProcessor( scanner );

final Iterable<? extends JavaFileObject> sources = manager.getJavaFileObjectsFromFiles( 
    Arrays.asList( 
        new File(CompilerExample.class.getResource("/SampleClassToParse.java").toURI()),
        new File(CompilerExample.class.getResource("/SampleClass.java").toURI())
    ) 
);

final CompilationTask task = compiler.getTask( null, manager, diagnostics, 
    null, null, sources );
task.setProcessors( Arrays.asList( processor ) );
task.call();

System.out.format( "Empty try/catch blocks: %d", scanner.getNumberOfEmptyTryBlocks() );

Once run against multiple source files, the code snippet above is going to print the following output in the console:

Empty try/catch blocks: 1

The Java Compiler Tree API may look a little bit low-level and it certainly is. Plus, being an internal API, it does not have well-supported documentation available. However, it gives the full access to the abstract syntax trees and it is a life-saver when you need to perform deep source code analysis and post-processing.

6. What’s next

In this part of the tutorial we have looked at programmatic access to Java Compiler API from within the Java applications. We also dug deeper, touched annotation processors and uncovered Java Compiler Tree API which provides the full access to abstract syntax trees of the Java source files being compiled (compilation units). In the next part of the tutorial we are going to continue in the same vein and take a closer look on annotation processors and their applicability.

7. Download

This was a lesson for the Java Compiler API, part 13 of Advanced Java Course. You can download the source code of the lesson here: advanced-java-part-13