How to do Continuous Integration with Java 8, NetBeans Platform 8, Jenkins, Jacoco and Sonar

Marco Di StefanoApril 17th, 2014Last Updated: April 16th, 2014

4 174 7 minutes read

Intro

Java 8 is there, the promised revolution is finally released, and I am sure that a lot of you are having in mind the same question “Should I use it in my project?”.

Well, I had the same question for few months and today that I have an answer I would like to share it with you. A lot of aspects have been influencing this decision but in this post I want to focus on one in particular that is:

Can I continue to do Continuous Integration with Java 8 and NetBeans Platform?

The main question was around the maturity of the tools necessary to do CI, and how easy was to integrate that with the ant build scripts of the NetBeans Platform.

Fortunately, we found that it is possible and easy to do!

I would also thanks Alberto Requena Sanchez for his contribution on this article.

The Technical Environment

Working in a project where Safety & Quality are the main drivers, CI is vital.

For this reason I started with my team a “proof of concept” to show that the following technologies were ready to work together:

Java 8, NetBeans 8.0 & Ant
JUnit 4 & Jacoco 0.7.1
Jenkins & Sonar 4.2

Scope of this post is to explain all the steps done to install and setup the necessary tools to have a completely working CI server for Java 8. Note that the proof has been done on a developer machine on Windows 7, but is easy to do the same in a Linux server.

The next diagram shows at high level the architecture that will be described in the post.

Java 8, NetBeans 8.0 & Ant

Java 8 is released, get it here, install it, study it (preferable) and start to use it!

We are using the NetBeans Platform 8.0 to create a modular application. This application has a Multi Layered Architecture where each layer is a Suite of Modules, and where the final executable is just an integrated set of Suites.

We are using Ant to build our projects, but if you are using Maven the procedure can even be simplified since the Sonar integration in Jenkins can be done via a plugin that uses Maven.

JUnit 4 & Jacoco 0.7.1

Naturally, we are doing unit tests, and for this reason we use JUnit 4. It is well integrated everywhere, specially in NetBeans.

Jacoco is a great tool for the generation of code coverage and since version 0.7.1 it fully supports Java 8.

Jenkins & Sonar 4.2

Jenkins is the engine of our CI server, it will integrate with all the above described technologies without any issue. The tested version is 1.554.

Sonar is doing all the quality analysis of the code. The release 4.2 has a full compatibility with Java 8.

Using Sonar with Ant needs a small library that contains the target to be integrated in Jenkins. If you are using Maven instead you can just install the plugin for Maven.

Starting the puzzle

Step 1 – NetBeans

Install Java 8 & NetBeans 8.0
Create a module suite with several modules, several classes and several jUnit tests
Commit the code into your source code version management server
Inside the harness of NetBeans
Create a folder in the harness named “jacoco-0.7.1″ containing the downloaded jacoco jars
Create a folder in the harness named “sonar-ant-task” and put inside the downloaded sonar ant jars
Create a file in the harness named sonar-jacoco-module.xml and paste the following code inside:

<?xml version="1.0" encoding="UTF-8"?>
<!--

-->
<project name="sonar-jacoco-module" basedir="." xmlns:jacoco="antlib:org.jacoco.ant" xmlns:sonar="antlib:org.sonar.ant">
<description>Builds the module suite otherSuite.</description>

<property name="jacoco.dir" location="${nbplatform.default.harness.dir}/jacoco-0.7.1"/>
<property name="result.exec.file" location="${jacoco.dir}/jacoco.exec"/>
<property name="build.test.results.dir" location="build/test/unit/results"/>

<property file="nbproject/project.properties"/>

<!-- Step 1: Import JaCoCo Ant tasks -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml">
<classpath path="${jacoco.dir}/jacocoant.jar"/>
</taskdef>

<!-- Target at the level of modules -->
<target name="-do-junit" depends="test-init">
<echo message="Doing testing for jacoco" />
<macrodef name="junit-impl">
<attribute name="test.type"/>
<attribute name="disable.apple.ui" default="false"/>
<sequential>
<jacoco:coverage destfile="${build.test.results.dir}/${code.name.base}_jacoco.exec">
<junit showoutput="true" fork="true" failureproperty="tests.failed" errorproperty="tests.failed" 
filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.results.dir}" timeout="${test.timeout}">
<batchtest todir="${build.test.@{test.type}.results.dir}">
<fileset dir="${build.test.@{test.type}.classes.dir}" includes="${test.includes}" excludes="${test.excludes}"/>
</batchtest>
<classpath refid="test.@{test.type}.run.cp"/>
<syspropertyset refid="test.@{test.type}.properties"/>
<jvmarg value="${test.bootclasspath.prepend.args}"/>
<jvmarg line="${test.run.args}"/>
<!--needed to have tests NOT to steal focus when running, works in latest apple jdk update only.-->
<sysproperty key="apple.awt.UIElement" value="@{disable.apple.ui}"/>
<formatter type="brief" usefile="false"/>
<formatter type="xml"/>
</junit>
</jacoco:coverage>
<copy file="${build.test.results.dir}/${code.name.base}_jacoco.exec" todir="${suite.dir}/build/coverage"/>
<!--
Copy the result of all the unit tests of all the modules into one common
folder at the level of the suite, so that sonar could find those files to
generate associated reports
-->
<copy todir="${suite.dir}/build/test-results">
<fileset dir="${build.test.results.dir}">
<include name="**/TEST*.xml"/>
</fileset>
</copy>
<fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail>
</sequential>
</macrodef>
<junit-impl test.type="${run.test.type}" disable.apple.ui="${disable.apple.ui}"/>
</target>

</project>

Scope of this file is to override the do-junit task adding the jacoco coverage, and to copy the result of the unit test of each module in the build of the suite, so that sonar will find all of them together to perform its analysis.

Create a file in the harness named sonar-jacoco-suite.xml and paste the following code inside

<?xml version="1.0" encoding="UTF-8"?>
<project name="sonar-jacoco-suite" basedir="." xmlns:jacoco="antlib:org.jacoco.ant" xmlns:sonar="antlib:org.sonar.ant">
<description>Builds the module suite otherSuite.</description>

<property name="jacoco.dir" location="${nbplatform.default.harness.dir}/jacoco-0.7.1"/>
<property name="result.exec.file" location="build/coverage"/>    

<!-- Define the SonarQube global properties (the most usual way is to pass these properties via the command line) -->
<property name="sonar.jdbc.url" value="jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8" />
<property name="sonar.jdbc.username" value="sonar" />
<property name="sonar.jdbc.password" value="sonar" />
<!-- Define the SonarQube project properties -->
<property name="sonar.projectKey" value="org.codehaus.sonar:example-java-ant" />
<property name="sonar.projectName" value="Simple Java Project analyzed with the SonarQube Ant Task" />
<property name="sonar.projectVersion" value="1.0" />
<property name="sonar.language" value="java" />
<!-- Load the project properties file for retrieving the modules of the suite -->
<property file="nbproject/project.properties"/>

<!-- Using Javascript functions to build the paths of the data source for sonar configuration -->
<script language="javascript"> 
<![CDATA[

// getting the value
modulesName = project.getProperty("modules");
modulesName = modulesName.replace(":",",");
res = modulesName.split(",");
srcModules = "";
binariesModules = "";
testModules = "";
//Build the paths  
for (var i=0; i<res.length; i++)
{
srcModules += res[i]+"/src,";
binariesModules += res[i]+"/build/classes,";
testModules += res[i]+"/test,";
}
//Remove the last comma
srcModules = srcModules.substring(0, srcModules.length - 1);
binariesModules = binariesModules.substring(0, binariesModules.length - 1);
testModules = testModules.substring(0, testModules.length - 1);
// store the result in a new properties
project.setProperty("srcModulesPath",srcModules);
project.setProperty("binariesModulesPath",binariesModules);
project.setProperty("testModulesPath",testModules);
]]>
</script>  
<!-- Display the values -->       
<property name="sonar.sources" value="${srcModulesPath}"/>
<property name="sonar.binaries" value="${binariesModulesPath}" />
<property name="sonar.tests" value="${testModulesPath}" />
<!-- Define where the coverage reports are located -->
<!-- Tells SonarQube to reuse existing reports for unit tests execution and coverage reports -->
<property name="sonar.dynamicAnalysis" value="reuseReports" />
<!-- Tells SonarQube where the unit tests execution reports are -->
<property name="sonar.junit.reportsPath" value="build/test-results" />
<!-- Tells SonarQube that the code coverage tool by unit tests is JaCoCo -->
<property name="sonar.java.coveragePlugin" value="jacoco" />
<!-- Tells SonarQube where the unit tests code coverage report is -->
<property name="sonar.jacoco.reportPath" value="${result.exec.file}/merged.exec" />
<!--  Step 1: Import JaCoCo Ant tasks  -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml">
<classpath path="${jacoco.dir}/jacocoant.jar"/>
</taskdef>    
<target name="merge-coverage">        
<jacoco:merge destfile="${result.exec.file}/merged.exec">
<fileset dir="${result.exec.file}" includes="*.exec"/>
</jacoco:merge>
</target>

<target name="sonar">
<taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml">
<!-- Update the following line, or put the "sonar-ant-task-*.jar" file in your "$HOME/.ant/lib" folder -->
<classpath path="${harness.dir}/sonar-ant-task-2.1/sonar-ant-task-2.1.jar" />
</taskdef>

<!-- Execute the SonarQube analysis -->
<sonar:sonar />
</target>

</project>

Scope of this file is to define at the level of the suite the sonar configuration and the sonar ant task. If you are using for sonar some special database or special users is here that you must change the configuration.

Another task that is defined is the jacoco merge that will actually take all the generated exec for each module and merge them into one single exec in the build of the suite, to permit sonar to make its analysis.

Replace the content of the build.xml of each module with this one:

<description>Builds, tests, and runs the project com.infrabel.jacoco.</description>
<property file="nbproject/suite.properties"/>
<property file="${suite.dir}/nbproject/private/platform-private.properties"/>
<property file="${user.properties.file}"/>
<import file="${nbplatform.default.harness.dir}/sonar-jacoco-module.xml"/>
<import file="nbproject/build-impl.xml"/>

Replace the content of the build.xml of each suite with this one:

<description>Builds the module suite otherSuite.</description>
<property file="nbproject/private/platform-private.properties"/>
<property file="${user.properties.file}"/>
<import file="${nbplatform.default.harness.dir}/sonar-jacoco-suite.xml"/>
<import file="nbproject/build-impl.xml"/>

Step 2 – Jenkins

In “Manage Jenkins -> Manage Plugins” go in the available list and install (if not already present) the following plugins:

JaCoCo
Mercurial or Subversion
Sonar

If you are behind a firewall or proxy and getting issue to configure the network settings you can always download and install them manually from here. In this case remember to download also the dependencies of each plugin first.

In “Manage Jenkins -> Configure System” check that all plugins are correctly setup, see the following screenshots to have an example (replace the folders with the good ones for you):

Create a new free style project, configure the version control of your preference and in the “Build” panel add the following three “Invoce Ant” tasks:

Finally in the “Post-build Actions” panel add a new “Record Jacoco Coverage Report” configured like this one:

Step 3 – Sonar

Create a database following this script, and optionally run this query to make the connection work:

GRANT ALL PRIVILEGES ON 'sonar'.* TO 'sonar'@'localhost';

Go in the configuration file of sonar (sonar.properties) and enable the use of MySQL, the file is located in the conf folder of the installation

# Permissions to create tables, indices and triggers 
# must be granted to JDBC user.
# The schema must be created first.
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar

#----- MySQL 5.x
# Comment the embedded database and uncomment the following 
# line to use MySQL
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true

In the configuration of sonar update the java plugin if necessary to be compatible with Java 8
If necessary go and configure your proxy always in the sonar.properties file

Done!

Now everything is set-up, you can go in NetBeans, do a build, commit your code, then in Jenkins launch the build and after the build is ok check the project in Sonar.

That’s all! I hope I did not forget anything, but in case if you find some errors during the process do not hesitate to leave a comment, I will try to find the solution.

Reference:

How to do Continuous Integration with Java 8, NetBeans Platform 8, Jenkins, Jacoco and Sonar from our JCG partner Marco Di Stefano at the Refactoring Ideas blog.