Hooking into the Jenkins (Hudson) API, Part 2

Kelly RobinsonAugust 10th, 2012Last Updated: October 22nd, 2012

0 86 8 minutes read

This post continues from Part 1 of the tutorial. It’s been almost a year, but I finally had some time to revisit some code I wrote for interacting with the Jenkins api. I’ve used parts of this work to help manage a number of Jenkins build servers, mostly in terms of keeping plugins in sync and moving jobs from one machine to another. For this article I’m going to be primarily focusing on the CLI jar functionality and some of the things you can do with it. This has mostly been developed against Jenkins but I did some light testing with Hudson and it worked there for everything I tried, so the code remains mostly agnostic as to your choice of build server.

The project structure

The code is hosted on Github, and provides a Gradle build which downloads and launches a Jenkins(or Hudson) server locally to execute tests. The server is set to use the Gradle build directory as its working directory, so it can be deleted simply by executing gradle clean. I tried it using both the Jenkins and the Hudson versions of the required libraries and, aside from some quirks between the two CLI implementations, they continue to function very much the same. If you want to try it with Hudson instead of Jenkins, pass in the command flag -Pswitch and the appropriate war and libraries will be used. The project is meant to be run with Gradle 1.0-milestone-8, and comes with a Gradle wrapper for that version. Most of the code remains the same since the original article, but there are some enhancements and changes to deal with the newer versions of Jenkins and Hudson.
The library produced by this project is published as a Maven artifact, and later on I’ll describe exactly how to get at it. There are also some samples included that demonstrate using that library in Gradle or Maven projects, and in Groovy scripts with Grapes. We’re using Groovy 1.8.6, Gradle 1.0-milestone-8 and Maven 3.0.3 to build everything.

Getting more out of the CLI

As an alternative to the api, the CLI jar is a very capable way of interacting with the build server. In addition to a variety of built-in commands, Groovy scripts can be executed remotely, and with a little effort we can easily serialize responses in order to work with data extracted on the server. As an execution environment, the server provides a Groovysh shell and stocks it with imports for the hudson.model package. Also passed into the Binding is the instance of the Jenkins/Hudson singleton object in that package. In these examples I’m using the backwards-compatible Hudson version, since the code is intended to be runnable on either flavor of the server.

The available commands

There’s a rich variety of built-in commands, all of which are implemented in the hudson.cli package. Here are the ones that are listed on the CLI page of the running application:

build: Builds a job, and optionally waits until its completion.
cancel-quiet-down: Cancel the effect of the “quiet-down” command.
clear-queue: Clears the build queue
connect-node: Reconnect to a node
copy-job: Copies a job.
create-job: Creates a new job by reading stdin as a configuration XML file.
delete-builds: Deletes build record(s).
delete-job: Deletes a job
delete-node: Deletes a node
disable-job: Disables a job
disconnect-node: Disconnects from a node
enable-job: Enables a job
get-job: Dumps the job definition XML to stdout
groovy: Executes the specified Groovy script.
groovysh: Runs an interactive groovy shell.
help: Lists all the available commands.
install-plugin: Installs a plugin either from a file, an URL, or from update center.
install-tool: Performs automatic tool installation, and print its location to stdout. Can be only called from
inside a build.
keep-build: Mark the build to keep the build forever.
list-changes: Dumps the changelog for the specified build(s).
login: Saves the current credential to allow future commands to run without explicit credential information.
logout: Deletes the credential stored with the login command.
mail: Reads stdin and sends that out as an e-mail.
offline-node: Stop using a node for performing builds temporarily, until the next “online-node” command.
online-node: Resume using a node for performing builds, to cancel out the earlier “offline-node” command.
quiet-down: Quiet down Jenkins, in preparation for a restart. Don’t start any builds.
reload-configuration: Discard all the loaded data in memory and reload everything from file system. Useful when
you modified config files directly on disk.
restart: Restart Jenkins
safe-restart: Safely restart Jenkins
safe-shutdown: Puts Jenkins into the quiet mode, wait for existing builds to be completed, and then shut down
Jenkins.
set-build-description: Sets the description of a build.
set-build-display-name: Sets the displayName of a build
set-build-result: Sets the result of the current build. Works only if invoked from within a build.
shutdown: Immediately shuts down Jenkins server
update-job: Updates the job definition XML from stdin. The opposite of the get-job command
version: Outputs the current version.
wait-node-offline: Wait for a node to become offline
wait-node-online: Wait for a node to become online
who-am-i: Reports your credential and permissions

It’s not immediately apparent what arguments are required for each, but they almost universally follow a CLI pattern of printing usage details when called with no arguments. For instance, when you call the build command with no arguments, here’s what you get back in the error stream:

Argument “JOB” is required
java -jar jenkins-cli.jar build args…
Starts a build, and optionally waits for a completion.
Aside from general scripting use, this command can be
used to invoke another job from within a build of one job.
With the -s option, this command changes the exit code based on
the outcome of the build (exit code 0 indicates a success.)
With the -c option, a build will only run if there has been
an SCM change
JOB : Name of the job to build
-c : Check for SCM changes before starting the build, and if there’s no
change, exit without doing a build
-p : Specify the build parameters in the key=value format.
-s : Wait until the completion/abortion of the command

Getting data out of the system

All of the interaction with the remote system is handled by streams and it’s pretty easy to craft scripts that will return data in an easily parseable String format using built-in Groovy facilities. In theory, you should be able to marshal more complex objects as well, but let’s keep it simple for now. Here’s a Groovy script that just extracts all of the job names into a List, calling the Groovy inspect method to quote all values.

 
@GrabResolver(name = 'glassfish', root = 'http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name = "github", root = "http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import org.kar.hudson.api.cli.HudsonCliApi

String rootUrl = 'http://localhost:8080'
HudsonCliApi cliApi = new HudsonCliApi()
OutputStream out = new ByteArrayOutputStream()
cliApi.runCliCommand(rootUrl, ['groovysh', 'hudson.jobNames.inspect()'], System.in, out, System.err)
List allJobs = Eval.me(cliApi.parseResponse(out.toString()))
println allJobs

Once we get the response back, we do a little housekeeping to remove some extraneous characters at the beginning of the String, and use Eval.me to transform the String into a List. Groovy provides a variety of ways of turning text into code, so if your usage scenario gets more complicated than this simple case you can use a GroovyShell with a Binding or other alternative to parse the results into something useful. This easy technique extends to Maps and other types as well, making it simple to work with data sent back from the server.

Some useful examples

Finding plugins with updates and and updating all of them

Here’s an example of using a Groovy script to find all of the plugins that have updates available, returning that result to the caller, and then calling the CLI ‘install-plugin’ command on all of them. Conveniently, this command will either install a plugin if it’s not already there or update it to the latest version if already installed.

             
def findPluginsWithUpdates = '''
Hudson.instance.pluginManager.plugins.inject([]) { List toUpdate, plugin ->
    if(plugin.hasUpdate())
    {
        toUpdate << plugin.shortName
    }
    toUpdate
}.inspect()
'''
OutputStream updateablePlugins = new ByteArrayOutputStream()
cliApi.runCliCommand(rootUrl, ['groovysh', findPluginsWithUpdates], System.in, updateablePlugins, System.err)

def listOfPlugins = Eval.me(parseOutput(updateablePlugins.toString()))
listOfPlugins.each{ plugin ->
    cliApi.runCliCommand(rootUrl, ['install-plugin', plugin])
}

Install or upgrade a suite of Plugins all at once

This definitely beats using the ‘Manage Plugins’ UI and is idempotent so running it more than once can only result in possibly upgrading already installed Plugins. This set of plugins might be overkill, but these are some plugins I recently surveyed for possible use.

@GrabResolver(name='glassfish', root='http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name="github", root="http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import static java.net.HttpURLConnection.*
import org.kar.hudson.api.*
import org.kar.hudson.api.cli.HudsonCliApi

String rootUrl = 'http://localhost:8080'
HudsonCliApi cliApi = new HudsonCliApi()

['groovy', 'gradle', 'chucknorris', 'greenballs', 'github', 'analysis-core', 'analysis-collector', 'cobertura',
        'project-stats-plugin','audit-trail', 'view-job-filters', 'disk-usage', 'global-build-stats',
        'radiatorviewplugin', 'violations', 'build-pipeline-plugin', 'monitoring', 'dashboard-view',
        'iphoneview', 'jenkinswalldisplay'].each{ plugin ->
    cliApi.runCliCommand(rootUrl, ['install-plugin', plugin])
}

//  Restart a node, required for newly installed plugins to be made available.
cliApi.runCliCommand(rootUrl, 'safe-restart')

Finding all failed builds and triggering them

It’s not all that uncommon that a network problem or infrastructure event can cause a host of builds to fail all at once. Once the problem is solved this script can be useful for verifying that the builds are all in working order.

@GrabResolver(name = 'glassfish', root = 'http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name = "github", root = "http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import org.kar.hudson.api.cli.HudsonCliApi

String rootUrl = 'http://localhost:8080'
HudsonCliApi cliApi = new HudsonCliApi()
OutputStream out = new ByteArrayOutputStream()
def script = '''hudson.items.findAll{ job ->
    job.isBuildable() && job.lastBuild && job.lastBuild.result == Result.FAILURE
}.collect{it.name}.inspect()
'''
cliApi.runCliCommand(rootUrl, ['groovysh', script], System.in, out, System.err)
List failedJobs = Eval.me(cliApi.parseResponse(out.toString()))
failedJobs.each{ job ->
    cliApi.runCliCommand(rootUrl, ['build', job])
}

Open an interactive Groovy shell

If you really want to poke at the server you can launch an interactive shell to inspect state and execute commands. The System.in stream is bound and responses from the server are immediately echoed back.

@GrabResolver(name = 'glassfish', root = 'http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name = "github", root = "http://kellyrob99.github.com/Jenkins-api-tour/repository")
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')
import org.kar.hudson.api.cli.HudsonCliApi
/**
 * Open an interactive Groovy shell that imports the hudson.model.* classes and exposes
 * a 'hudson' and/or 'jenkins' object in the Binding which is an instance of hudson.model.Hudson
 */
HudsonCliApi cliApi = new HudsonCliApi()
String rootUrl = args ? args[0] :'http://localhost:8080'
cliApi.runCliCommand(rootUrl, 'groovysh')

Updates to the project

A lot has happened in the last year and all of the project dependencies needed an update. In particular, there have been some very nice improvements to Groovy, Gradle and Spock. Most notably, Gradle has come a VERY long way since version 0.9.2. The JSON support added in Groovy 1.8 comes in handy as well. Spock required a small tweak for rendering dynamic content in test reports when using @Unroll, but that’s a small price to pay for features like the ‘old’ method and Chained Stubbing. Essentially, in response to changes in Groovy 1.8+, a Spock @Unroll annotation needs to change from:

@Unroll('querying of #rootUrl should match #xmlResponse')

to a Closure encapsulated GString expression:

@Unroll({'querying of $rootUrl should match $xmlResponse'})

It sounds like the syntax is still in flux and I’m glad I found this discussion of the problem online.

Hosting a Maven repository on Github

Perhaps you noticed from the previous script examples, we’re referencing a published library to get at the HudsonCliApi class. I read an interesting article last week which describes how to use the built-in Github Pages for publishing a Maven repository. While this isn’t nearly as capable as a repository like Nexus or Artifactory, it’s totally sufficient for making some binaries available to most common build tools in a standard fashion. Simply publish the binaries along with associated poms in the standard Maven repo layout and you’re off to the races! Each dependency management system has its quirks(I’m looking at you Ivy!) but they’re pretty easy to work around, so here’s examples for Gradle, Maven and Groovy Grapes to use the library produced by this project code. Note that some of the required dependencies for Jenkins/Hudson aren’t available in the Maven central repository, so we’re getting them from the Glassfish repo.

Gradle

Pretty straight forward, this works with the latest version of Gradle and assumes that you are using the Groovy plugin.

repositories {
    mavenCentral()
    maven {
        url 'http://maven.glassfish.org/content/groups/public/'
    }
    maven {
        url 'http://kellyrob99.github.com/Jenkins-api-tour/repository'
    }
}
dependencies {
    groovy 'org.codehaus.groovy:groovy-all:${versions.groovy}'
    compile 'org.kar:hudson-api:0.2-SNAPSHOT'
}

Maven

Essentially the same content in xml and in this case it’s assumed that you’re using the GMaven plugin

<repositories>
    <repository>
        <id>glassfish</id>
        <name>glassfish</name>
        <url>http://maven.glassfish.org/content/groups/public/</url>
    </repository>
    <repository>
        <id>github</id>
        <name>Jenkins-api-tour maven repo on github</name>
        <url>http://kellyrob99.github.com/Jenkins-api-tour/repository</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-all</artifactId>
        <version>${groovy.version}</version>
    </dependency>
    <dependency>
        <groupId>org.kar</groupId>
        <artifactId>hudson-api</artifactId>
        <version>0.2-SNAPSHOT</version>
    </dependency>
</dependencies>

Grapes

In this case there seems to be a problem resolving some transitive dependency for an older version of Groovy which is why there’s an explicit exclude for it.

@GrabResolver(name='glassfish', root='http://maven.glassfish.org/content/groups/public/')
@GrabResolver(name='github', root='http://kellyrob99.github.com/Jenkins-api-tour/repository')
@Grab('org.kar:hudson-api:0.2-SNAPSHOT')
@GrabExclude('org.codehaus.groovy:groovy')

Links

Related posts:

Reference: Hooking into the Jenkins(Hudson) API, Part 2 from our JCG partner Kelly Robinson at the The Kaptain on … stuff blog.