Some of the names mentioned above seem not to be established enough while others may require a discussion about their usage patterns. Having this in mind I’ve decided to explore each solution in its own blog post, setting a goal of providing enough information so that people can choose what works best.
This first post explores Maven snapshot and release dependencies. The second post covers artifact and snapshot dependencies provided by TeamCity, and the third and final part will cover artifact and build dependencies provided by the TeamCity Artifactory plugin.
Internal and External Dependencies
Build processes may run in total isolation by checking out the entire code base and building an application from scratch. This is the case for projects where relevant binary dependencies (if there are any) are kept in VCS together with the project sources. However, in many other cases build scripts rely on internal or external dependencies of some sort.
Internal dependencies are satisfied by our own code where we have full control over the project which can be split into multiple modules or sub-projects. External dependencies are satisfied by someone else’s code (on which we have no control) and we consume it or use it as clients. This can be a third-party library such as Spring or a component developed by another team.
This distinction is important since internal and external dependencies are usually accompanied by different release and upgrade cycles: internal dependencies may be modified, rebuilt and updated on an hourly basis while external dependencies’ release cycle is significantly slower with users applying the updates even less frequently, if at all. This is largely driven by the fact that internal dependencies are under our own control and have a narrow-scoped impact, limited by a specific project or module while external dependencies can only be used as-is, their impact is potentially company or world-wide, they are not scoped by any project and can be used anywhere. Naturally, this requires significantly higher standards of release stability, compatibility and maturity, hence slower release and update cycles.
Another aspect of “internal vs. external” dependency characteristics is expressed in how their versions are specified in a build script. Internal dependencies are usually defined using snapshot versions while external dependencies use release versions. The definition of “snapshot” and “release” versions was coined by Maven, which pioneered the idea of managing dependencies by a build tool. If you’re familiar with automatic dependencies management feel free to skip the following section which provides a quick overview of how it works.
Automatic Dependencies Management
In Maven dependencies are specified declaratively in a build script, an approach later followed by newer build tools such as Gradle, Buildr and sbt.
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>1.8.6</version>
<scope>compile</scope>
</dependency>
compile "org.codehaus.groovy:groovy-all:1.8.6"
compile.with "org.apache.axis2:axis2:jar:1.6.1"
libraryDependencies += "org.twitter4j" % "twitter4j-core" % "2.2.5"
Every dependency is identified by its coordinates and scope. Coordinates unambiguously specify the library and version used, while scope defines its visibility and availability in build tasks such as compilation or tests invocation.
For instance, "compile org.codehaus.groovy:groovy-all:1.8.6" would designate a Groovy "org.codehaus.groovy:groovy-all" distribution for version "1.8.6", used for source compilation and test invocation. Switching the scope to “test” or “runtime” would then narrow down the library visibility to tests-only or runtime-only, respectively.
When a build starts, dependencies are either located in a local artifacts repository managed by a build tool (similar to a browser cache) or downloaded from remote repositories, either public or private, such as Maven Central, Artifactory or Nexus. The build tool then adds the artifacts resolved to the corresponding classpaths according to their scopes. When assembling build artifacts, such as "*.war" or "*.ear" archives, all required dependencies are correctly handled and packaged as well.
Though dependencies management seems to be an essential part of almost any build, not all build tools provide a built-in support for it: Ant and MSBuild lack this capability, a gap later addressed by Ivy and NuGet to some extent. However, Ivy’s adoption was slower compared to Maven, while NuGet is a .NET-only tool. Over time, Maven artifact repositories and Maven Central have become a de facto mechanism for distributing and sharing Java artifacts. Being able to resolve and deploy these using Maven repositories has become a “must have” ability for all newer Java build tools.
Release and Snapshot Dependencies
As I mentioned previously, internal dependencies are normally defined using snapshot versions while external dependencies use release versions. Let’s look into release versions first as they are easier to reason about.
Release dependencies are those which have a fixed version number, such as the "1.8.6" version of the Groovy distribution. Whatever artifact repository is used by the build and whenever it attempts to locate this dependency, it is always expected to resolve the exact same artifact. This is the main principle of release dependencies: “Same version = same artifact”. Due to this fact, build tools do not check for a release dependency update once it is found and will only re-download the artifact if the local cache was emptied. And this all makes sense, of course, since we never expect to find divergent artifacts of the same library carrying an identical version number!
Snapshot dependencies are different and, as a result, way trickier to deal with. Snapshot dependency versions end with a special "-SNAPSHOT" keyword, like "3.2.0-SNAPSHOT". This keyword signals the build tools to periodically check an artifact with a remote repository for updates; by default, Maven performs this check on a daily basis. The function of snapshot dependencies, then, is to depend on someone else’s work-in-progress (think “nightly builds”): when product development moves from version "X" to version "X+1" its modules are versioned "X+1-SNAPSHOT".
Snapshot Dependencies Uncertainty
If the main principle of release dependencies was “Same version = same artifact” (after version ‘X’ of library is released, its artifacts are identical all around the world, forever), snapshot dependencies’ principle is “Same version = ever-updating artifact”. The benefit of this approach is that it enables retrieving frequent updates without the need to produce daily releases which would be highly impractical. The downside of it, however, is uncertainty – using snapshot dependencies in a build script makes it harder to know which version was used in a specific build execution. My "maven-about-plugin" stores a textual “about” file in every snapshot artifact in order to better identify its origins such as VCS revision and build number; this can be helpful but it only solves half of the problem.
Being a moving target by their definition, snapshot dependencies do not allow us to pin down versions on which we depend and therefore build reproducibility becomes harder to achieve. Also, in a series or pipeline of builds (when a finished build triggers an invocation of subsequent ones), an artifact produced by initial pipeline steps is not necessarily consumed by the closing ones as it may be long overridden since then by other build processes running at the same time.
One possible approach in this situation is to lock-down a dependency version in a build script using timestamp so it becomes "3.2.0-20120119.134529-1" rather than "3.2.0-SNAPSHOT". This effectively makes snapshot dependencies identical to release dependencies and disables an automatic update mechanism, making it impossible to use an up-to-date version even when one is available unless the timestamp is updated.
As you see, snapshot dependencies can be used where it makes sense but it should be done with caution and in small doses. If possible, it is best to manage a separate release lifecycle for every reusable component and let its clients use periodically updated release dependencies.
Summary
This article has provided an overview of automatic dependencies management by Java build tools together with an introduction to Maven release and snapshot dependencies. It also explained how snapshot dependencies’ advantages become debatable in the context of build reproducibility and build pipelines.
The following blog posts will explore the TeamCity build chains and Artifactory build isolation which allow to use consistent, reproducible and up-to-date snapshot versions throughout a chain of builds without locking down their timestamps in a build script. More to come!
Reference: Maven Build Dependencies from our JCG partner Evgeny Goldin at the Goldin++ blog.
