Discover faster, more efficient performance monitoring with an enterprise APM product learning from your apps. Take the AppDynamics APM Guided Tour!
The AppDynamics Java agent, like many Java applications, uses logging extensively. We have used Log4J as our logging framework for many years. And while the latest release of Log4J was in 2012 – and the Apache Foundation announced end-of-life for Log4J in August 2015 – we didn’t upgrade to Log4J2 because of the need to maintain support for the Java 5 VM and other competing priorities. However, our recent move from a monolithic repository to a product-specific repository has made the upgrade possible.
Log4J2 is full of enticing features. For example, the framework dramatically increases the speed of logging and reduces memory usage by providing garbage-free logging. Through native support for async logging, we can further reduce the already small amount of time we spend logging when running on customer applications. Since compression is also a native feature, our agent can tolerate more logging while simultaneously reducing file storage requirements. Both features allow us to add more frequent and better-quality logging that contains actionable information for our customer success and dev teams to help our customers.
Migration Goals and Challenges
So, what did we want to accomplish with the migration? And what were the challenges we faced during the move?
Let’s start with challenges:
– We had to namespace the framework packages to isolate our use of Log4J from our customers’ logging frameworks and we also needed to make the source Java 5 compatible because standard Log4J2 requires Java 1.6 and above.
– Since almost every class uses logging, we had to find a way to make these changes incremental and (relatively) easy to review to maintain the high quality that’s necessary in a production monitoring agent.
– We had to be able to fall back to Log4J (which is proven to work) in case Log4J2 initialization fails.
Our first goal was to repackage the jar with Java 5 compatible source. This step was easy. We programmatically refactored all the classes to namespace their packages. We manually fixed a few issues involving APIs that only Java 6 and above supports, such as String.isEmpty().
The second step was to test the framework in a compatible environment. We used a docker container that has Java 5 installed, and created a test application mirroring our agent structure. This step took time because we needed to figure out how the configuration and customization were going to work with our agent. For example, one of the features that we have is agent error safety. We silence the logs and remove instrumentation if our agent code experiences too many internal errors. Another feature we have is reusing node names. We buffer the logging events and only write them to file after we know the node name from our UI. Using a test application, we were able to mimic all of these features in preparation for the migration.
In order to enable reversibility, we still have both frameworks present at the same time. We used the bridge pattern to extract logging to a separate shared package. This allows us to have multiple logging frameworks present in the code base, and we can easily switch between them at runtime. It also allows us to upgrade logging frameworks in the future, providing high flexibility and changeability. This step was significant because we had to change the build scripts and change every single file that uses a logger.
Lastly, we simply moved the Log4J2 version of our custom appenders we created from the second step, copied the configuration code over, and with that – we successfully upgraded our logging framework!
Log4J2 Log Correlation Support in 4.4
While working with Log4j2, we also took the opportunity to add support for it to our log correlation feature.
Log Correlation enables the user to designate a spot in their log appender pattern for us to insert our business transaction (BT) request guid at runtime. Any call made to the logger in the context of a BT will dynamically inject the guid, regardless of whether the line ultimately ends up in a file or on the console.
The presence of these guids in the log output empowers log processing applications, including our own Log Analytics product, but also others such as Splunk. Using them, we can correlate any lines logged by individual transaction to the snapshot data we collected about that request on the APM side, all without requiring any changes to the customer application. Conversely, it also enables users of our Controller to easily transition from a BT snapshot to the exact lines that occurred during that BT request in their logs.
Logging frameworks supported in addition to the new support for Log4J2 include Log4J, Logback, and Slf4J.
Final Thoughts
Working toward a product-wide upgrade is daunting at first. However, once it’s broken up into small independent steps, it becomes much more manageable. It seems harder to run a 10k than it does to run ten 1ks. The upgrade went smoothly because each step made changes to the product while keeping it functional and ready to ship. This is a boon to faster build verification and code review.
To lean more, see our documentation on business transaction and log correlation. Want to see how AppDynamics Log Analytics works? Start a free trial today.
Discover faster, more efficient performance monitoring with an enterprise APM product learning from your apps. Take the AppDynamics APM Guided Tour!
Haojun Li is a co-author of this blog post. Haojun is a software engineer that has been with AppDynamics for approximately 5 months. He is a fresh grad from UC Berkeley with a degree in Computer Science and Statistics. He enjoys sailing and biking on the road during the weekend.
