In Ultra ESB we use a special hot-swap classloader that allows us to reload Java classes on demand. This allows us to literally hot-swap our deployment units – load, unload, reload with updated classes, and phase-out gracefully – without restarting the JVM.
Windows: supporting the forbidden land
In Ultra ESB Legacy the loader was working fine on Windows, but on the newer X-version it seemed to be having some hiccups. We are not supporting Windows as a target platform, so it didn’t matter much – until recently, when we decided to support non-production distros on Windows. (Our enterprise integration IDE UltraStudio runs fine on Windows, so Windows devs, you are all covered.)
TDD FTW
Fixing the classloader was a breeze, and all tests were passing; but I wanted to back my fixes up with some extra tests, so I wrote a few new ones. Most of these involved creating a new JAR file in a subditrectory under the system temp directory, and using the hot-swap classloader to load different artifacts that were placed inside the JAR. For extra credit on best practices, I also made sure to add some cleanup logic to delete the temp subdirectory via FileUtils.deleteDirectory().
And then, things went nuts.
And the tear-down was no more.
All tests were passing, in both Linux and Windows; but the final tear-down logic was failing in Windows, right at the point where I delete the temp subdirectory.
Being on Windows, I didn’t have the luxury of lsof; fortunately, Sysinternals already had just the thing I needed: handle64.
Finding the culprit was pretty easy: hit a breakpoint in tearDown() just before the directory tree deletion call, and run a handle64 {my-jar-name}.jar.
Bummer.
My test Java process, was holding a handle to the test JAR file.
Hunting for the leak
No. Seriously. I didn’t.
Naturally, my first suspect was the classloader itself. I spent almost half an hour going over the classloader codebase again and again. No luck. Everything seemed rock solid.
The “leak dumper”; a.k.a my Grim Reaper for file handles
My best shot was to see what piece of code had opened the handler to the JAR file. So I wrote a quick-n-dirty patch to Java’s FileInputStream and FilterInputStream that would dump acquire-time stacktrace snapshots; whenever a thread holds a stream open for too long.
This “leak dumper” was partly inspired by our JDBC connection pool that detects unreleased connections (subject to a grace period) and then dumps the stacktrace of the thread that borrowed it – back at the time it was borrowed. (Kudos to Sachini, my former colleague-intern at AdroitLogic.)
The leak, exposed!
Sure enough, the stacktrace revealed the culprit:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | id: 174 created: 1570560438355--filter-- java.io.FilterInputStream.<init>(FilterInputStream.java:13) java.util.zip.InflaterInputStream.<init>(InflaterInputStream.java:81) java.util.zip.ZipFile$ZipFileInflaterInputStream.<init>(ZipFile.java:408) java.util.zip.ZipFile.getInputStream(ZipFile.java:389) java.util.jar.JarFile.getInputStream(JarFile.java:447) sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:162) java.net.URL.openStream(URL.java:1045) org.adroitlogic.x.base.util.HotSwapClassLoader.loadSwappableClass(HotSwapClassLoader.java:175) org.adroitlogic.x.base.util.HotSwapClassLoader.loadClass(HotSwapClassLoader.java:110) org.adroitlogic.x.base.util.HotSwapClassLoaderTest.testServiceLoader(HotSwapClassLoaderTest.java:128) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86) org.testng.internal.Invoker.invokeMethod(Invoker.java:643) org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820) org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128) org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) org.testng.TestRunner.privateRun(TestRunner.java:782) org.testng.TestRunner.run(TestRunner.java:632) org.testng.SuiteRunner.runTest(SuiteRunner.java:366) org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) org.testng.SuiteRunner.run(SuiteRunner.java:268) org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) org.testng.TestNG.runSuitesLocally(TestNG.java:1169) org.testng.TestNG.run(TestNG.java:1064) org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72) org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123) |
Gotcha!
1 2 3 4 5 | java.io.FilterInputStream.<init>(FilterInputStream.java:13) ... sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:162) java.net.URL.openStream(URL.java:1045) org.adroitlogic.x.base.util.HotSwapClassLoader.loadSwappableClass(HotSwapClassLoader.java:175) |
But still, that didn’t tell the whole story. If URL.openStream() opens the JAR, why does it not get closed when we return from the try-with-resources block?
01 02 03 04 05 06 07 08 09 10 11 12 | try (InputStream is = jarURI.toURL().openStream()) { byte[] bytes = IOUtils.toByteArray(is); Class<?> clazz = defineClass(className, bytes, 0, bytes.length); ... logger.trace(15, "Loaded class {} as a swappable class", className); return clazz; } catch (IOException e) { logger.warn(16, "Class {} located as a swappable class, but couldn't be loaded due to : {}, " + "trying to load the class as a usual class", className, e.getMessage()); ... } |
Into the wild: JarURLConnection, URLConnection, and beyond
Thanks to Sun Microsystems who made it OSS, I could browse through the JDK source, right up to this shocking comment – all the way down, in java.net.URLConnection:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 | private static boolean defaultUseCaches = true; /** * If <code>true</code>, the protocol is allowed to use caching * whenever it can. If <code>false</code>, the protocol must always * try to get a fresh copy of the object. * <p> * This field is set by the <code>setUseCaches</code> method. Its * value is returned by the <code>getUseCaches</code> method. * <p> * Its default value is the value given in the last invocation of the * <code>setDefaultUseCaches</code> method. * * @see java.net.URLConnection#setUseCaches(boolean) * @see java.net.URLConnection#getUseCaches() * @see java.net.URLConnection#setDefaultUseCaches(boolean) */ protected boolean useCaches = defaultUseCaches; |
Yep, Java does cache JAR streams!
From sun.net.www.protocol.jar.JarURLConnection:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 | class JarURLInputStream extends FilterInputStream { JarURLInputStream(InputStream var2) { super(var2); } public void close() throws IOException { try { super.close(); } finally { if (!JarURLConnection.this.getUseCaches()) { JarURLConnection.this.jarFile.close(); } } } } |
If (well, because) useCaches is true by default, we’re in for a big surprise!
Let Java cache its JARs, but don’t break my test!
JAR caching would probably improve performance; but does that mean I should stop cleaning up after – and leave behind stray files after each test?
(Of course I could say file.deleteOnExit(); but since I was dealing with a directory hierarchy, there was no guarantee that things would get deleted in order, and undeleted directories would be left behind.)
So I wanted a way to clean up the JAR cache – or at least purge just my JAR entry; after I am done, but before the JVM shuts down.
Disabling JAR caching altogether – probably not a good idea!
URLConnection does offer an option to avoid caching connection entries:
01 02 03 04 05 06 07 08 09 10 | /** * Sets the default value of the <code>useCaches</code> field to the * specified value. * * @param defaultusecaches the new value. * @see #getDefaultUseCaches() */ public void setDefaultUseCaches(boolean defaultusecaches) { defaultUseCaches = defaultusecaches; } |
It would have been perfect if caching could be disabled per file/URL, as above; our classloader caches all entries as soon as it opens a JAR, so it never needs to open/read that file again. However, once a JAR is open, caching cannot be disabled on it; so once our classloader has opened the JAR, there’s no getting rid of the cached file handle – until the JVM itself shuts down!
URLConnection also allows you to disable caching by default for all subsequent connections:
01 02 03 04 05 06 07 08 09 10 | /** * Sets the default value of the <code>useCaches</code> field to the * specified value. * * @param defaultusecaches the new value. * @see #getDefaultUseCaches() */ public void setDefaultUseCaches(boolean defaultusecaches) { defaultUseCaches = defaultusecaches; } |
However, if you disable it once, the whole JVM could be affected from that moment onwards – since it probably applies to all URLConnection-based implementations. As I said before, that could hinder performance – not to mention deviating my test from cache-enabled, real-world behavior.
Down the rabbit hole (again!): purging manually from the JarFileFactory
The least-invasive option is to remove my own JAR from the cache, when I know I’m done.
And good news, the cache – sun.net.www.protocol.jar.JarFileFactory – already has a close(JarFile) method that does the job.
But sadly, the cache class is package-private; meaning there’s no way to manipulate it from within my test code.
Reflection to the rescue!
Thanks to reflection, all I needed was one little “bridge” that would access and invoke jarFactory.close(jarFile) on behalf of me:
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | class JarBridge { static void closeJar(URL url) throws Exception { // JarFileFactory jarFactory = JarFileFactory.getInstance(); Class<?> jarFactoryClazz = Class.forName("sun.net.www.protocol.jar.JarFileFactory"); Method getInstance = jarFactoryClazz.getMethod("getInstance"); getInstance.setAccessible(true); Object jarFactory = getInstance.invoke(jarFactoryClazz); // JarFile jarFile = jarFactory.get(url); Method get = jarFactoryClazz.getMethod("get", URL.class); get.setAccessible(true); Object jarFile = get.invoke(jarFactory, url); // jarFactory.close(jarFile); Method close = jarFactoryClazz.getMethod("close", JarFile.class); close.setAccessible(true); //noinspection JavaReflectionInvocation close.invoke(jarFactory, jarFile); // jarFile.close(); ((JarFile) jarFile).close(); }} |
And in my test, I just have to say:
1 | JarBridge.closeJar(jarPath.toUri().toURL()); |
Right before deleting the temp directory.
So, what’s the take-away?
Nothing much for you, if you are not directly dealing with JAR files; but if you are, you might run into this kind of obscure “file in use” errors. (That would hold true for other URLConnection-based streams as well.)
If you happen to be as (un)lucky as I was, just recall that some notorious blogger had written some hacky “leak dumper” patch JAR that would show you exactly where your JAR (or non-JAR) leak is.
Adieu!
Published on Java Code Geeks with permission by Janaka Bandara, partner at our JCG program. See the original article here: JAR File Handles: Clean Up After Your Mess! Opinions expressed by Java Code Geeks contributors are their own. |
That was super helpful! Not as big of an impact in my case, but I was wondering, why some test/temp jar files were not cleaned up. Even deleteOnExit seemed not to work…
Found the caching, searched how to get rid of it and found this. Thanks! :)
Quite a nice article. We’re in a similar situation, our application has a plugin mechanism and we arrived at a similar solution where we need to clear the cache via reflection. The unfortunate part is that recent Java versions do not allow this kind of reflection on internal classes anymore. I think there’s “–add-opens” to allow reflection of internals, but that doesn’t seem like a proper solution and might also stop working in a future Java release.
There is a less hacky solution, at least without using reflection and also working with jdk9+. When accessing an EXISTING resource in the jar using the jar protocol, the returned URLConnection can be cast to JarURLConnection and give you access to the cached JarFile. At this step, you only need to close the JarFile to release the lock.
Path jarPath = <path to locked jar>; // here i assume a jar has a META-INF folder URL jarResourceUrl = new URL("jar", "", jarPath.toUri() + "!/META-INF"); URLConnection connection = jarResourceUrl.openConnection(); if (connection instanceof JarURLConnection juc) { JarFile jf = juc.getJarFile(); jf.close(); }