Join the war on

Classloader leaks

Mattias Jiderhamn

Mattias Jiderhamn – java.jiderhamn.se

 :-(

java.lang.OutOfMemoryError:
Metaspace
PermGen space

Mattias Jiderhamn – java.jiderhamn.se

Local dev env

Mattias Jiderhamn – java.jiderhamn.se

Continuous Deploy

.war

Mattias Jiderhamn – java.jiderhamn.se

 Agenda
What does it mean?
Why does it happen?
Where does it leak?
How to avoid?
OSS examples
Training!
Mattias Jiderhamn – java.jiderhamn.se
What?
Mattias Jiderhamn – java.jiderhamn.se
 JVM memory
PermGen /
Stack
Metaspace

Heap

Mattias Jiderhamn – java.jiderhamn.se

 JVM memory
Per thread
Stack Local variables and
method parameters

Mattias Jiderhamn – java.jiderhamn.se

 JVM memory
Object instances

Heap
Young generation
Eden Survivor Old generation /
space space tenured space

Mattias Jiderhamn – java.jiderhamn.se

 JVM memory
Permanent Generation
java.lang.Class instances PermGen /
Metaspace
etc
Named before JEE and
class unloading
Renamed Metaspace in Java 8
Aka Method area
Mattias Jiderhamn – java.jiderhamn.se
 What?
java.lang.OutOfMemoryError:
PermGen space / Metaspace

= Too many classes are loaded

Mattias Jiderhamn – java.jiderhamn.se

 Why?
Mattias Jiderhamn – java.jiderhamn.se
 Reason for OOME
java.lang.OutOfMemoryError:
PermGen space / Metaspace
1. Your application is too large
-XX:MaxPermSize=256M
Java 8: Metaspace auto increase by default
2. java.lang.Class instances could not
be garbage collected after redeploy

Mattias Jiderhamn – java.jiderhamn.se

 Reference types
• Strong (i.e. normal) reference
– Never GC:ed if reachable
• Soft reference
– GC:ed before OutOfMemoryError
• Weak reference (WeakHashMap)
– GC:ed when no Strong or Soft refs
• Phantom reference
– … won’t prevent GC
Mattias Jiderhamn – java.jiderhamn.se
 Example
WeakHashMap

Map m = new WeakHashMap();

Foo myFoo = new Foo();
m.put(myFoo, ”bar”);
myFoo = null;

Foo

Mattias Jiderhamn – java.jiderhamn.se

GC reachability

GC roots
Mattias Jiderhamn – java.jiderhamn.se
 Redeploy
Application Server

ClassLoader ClassLoader ClassLoader

app.war app.war’ app.war’’

Mattias Jiderhamn – java.jiderhamn.se

 ClassLoader refs
ClassLoader

Class Class Class

Instance

Mattias Jiderhamn – java.jiderhamn.se

 How leaks happen
Application Server

ClassLoader

Class Class Class

Instance
GC root

Mattias Jiderhamn – java.jiderhamn.se

Where?
Mattias Jiderhamn – java.jiderhamn.se
 Application Server
• Application Server bugs
?
• Logging frameworks
– Apache Commons Logging
Unless LogFactory.release()
– Log4j - some configs
Unless LogManager.shutdown()
– java.util.logging custom Level
• Bean Validation API (JSR 303)
• Unified Expression Language (javax.el)

Mattias Jiderhamn – java.jiderhamn.se

 GC roots
• Class loaded by system ClassLoader
– static field in JDK classes (java.* etc)
• Live thread
– Stack – local vars, method params
– java.lang.Thread instance
• Object held as synchronization monitor
• JNI references
• JVM specials…
Mattias Jiderhamn – java.jiderhamn.se
 System classes
• java.sql.DriverManager
• Bean introspection cache, shutdown hooks,
custom default Authenticator, custom
security Provider, custom MBeans, custom
ThreadGroup, custom property editor, …
• Reference to contextClassLoader of first caller

Mattias Jiderhamn – java.jiderhamn.se

DriverManager

Mattias Jiderhamn – java.jiderhamn.se

 DriverManager
JRE
Application Server
DriverManager
1) …

ClassLoader

com.mysql.jdbc.Driver

app.war

mysql-jdbc.jar
… or 2)
DriverManager.deregisterDriver(…)
Mattias Jiderhamn – java.jiderhamn.se
 Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {

...

/** Called when application is undeployed */

public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
}
}

Mattias Jiderhamn – java.jiderhamn.se

 Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer

Mattias Jiderhamn – java.jiderhamn.se

 Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {

...

/** Called when application is undeployed */

public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
// Stop threads here!
}
}

Mattias Jiderhamn – java.jiderhamn.se

 Stopping threads
public class MyThread extends Thread
{

  public void run() {
  while(true) { // Bad idea!
    // Do something 
   }
  }

}
Mattias Jiderhamn – java.jiderhamn.se
 Stopping threads
public class MyThread extends Thread
{
 private volatile boolean running = true;
  private boolean running = true;

  public void run() {
  while(running) { // Until stopped
    // Do something 
   }
  }

  public void shutdown() {
    running = false;
  } Kabutz / Java Specialists’ - The Law of the Blind Spot
Heinz
} Mattias Jiderhamn – java.jiderhamn.se
 Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer
• ThreadLocal
Mattias Jiderhamn – java.jiderhamn.se
 ThreadLocal
ThreadLocal

WeakHashMap<Thread, ?>

Thread Foo

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
ThreadLocal JavaDoc:
”Each thread holds an implicit
reference to its copy of a thread-
local variable as long as the thread
is alive and the ThreadLocal
instance is accessible; …”

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
Thread

put() ThreadLocalMap
Entry

ThreadLocal Foo

set()
Mattias Jiderhamn – java.jiderhamn.se
 ThreadLocal (?)
Pooled threads:
• Threads may outlive ClassLoader
• ThreadLocal → ThreadGlobal!

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
Thread

ThreadLocalMap
Entry

ThreadLocal Foo
static

Class ClassLoader Class

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
Thread

ThreadLocalMap
Entry
Stale entry

ThreadLocal Foo

ClassLoader Class

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
ThreadLocalMap JavaDoc:
”However, since reference queues
are not used, stale entries are
guaranteed to be removed only
when the table starts running out
of space”

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
• Custom value (incl references)
–static ThreadLocal = leak
–otherwise = unpredicted GC
• Custom ThreadLocal = no leak

Mattias Jiderhamn – java.jiderhamn.se

 ThreadLocal
Always clear your ThreadLocals!
try {
myThreadLocal.set(foo);
…
}
finally {
myThreadLocal.remove();
}
Mattias Jiderhamn – java.jiderhamn.se
 Known offenders
Apache ActiveMQ JarURLConnection
Apache Axis Java Advanced Imaging (JAI)
Apache Batik Javassist
Apache Commons Pool / DBCP Java Cryptography Architecture (JCA)
Apache CXF Java Server Faces 2
AWT Toolkit javax.security.auth.Policy/login.Configuration
Bean Validation API / JSR 303 JGroups
CGLIB (Hibernate / Spring / JBoss / Logback
Apache Geronimo) JAXB
dom4j Mozilla Rhino
EclipseLink MVEL
GeoTools OpenOffice Java Uno RunTime (JURT)
Google Guice Oracle JDBC
Groovy RMI
GWT / javax.imageio Serialization (<= JDK 1.4.2)
Hessian Spring framework
iCal4J Unified Expression Language / javax.el
Infinispan URLConnection + HTTP 1.1
IntrospectionUtils
XML parsing (DocumentBuilderFactory)
Mattias Jiderhamn – java.jiderhamn.se
 How?
Mattias Jiderhamn – java.jiderhamn.se
 Leak Prevention Lib
• Application server independent
– Version 2 core module is pure Java SE!
• Covers a lot more than Tomcat
– System class references avoided/cleared
– Threads are stopped
– ThreadLocals are cleared
– Known offenders handled
• Logs warnings
• Apache 2 licensed
– You may modify and redistribute
Mattias Jiderhamn – java.jiderhamn.se
 Leak Prevention Lib
Zero runtime dependencies
No required config
Maven pom.xml (Servlet 3.0+):
<dependency>
<groupId>se.jiderhamn.classloader-leak-prevention</groupId>
<artifactId>classloader-leak-prevention-servlet3</artifactId>
<version>2.0.1</version>
</dependency>

Mattias Jiderhamn – java.jiderhamn.se

 Tomcat
Cleanup
Thread Thread
remove() get()
ThreadLocalMap set()
Unsafe! remove()

Removed in 6.0.27
Bugzilla #48895

Tomcat 7.0.6+ renews the thread pool

whenever an application is redeployed.
Disposable threads for lifecycle events.
Bugzilla #49159

Mattias Jiderhamn – java.jiderhamn.se

 Leak Prevention Lib
Cleanup Thread
Thread
set(null) ThreadLocalMap
Entry
Stale

ThreadLocal Foo

Mattias Jiderhamn – java.jiderhamn.se

Open Source
examples
Mattias Jiderhamn – java.jiderhamn.se
 Bean Validation API
Version 1.0.0.GA
javax.validation.Validation

Application Server
validation-api-1.0.0.GA.jar

app.war
hibernate-validator.jar

Mattias Jiderhamn – java.jiderhamn.se

 javax.el API
javax.el.BeanELResolver

Mattias Jiderhamn – java.jiderhamn.se

 Apache CXF
org.apache.cxf.transport.http.CXFAuthenticator

CXF-5442

Mattias Jiderhamn – java.jiderhamn.se

 OpenOffice JURT
com.sun.star.lib.util.AsynchronousFinalizer

Mattias Jiderhamn – java.jiderhamn.se

Training!
Gear up!

Mattias Jiderhamn – java.jiderhamn.se

 Free tool
Eclipse Memory Analyzer
aka MAT
eclipse.org/mat

Mattias Jiderhamn – java.jiderhamn.se

 Heap dump
Run application server with

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/temp

Mattias Jiderhamn – java.jiderhamn.se

 Links
java.jiderhamn.se @mjiderhamn
github.com/mjiderhamn
• ClassLoader Leak Prevention Library
• Step by step guide: acquire heap
dump and analyze for leaks using
Eclipse Memory Analyzer (MAT)
• List of known offenders and links to bug reports
• Submit reports and pull requests!

???
Join the fight!
Mattias Jiderhamn – java.jiderhamn.se