Insane - The postmortem memory leak analysis toolAuthor: Petr Nejedly, NetBeans
$Revision: 220.127.116.11 $
Document History:available in CVS
- How does it work
- IDE plugin
When doing performance work, one often faces questions like "how big is this structure", "what happens if I register one more thing here" and so on. When chasing some hard-to-reproduce memory leak, one often needs to come to a long-running full-heap application and analyze its heap without restarting it. There are several approaches to these problems, but none of them seemed optimal:
- Profiling structures using profiler needs complicated instrumentation and is hard to perform automatically from memory regression tests.
- Long-term running the application in the instrumented environment significantly reduces its performance and you need to prepare instrumentation in advance.
Where did the name came from? OK, I consider it quite insane to introspect the whole heap from inside of the application doing the introspection. I consider dumping the whole heap image to a XML file hundred megabytes long even more insane. And finally, how would you call a person trying to parse that XML file?
How does it workFirst of all, it has two modi of operation: measurement of a structure size and analysis of the heap. Both modi are based on the same engine. The engine performs a BFS scan of the object graph from given collection of roots, counting visited instances of each type and their size. The size computation is based on the Sun JDK1.3/1.4 storage format and for these JDKs is exact. The engine puts each newly found object into an identity map, assigns it unique ID and adds it to the end of processing queue. Already known objects are ignored. Processing ends when the queue is empty - the algorithm has traversed full transitive closure of given roots. The engine is able to skip some objects, specifically java.lang.Class instances, instances of its own classes and instances from the user filter. The output of the engine is the size distribution collection, containing instance count and total size of each found class.
In the structure size measurement mode, the "roots" collection is usually a singleton collection containing just the reference to the measured structure. It is also possible to specify a filter - objects which to skip.
In the heap analysis mode, the tool does a best effort to find all the real JVM GC roots using introspection on all known classloaders and evaluating the static references. This is also the limitation of the tool - it can't find stack-local and JNI references, but it is only a marginal problem in the event-driven application domain. In this mode, the tool uses a visitor pattern to notify all found objects and all the references between objects. The default implementation of the visitor logs all the information to a simple format XML file.
Structure size evaluation/testingThe Insane engine was integrated into xtest as several variants of method
NbTestCase.assertGC. The usage is quite obvious from the Javadoc, so only a few tips:
- Don't confuse
assertSize(String,Collection,int). The former is to be used if you want to verify the transitive size of single instance (
String, your structure, even some
Collection), while the latter is useful for verifying the transitive size of several objects at once, not counting the overhead of the passed collection.
- If you want to just measure the size of your structure, pass zero limit, it will fail and the failure message will contain the measured size and the distribution among all found object types.
- The most general variant
assertSize(String,Collection,int,Object)allows you to pass a filter - objects to skip. If the engine encounters an object from the skip array, it treates the reference pointing to it as
null. This is usefull if you need to measure some structure pointing to some shared object, which may in turn reference the rest of the world. As an example, imagine a structure containing a
WeakReferencecontains a reference to the
ReferenceQueue, which contains a linked list of all cleared
Post-mortem analysisThis part started as NetBeans specific, as the tool was invented for improving NetBeans, however it can be generalized. The idea is to add the library to a classpath and create a hook to serve a request for generating heap dump. This hook can be for example a UI widget or a service that can be invoked from outside or running process. It means that it is possible to integrate with server application as well as with desktop application.
The format of the file produced by IDE plugin is self-describing,
there are only two types of tags,
object tags carry ID, name, size of the instance and,
in case the type is
[C, the textual representation of the
character array. The
ref tags are either static references,
carrying the fully qualified field name and the ID of the referenced
object, or instance references, carying also the ID of the referring object.
For simpler reference graphs, you can find the cause of the leak by hand.
You can use your favorite tools on this file,
grep seems to be very
effective one. Using
grep -f, you can even trace references to several
instances at once. If you know the IDs of the objects that should
be already collected, you can use
InsaneParser to find
the static references to the objects. Again,
is very rough tool, for now you have to edit the code to fill in the path
to the dump and IDs of the objects to trace.
An example how to get distribution of char arrays mostly referenced from
egrep '\[C' mdr_dump_09-07.xml | cut -d\' -f8 | sort | uniq -c.
More interesting analysis will often either perform some statistic computation of objects found in the dump or will run various queries. This query processing can include computing set of reachable objects or limited set of these objects like ignoring those that are reachable through weak references. The chain of references starting from static root reference is helpfull to find how the object is held in the application.
IDE pluginOne of the first goals of INSANE was to make it possible to analyze memory leaks in the IDE. To achieve this there is a small add-on module that can be installed into NetBeans and can generate dump of heap content on a user request. Typicaly if you use the IDE for some time and then you realize (e.g. from Memory Meter toolbar) it is using too much memory regardless of your attempts to close everything you can, it's the time for Insane.
- Install the module if you have not installed it yet.
- Generate a memory dump.
- You can do this either by clicking on a button in a memory toolbar (note that memory toolbar that also plots graph of memory usage is hidden by default).
- or using command-line interface by starting the IDE again with extra parameters
--dumpheap<name of generated dump file>.
(It's not much user friendly yet, but you're a programmer, right?). It will run for a few minutes (even native tools spend some time generating the dump, be patient) and finally produce a (large) XML file at the specified location. It will also print a sorted distribution of bytes and instance counts for each type. Just by looking at the distribution, you can guess what is wrong, what is there and shouldn't be. For detailed analysis, check the content of the file.
The module is currently available from NetBeans Update Center Beta for NetBeans 5.0 users and from Development Update Center for users of daily development builds.
Note that the attempt to generate the dump can fails if there is not enough space on Java heap to keep tracking information about dumped data.
Insane sources are available in NetBeans CVS repository in the performance CVS module.
performance/insanelib contains the scanner and the model of Java heap
together with utilities.
These utilities include support for scanning of data structures as well
as support for reading of dumps and their quering.
Samples of using the Insane are in
performance/insanelib/demo. Another use of Insane is a NetBeans module that allows to create a dump of Java heap
from a running IDE instance. This module can be found in
If you do not have a CVS account on the NetBeans CVS server, you can do the checkout as anoncvs user:
cvs -d :pserver:firstname.lastname@example.org:/cvs co performance/insanelib performance/insane
A presentation was held on JavaONE 2005 about INSANE. You can download the presentation here.
Another way how to generate a heap dump is to use memory dumper that is part of
Java SE 6 also known as Mustang using
More details about this topic can be found in
Trouble-Shooting and Diagnostic Guide and
various blog entries.