Insane - The postmortem memory leak analysis tool
Author:
Petr Nejedly, NetBeans
$Revision: 1.11 $
Document History:available in CVS
Content:
About
When doing performance work, one often faces questions like
"how big is this structure", "what happens if I register one more thing here"
and so on. When chasing some hard-to-reproduce memory leak, one often needs
to come to a long-running full-heap application and analyze its heap
without restarting it. There are several approaches to these problems,
but none of them seemed optimal:
- Profiling structures using profiler needs complicated instrumentation
and is hard to perform automatically from memory regression tests.
- Long-term running the application in the instrumented environment
significantly reduces its performance and you need to prepare instrumentation
in advance.
This lead me to the development of the
Insane technology, quite simple
solution for reflective inspection of heap from inside of running VM.
Where did the name came from? OK, I consider it quite insane
to introspect the whole heap from inside of the application doing the
introspection. I consider dumping the whole heap image to a XML file
hundred megabytes long even more insane. And finally, how would you call
a person trying to parse that XML file?
How does it work
First of all, it has two modi of operation: measurement of a structure size
and analysis of the heap. Both modi are based on the same engine.
The engine performs a BFS scan of the object graph from given collection
of roots,
counting visited instances of each type and their size. The size computation
is based on the Sun JDK1.3/1.4 storage format and for these JDKs is exact.
The engine puts each newly found object into an identity map, assigns
it unique ID and adds it to the end of processing queue. Already known objects
are ignored. Processing ends when the queue is empty - the algorithm has
traversed full transitive closure of given roots.
The engine is able to skip some objects, specifically java.lang.Class instances,
instances of its own classes and instances from the user filter.
The output of the engine is the size distribution collection, containing
instance count and total size of each found class.
In the structure size measurement mode, the "roots" collection is usually
a singleton collection containing just the reference to the measured
structure. It is also possible to specify a filter - objects which to skip.
In the heap analysis mode, the tool does a best effort to find
all the real JVM GC roots using introspection on all known classloaders
and evaluating the static references. This is also the limitation
of the tool - it can't find stack-local and JNI references, but it is only
a marginal problem in the event-driven application domain.
In this mode, the tool uses a visitor pattern to notify all found
objects and all the references between objects. The default implementation
of the visitor logs all the information to a simple format XML file.
Usage
Structure size evaluation/testing
The Insane engine was integrated into xtest as several variants of method
NbTestCase.assertSize and NbTestCase.assertGC. The usage is quite obvious from the Javadoc,
so only a few tips:
- Don't confuse
assertSize(String,int,Object)
with assertSize(String,Collection,int). The former is to be used
if you want to verify the transitive size of single instance (String,
your structure, even some Collection), while the latter
is useful for verifying the transitive size of several objects at once,
not counting the overhead of the passed collection.
- If you want to just measure the size of your structure, pass zero limit,
it will fail and the failure message will contain the measured size
and the distribution among all found object types.
- The most general variant
assertSize(String,Collection,int,Object[]) allows you to pass
a filter - objects to skip. If the engine encounters an object from the
skip array, it treates the reference pointing to it as null.
This is usefull if you need to measure some structure pointing
to some shared object, which may in turn reference the rest of the world.
As an example, imagine a structure containing a WeakReference.
The WeakReference contains a reference to the
ReferenceQueue, which contains a linked list of all
cleared References.
Post-mortem analysis
This part started as
NetBeans specific, as the tool was invented for improving
NetBeans, however it can be generalized.
The idea is to add the library to a classpath and create a hook to serve
a request for generating heap dump.
This hook can be for example a UI widget or a service that can be invoked from
outside or running process.
It means that it is possible to integrate with server application as well as with
desktop application.
The format of the file produced by IDE plugin is self-describing,
there are only two types of tags, object and ref.
The object tags carry ID, name, size of the instance and,
in case the type is [C, the textual representation of the
character array. The ref tags are either static references,
carrying the fully qualified field name and the ID of the referenced
object, or instance references, carying also the ID of the referring object.
For simpler reference graphs, you can find the cause of the leak by hand.
You can use your favorite tools on this file, grep seems to be very
effective one. Using grep -f, you can even trace references to several
instances at once. If you know the IDs of the objects that should
be already collected, you can use InsaneParser to find
the static references to the objects. Again, InsaneParser
is very rough tool, for now you have to edit the code to fill in the path
to the dump and IDs of the objects to trace.
An example how to get distribution of char arrays mostly referenced from
String objects:
egrep '\[C' mdr_dump_09-07.xml | cut -d\' -f8 | sort | uniq -c.
More interesting analysis will often either perform some statistic computation
of objects found in the dump or will run various queries.
This query processing can include computing set of reachable objects
or limited set of these objects like ignoring those that are reachable
through weak references.
The chain of references starting from static root reference is helpfull
to find how the object is held in the application.
IDE plugin
One of the first goals of INSANE was to make it possible to analyze memory
leaks in the IDE.
To achieve this there is a small add-on module that can be installed into
NetBeans and can generate dump of heap content on a user request.
Typicaly if you use the IDE for some time and then you realize (e.g. from
Memory Meter toolbar) it is using too much memory regardless of your attempts to close
everything you can, it's the time for Insane.
- Install the module if you have not installed it yet.
- Generate a memory dump.
- You can do this either by clicking on a button in a memory toolbar
(note that memory toolbar that also plots graph of memory usage is hidden by default).
- or using command-line interface by starting the IDE again with extra parameters
--dumpheap <name of generated dump file>.
(It's not much user friendly yet, but you're a programmer,
right?). It will run for a few minutes (even native tools spend some time
generating the dump, be patient) and finally produce a (large) XML file
at the specified location. It will also print a sorted distribution
of bytes and instance counts for each type. Just by looking at the
distribution, you can guess what is wrong, what is there and shouldn't be.
For detailed analysis, check the content of the file.
The module is currently available from NetBeans Update Center Beta for NetBeans 5.0 users
and from Development Update Center for users of daily development builds.
Note that the attempt to generate the dump can fails if there is not enough space
on Java heap to keep tracking information about dumped data.
References
Insane sources are available in NetBeans CVS repository in the performance CVS module.
The performance/insanelib contains the scanner and the model of Java heap
together with utilities.
These utilities include support for scanning of data structures as well
as support for reading of dumps and their quering.
Samples of using the Insane are in performance/insanelib/demo. Another use of Insane is a NetBeans module that allows to create a dump of Java heap
from a running IDE instance. This module can be found in performance/insane.
If you do not have a CVS account on the NetBeans CVS server, you can do the checkout as anoncvs user:
cvs -d :pserver:anoncvs@cvs.netbeans.org:/cvs co performance/insanelib performance/insane
A presentation was held on JavaONE 2005 about INSANE. You can download the presentation here.
Another way how to generate a heap dump is to use memory dumper that is part of
Java SE 6 also known as Mustang using
jmap utility.
More details about this topic can be found in
Trouble-Shooting and Diagnostic Guide and
various blog entries.
