Insane - The postmortem memory leak analysis tool

Author: Petr Nejedly, NetBeans
$Revision: 1.1.1.1 $
Document History:available in CVS

Content:


About

When doing performance work, one often faces questions like "how big is this structure", "what happens if I register one more thing here" and so on. When chasing some hard-to-reproduce memory leak, one often needs to come to a long-running full-heap application and analyze its heap without restarting it. There are several approaches to these problems, but none of them seemed optimal:

  • Profiling structures using profiler needs complicated instrumentation and is hard to perform automatically from memory regression tests.
  • Long-term running the application in the instrumented environment significantly reduces its performance and you need to prepare instrumentation in advance.
This lead me to the development of the Insane technology, quite simple solution for reflective inspection of heap from inside of running VM.

Where did the name came from? OK, I consider it quite insane to introspect the whole heap from inside of the application doing the introspection. I consider dumping the whole heap image to a XML file hundred megabytes long even more insane. And finally, how would you call a person trying to parse that XML file?

How does it work

First of all, it has two modi of operation: measurement of a structure size and analysis of the heap. Both modi are based on the same engine. The engine performs a BFS scan of the object graph from given collection of roots, counting visited instances of each type and their size. The size computation is based on the Sun JDK1.3/1.4 storage format and for these JDKs is exact. The engine puts each newly found object into an identity map, assigns it unique ID and adds it to the end of processing queue. Already known objects are ignored. Processing ends when the queue is empty - the algorithm has traversed full transitive closure of given roots. The engine is able to skip some objects, specifically java.lang.Class instances, instances of its own classes and instances from the user filter. The output of the engine is the size distribution collection, containing instance count and total size of each found class.

In the structure size measurement mode, the "roots" collection is usually a singleton collection containing just the reference to the measured structure. It is also possible to specify a filter - objects which to skip.

In the heap analysis mode, the tool does a best effort to find all the real JVM GC roots using introspection on all known classloaders and evaluating the static references. This is also the limitation of the tool - it can't find stack-local and JNI references, but it is only a marginal problem in the event-driven application domain. In this mode, the tool uses a visitor pattern to notify all found objects and all the references between objects. The default implementation of the visitor logs all the information to a simple format XML file.

Usage

Structure size evaluation/testing

The Insane engine was integrated into xtest as several variants of method NbTestCase.assertSize and NbTestCase.assertGC. The usage is quite obvious from the Javadoc, so only a few tips:
  • Don't confuse assertSize(String,int,Object) with assertSize(String,Collection,int). The former is to be used if you want to verify the transitive size of single instance (String, your structure, even some Collection), while the latter is useful for verifying the transitive size of several objects at once, not counting the overhead of the passed collection.
  • If you want to just measure the size of your structure, pass zero limit, it will fail and the failure message will contain the measured size and the distribution among all found object types.
  • The most general variant assertSize(String,Collection,int,Object[]) allows you to pass a filter - objects to skip. If the engine encounters an object from the skip array, it treates the reference pointing to it as null. This is usefull if you need to measure some structure pointing to some shared object, which may in turn reference the rest of the world. As an example, imagine a structure containing a WeakReference. The WeakReference contains a reference to the ReferenceQueue, which contains a linked list of all cleared References.

Post-mortem analysis

This part started as
NetBeans specific, as the tool was invented for improving NetBeans, however it can be generalized. The idea is to add the library to a classpath and create a hook to serve a request for generating heap dump. This hook can be for example a UI widget or a service that can be invoked from outside or running process. It means that it is possible to integrate with server application as well as with desktop application.

The format of the file produced by IDE plugin is self-describing, there are only two types of tags, object and ref. The object tags carry ID, name, size of the instance and, in case the type is [C, the textual representation of the character array. The ref tags are either static references, carrying the fully qualified field name and the ID of the referenced object, or instance references, carying also the ID of the referring object.

For simpler reference graphs, you can find the cause of the leak by hand. You can use your favorite tools on this file, grep seems to be very effective one. Using grep -f, you can even trace references to several instances at once. If you know the IDs of the objects that should be already collected, you can use InsaneParser to find the static references to the objects. Again, InsaneParser is very rough tool, for now you have to edit the code to fill in the path to the dump and IDs of the objects to trace.

An example how to get distribution of char arrays mostly referenced from String objects: egrep '\[C' mdr_dump_09-07.xml | cut -d\' -f8 | sort | uniq -c.

More interesting analysis will often either perform some statistic computation of objects found in the dump or will run various queries. This query processing can include computing set of reachable objects or limited set of these objects like ignoring those that are reachable through weak references. The chain of references starting from static root reference is helpfull to find how the object is held in the application.

IDE plugin

One of the first goals of INSANE was to make it possible to analyze memory leaks in the IDE. To achieve this there is a small add-on module that can be installed into NetBeans and can generate dump of heap content on a user request. Typicaly if you use the IDE for some time and then you realize (e.g. from Memory Meter toolbar) it is using too much memory regardless of your attempts to close everything you can, it's the time for Insane.
  • Install the module if you have not installed it yet.
  • Generate a memory dump.
    • You can do this either by clicking on a button in a memory toolbar (note that memory toolbar that also plots graph of memory usage is hidden by default).
    • or using command-line interface by starting the IDE again with extra parameters --dumpheap <name of generated dump file>.

(It's not much user friendly yet, but you're a programmer, right?). It will run for a few minutes (even native tools spend some time generating the dump, be patient) and finally produce a (large) XML file at the specified location. It will also print a sorted distribution of bytes and instance counts for each type. Just by looking at the distribution, you can guess what is wrong, what is there and shouldn't be. For detailed analysis, check the content of the file.

The module is currently available from NetBeans Update Center Beta for NetBeans 5.0 users and from Development Update Center for users of daily development builds.

Note that the attempt to generate the dump can fails if there is not enough space on Java heap to keep tracking information about dumped data.

References

Insane sources are available in NetBeans CVS repository in the performance CVS module. The performance/insanelib contains the scanner and the model of Java heap together with utilities. These utilities include support for scanning of data structures as well as support for reading of dumps and their quering. Samples of using the Insane are in performance/insanelib/demo. Another use of Insane is a NetBeans module that allows to create a dump of Java heap from a running IDE instance. This module can be found in performance/insane.

If you do not have a CVS account on the NetBeans CVS server, you can do the checkout as anoncvs user:

cvs -d :pserver:anoncvs@cvs.netbeans.org:/cvs co performance/insanelib performance/insane

A presentation was held on JavaONE 2005 about INSANE. You can download the presentation here.

Another way how to generate a heap dump is to use memory dumper that is part of Java SE 6 also known as Mustang using jmap utility. More details about this topic can be found in Trouble-Shooting and Diagnostic Guide and various blog entries.

Project Features

About this Project

Performance was started in November 2009, is owned by tpavek, and has 18 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20131025.e7cbc9d). © 2013, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close