The doctor is in: anamneses of a JRA analysis

The question is: once you get a JRA analysis (.JRA), what indicators do you look at to detect problems? What are the "rules of thumb" to detect if your application is in trouble, or it lies within reasonable limits?

Open the JRA with a JRMC (I am using 3.1)

"General" tab:
percentage of CPU and HEAP:
any heap on average above 80% is a clear indicator of low memory available
any CPU on average above 70% indicates a CPU constraint

"Memory" tab
GC statistics: OLD collections should be < 1/5th of the YOUNG Collections
Average time between YC should be > 10 s
All collections total pause time should be < 1/20th of the whole recording time

GC Call trees:
see if any single method triggers more than 5% of all GCs

GC Charts: look for patterns of dense brown (OC), they signal serious memory struggle

in Allocation, look for any single threads allocating large amount of RAM.

Code
Hot Methods: see that no single method takes more than 5% time. If it does, seek that no single stacktrace is responsible for more than 5% of that time.

Threads/Locks
in the Total Threads Count chart, look for times when the number of total threads jumps high, it's a sure signal of high contention
in JVM locks, look for a Lock Name which is much more utilised than others. Typically if the Class Library Cache if very high this means lots of classes dynamically loaded (probably)

Latency
I would recommend to keep always Latency Analysis ON, with an interval of 100ms in order not to overload the system.
Look at the Java Blocked Lock Class list, add the "total latency" column to the table and sort by the most relevant item. Right click, "set operative set", and check "show only Operative Set".
At this point, go to the Log tab, check "show only Operative Set", this will show you all the events blocking on the given lock, inclusive of stacktrace.
The red stripe on top will show you when the latency reaches a top: you can slide the lateral handles to zoom on the area.

The Graph tab is excellent to identify vertical patterns of color: if you see a vertical stripe of blue (Socket Read) or red (Java Blocked) it means that something is blocking a lot of threads on a single operation.

See also http://download.oracle.com/docs/cd/E13188_01/jrockit/docs142/usingJRA/looking.html