Monitor cpu usage per thread in java...

There is a thread on SO , from which I discovered the TopThreads plugin for JConsole:

http://arnhem.luminis.eu/new_version_topthreads_jconsole_plugin/

jconsole -J-Djava.class.path=C:\Oracle\MIDDLE~1\JDK160~1\lib\jconsole.jar;C:\Oracle\MIDDLE~1\JDK160~1\lib\tools.jar;C:\Oracle\Middleware\wlserver_10.3\server\lib\wlfullclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote -pluginpath "c:\pierre\downloads\topthreads-1.1.jar" -debug

in the jconsole output you should see "Plugin class net.luminis.jmx.topthreads.PluginAdapter loaded."

and here you are: AWESOME!!!

Surely you can script it with ThreadJMX MBean, to create a monitoring tool, but this JConsole UI is so nice...

Also great is this trick: run TOP, then SHIFT-H, take the decimal value of the threads who eat more CPU, convert to HEX and match it to the nid in a jstack threadDump
http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is-consuming-most-cpu/

WebLogic 11 Log Message Format

In Java, I can rename a thread:

String originalName = Thread.currentThread().getName();
Thread.currentThread().setName(originalName + " - My custom name here");



Here the WebLogic 11 Log Format doc

For example this is an error message:

####<Dec 18, 2012 11:55:29 AM CET> <Error> <ALSB Logging> <acme108> <osbpr1ms2> <[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <465bebdd30aa6335:1ab575e0:13b48cc30e9:-8000-00000000007651df> <1355828129705> <BEA-000000> < [null, null, null, ERROR] ****CheckMachineExists_PS - Fault in SEH****: con:fault xmlns:con="http://www.bea.com/wli/sb/context"

this is:

Locale-formatted Timestamp <Dec 18, 2012 11:55:29 AM CET>
Severity <Error>
Subsystem <ALSB Logging>
Machine Name <acme108>
Server Name <osbpr1ms2>
Thread ID <[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'>
User ID <<anonymous>>
Transaction ID <>
Diagnostic Context ID <465bebdd30aa6335:1ab575e0:13b48cc30e9:-8000-00000000007651df>
Raw Time Value <1355828129705>
Message ID <BEA-000000>
Message Text < [null, null, null, ERROR] ****CheckMachineExists_PS - Fault in SEH****: con:fault xmlns:con="http://www.bea.com/wli/sb/context"


Here some explanation on what a Diagnostic Context ID is.

Mysterious new Threads in WebLogic

oracle.dfw.impl.incident


oracle.dfw.incident.FloodController


oracle.dfw.incident.IncidentCreatorThread


for a quick intro to Diagnostic Framework, read here and more in details here



process reaper

for some explanation on this thread, read here

Weblogic and Threads

for those who have missed the event, Weblogic doesn't have any more the concept of configurable Execute Queues for threads.
Instead, we have a single Thread Pool, and the different Work Manager will poach threads from it.

More on this topic:

http://edocs.bea.com/wls/docs100/ConsoleHelp/pagehelp/Corecoreserverservermonitorthreadstitle.html