WebLogic Server Request Performance View

On how to configure the deployment to add a "Context ID" using the wldf-resource/wldf-instrumentation-monitor tag: http://docs.oracle.com/cd/E24329_01/web.1211/e24426/config_context.htm#WLDFC253

Thread Dump on Stuck thread with Diagnostic Image

See also https://blogs.oracle.com/jamesbayer/entry/weblogic_notifications_watch_t
See also http://docs.oracle.com/cd/E21764_01/web.1111/e13714/config_diag_images.htm

connect('weblogic', 'weblogic1', 't3://myserver.acme.com:7001')

edit()
startEdit()
#untarget existing default module
cd('/SystemResources/Module-FMWDFW')
set('Targets',jarray.array([], ObjectName))


theMailSessionName='MailSessionAdmin'
cd('/')
myMbean = getMBean('/MailSessions/' + theMailSessionName)
if (myMbean == None):
    print "createMailSession", theMailSessionName
    cmo.createMailSession(theMailSessionName)
    cd('/MailSessions/MailSessionAdmin')
    cmo.setJNDIName('MailSessionAdmin')
    prop = Properties()
    prop.setProperty('mail.smtp.host', 'smtp.acme.com')
    cmo.setProperties(prop)
    set('Targets',jarray.array([ObjectName('com.bea:Name=osbpl1as,Type=Server'), ObjectName('com.bea:Name=osbpl1cl,Type=Cluster')], ObjectName))
else:
    print "mailSession", theMailSessionName, "already exists"
    
#create custom module
cd('/')
cmo.createWLDFSystemResource('stuckthreadsmodule')

cd('/SystemResources/stuckthreadsmodule')
cmo.setDescription('create a diagnostic image whenever the server is in warning')

#create image notification
cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule')
cmo.createImageNotification('stuckthreaddiagnosticmodule')
cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule/ImageNotifications/stuckthreaddiagnosticmodule')
cmo.setEnabled(true)
cmo.setImageDirectory('logs/diagnostic_images')
cmo.setImageLockout(10)

#create email notification
cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule')
cmo.createSMTPNotification('stuckthreadnotificationemail')

cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule/SMTPNotifications/stuckthreadnotificationemail')
cmo.setEnabled(true)
cmo.setMailSessionJNDIName('MailSessionAdmin')
set('Recipients',jarray.array([String('pierluigi.vernetto@acme.com')], String))
cmo.setSubject(None)
cmo.setBody(None)

#create a watch
cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule')
cmo.createWatch('stuckthreadwatch')

#configure watch rule
cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule/Watches/stuckthreadwatch')
cmo.setRuleType('Log')
cmo.setEnabled(true)
cmo.setRuleExpression('(SEVERITY = \'Error\') AND ((MSGID = \'WL-000337\') OR (MSGID = \'BEA-000337\'))')
cmo.setAlarmType('AutomaticReset')
cmo.setAlarmResetPeriod(60000)

#assign notifications to watch
cd('/WLDFSystemResources/stuckthreadsmodule/WLDFResource/stuckthreadsmodule/WatchNotification/stuckthreadsmodule/Watches/stuckthreadwatch')
set('Notifications',jarray.array([ObjectName('com.bea:Name=stuckthreaddiagnosticmodule,Type=weblogic.diagnostics.descriptor.WLDFImageNotificationBean,Parent=[osbpl1do]/WLDFSystemResources[stuckthreadsmodule],Path=WLDFResource[stuckthreadsmodule]/WatchNotification[stuckthreadsmodule]/ImageNotifications[stuckthreaddiagnosticmodule]'), ObjectName('com.bea:Name=stuckthreadnotificationemail,Type=weblogic.diagnostics.descriptor.WLDFSMTPNotificationBean,Parent=[osbpl1do]/WLDFSystemResources[stuckthreadsmodule],Path=WLDFResource[stuckthreadsmodule]/WatchNotification[stuckthreadsmodule]/SMTPNotifications[stuckthreadnotificationemail]')], ObjectName))

cd('/SystemResources/stuckthreadsmodule')
set('Targets',jarray.array([ObjectName('com.bea:Name=osbpl1cl,Type=Cluster')], ObjectName))

validate()
save()
activate()

This will create:- a mail session "MailSessionAdmin"
- a diagnostic module "stuckthreadsmodule"
- inside it, a watch stuckthreadwatch
- the watch will use 2 notifications, stuckthreaddiagnosticmodule and stuckthreadnotificationemail




How to test:

mkdir -p /opt/oracle/apps/wlstuck/WEB-INF

make a small index.jsp

<% Thread.sleep(660000); %> and a web.xml

<web-app xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<display-name>wlstuck</display-name>
    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>
</web-app>

"Health Check Interval" (Configuration/Health Monitoring) defines how often the server checks its health.

"Stuck Thread Max Time" (Configuration/Tuning) defines how many seconds to wait before flagging the thread as stuck.

You sohuld get an email when the stuck thread makes the server go in WARNING state:

WatchTime: Apr 24, 2013 6:29:13 PM CEST 
WatchDomainName: osbpl1do 
WatchServerName: osbpl1ms1 
WatchSeverityLevel: Notice 
WatchName: stuckthreadwatch 
WatchRuleType: Harvester 
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: com.bea:Name=osbpl1ms1,Type=ServerRuntime//State = RUNNING  
WatchAlarmType: AutomaticReset 
WatchAlarmResetPeriod: 60000 
SMTPNotificationName: stuckthreadnotificationemail 

WebLogic WLDF incidents notified by email

Open WebLogic console

Services / Mail Sessions / new Mail Sessions

MailSessionAdmin : JavaMail Properties: mail.smtp.host=smtp.acme.com

here the full list of the JavaMail Properties


Diagnostic Modules - Module-FMWDFW - Watches and Notifications - Notifications - FMWDFWNotification; Type = SMTP (E-Mail)

SMTP Properties ; Mail Session Name = MailSessionAdmin

E-Mail Recipients = pvernetto@gmail.com

To test, prepare a exceptionGenerator.jsp containing:

<% throw new NullPointerException("hello"); %>
Deploy the JSP in a TestWebApp war to the WLS,
http://myserver.acme.com:7001/TestWebApp/exceptionGenerator.jsp
you should get this:

 at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
 at
 weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

 at
 weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

 at
 weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)

 at
 weblogic.servlet.internal.ServletStubImpl.onAddToMapException(ServletStubImpl.java:416)

 at
 weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:326)

 at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
 at
 weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

 at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)

 at java.security.AccessController.doPrivileged(Native Method)
 at
 oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
 at
 oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)

 at
 oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)

 at
 oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)

 at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
 at
 weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

 at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)

 at
 weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)

 at
 weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)

 at
 weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)

 at
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

 at
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

 at
 weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)

 at
 weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)

 at
 weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)

 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 at
 weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
 SUBSYSTEM = HTTP
 USERID =  SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '3'
 for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-101020  MACHINE = myserver TXID =  CONTEXTID =
 11d1def534ea1be0:65fd0c39:13b9bd510e7:-8000-00000000000001f7 TIMESTAMP =
 1355556858627
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod:
 30000
SMTPNotificationName: FMWDFWNotification
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

WatchTime: Dec 15, 2012 8:34:18 AM CET
WatchDomainName: osbpl1do
WatchServerName: osbpl1as
WatchSeverityLevel: Notice
WatchName: UncheckedException
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
WatchData: DATE = Dec 15, 2012 8:34:18 AM CET SERVER = osbpl1as MESSAGE = [ServletContext@453292093[app:TestWebApp module:TestWebApp.war path:/TestWebApp spec-version:2.5]] Servlet failed with Exception
java.lang.NullPointerException: hello
 at jsp_servlet.__exceptiongenerator._jspService(__exceptiongenerator.java:71)
 at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
 at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
 at weblogic.servlet.internal.ServletStubImpl.onAddToMapException(ServletStubImpl.java:416)
 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:326)
 at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
 at java.security.AccessController.doPrivileged(Native Method)
 at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
 at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
 at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
 at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
 at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
 at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
 at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
 SUBSYSTEM = HTTP USERID =  SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-101020 MACHINE = myserver TXID =  CONTEXTID = 11d1def534ea1be0:65fd0c39:13b9bd510e7:-8000-00000000000001f7 TIMESTAMP = 1355556858627
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000
SMTPNotificationName: FMWDFWNotification 

setting a WLDF watch to monitor a cluster

Here the official Oracle documentation on WLDF


Here my previous post on WLDF.



Here the WLST to create a WLDF Diagnostic Module with a watch on the number of members in the cluster:

domain=osb_domain
managed server=ms1
cluster=myCluster

It will generate a SNMP trap every time the number of AliveServerCount on ms1 is less than 3:

edit()
startEdit()
cd('/')
cmo.createWLDFSystemResource('ClusterMonitor')

cd('/SystemResources/ClusterMonitor')
cmo.setDescription('')

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/Harvester/ClusterMonitor')
cmo.createHarvestedType('weblogic.management.runtime.ClusterRuntimeMBean')

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/Harvester/ClusterMonitor/HarvestedTypes/weblogic.management.runtime.ClusterRuntimeMBean')
set('HarvestedAttributes',jarray.array([String('AliveServerCount'), String('CurrentSecondaryServer'), String('ForeignFragmentsDroppedCount'), String('FragmentsReceivedCount'), String('FragmentsSentCount'), String('MulticastMessagesLostCount'), String('PrimaryCount'), String('ResendRequestsCount'), String('SecondaryCount'), String('SecondaryServerDetails')], String))
set('HarvestedInstances',jarray.array([String('com.bea:Name=myCluster,ServerRuntime=ms1,Type=ClusterRuntime')], String))
cmo.setNamespace('ServerRuntime')

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/WatchNotification/ClusterMonitor')
cmo.createWatch('clusterGone')

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/WatchNotification/ClusterMonitor/Watches/clusterGone')
cmo.setRuleType('Harvester')
cmo.setEnabled(true)
cmo.setRuleExpression('(${ServerRuntime//[weblogic.management.runtime.ClusterRuntimeMBean]com.bea:Name=myCluster,ServerRuntime=ms1,Type=ClusterRuntime//AliveServerCount} < 3)')
cmo.setAlarmType('AutomaticReset')
cmo.setAlarmResetPeriod(60000)

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/WatchNotification/ClusterMonitor')
cmo.createSNMPNotification('snmptrap')

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/WatchNotification/ClusterMonitor/SNMPNotifications/snmptrap')
cmo.setEnabled(true)

cd('/WLDFSystemResources/ClusterMonitor/WLDFResource/ClusterMonitor/WatchNotification/ClusterMonitor/Watches/clusterGone')
set('Notifications',jarray.array([ObjectName('com.bea:Name=snmptrap,Type=weblogic.diagnostics.descriptor.WLDFSNMPNotificationBean,Parent=[osb_domain]/WLDFSystemResources[ClusterMonitor],Path=WLDFResource[ClusterMonitor]/WatchNotification[ClusterMonitor]/SNMPNotifications[snmptrap]')], ObjectName))

activate()

When the watch triggers, it logs this in the Server log:

####<16-Aug-2011 15:29:21 o'clock BST> <notice> <diagnostics> <pierrepc> <ms1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<wls Kernel>> <> <1b7e5955c26b51de:219aeffc:131d2f16195:-8000-0000000000000369> <1313504961943> <BEA-320068> <watch 'clusterGone' with severity 'Notice' on server 'ms1' has triggered at 16-Aug-2011 15:29:21 o'clock BST. Notification details: 
WatchRuleType: Harvester 
WatchRule: (${ServerRuntime//[weblogic.management.runtime.ClusterRuntimeMBean]com.bea:Name=myCluster,ServerRuntime=ms2,Type=ClusterRuntime//AliveServerCount} <3) ) 
WatchData: com.bea:Name=myCluster,ServerRuntime=ms1,Type=ClusterRuntime//AliveServerCount < 3  
WatchAlarmType: AutomaticReset 
WatchAlarmResetPeriod: 60000 
> 

If you need a free SNMP manager, download this Some snaphots here if you setup Watches and Notification through the WebLogic console: Selecting metrics to collect with the Harvester: select MBean

Selecting metrics to collect with the Harvester: select MBean attributes

selecting Managed Server instances from which to collect metrics

adding a clusterInTrouble notification which takes a diagnosticImage when the watch triggers:

assiging the clusterInTrouble notification to the clusterGone watch:

WLDF in action

This is a REALLY interesting article on how to setup SNMP alerts in WebLogic, using the WLDF framework

http://www.oracle.com/technetwork/articles/cico-wldf-091073.html

It's POWERFUL. One wonders why people don't use WLDF and SNMP by default to manage WebLogic domains.
The answer is probably that these products are not properly packaged for the end user, they are still cumbersome to setup and exploit... so people still rely on 3rd party agents and monitoring tools like Nagios, Hyperic etc.

It's a pity that such a valuable tool receives so little attention.