Fun run of JClarity Censum - Garbage Collection analysis tool

I have downloaded the 14 days trial of JClarity Censum. It's EXTREMELY simple to use and gives you a quick feedback on your GC issues.

First, you HAVE to enable a few flags: -XX:+PrintTenuringDistribution and -XX:+PrintGCDetails (safe in PROD). Then you run your tests for some 24 hours and feed your GC log into Censum.

It gives you statistical info like:

Longest GC pause (I had 5 seconds, quite bad)

Percentage of time spent in GC pauses (mine is 9%, pretty bad, should be max 5%)

Average memory allocation rate (I had 90 MB / second, quite bad)

Max memory allocation rate (I had 900 MB / second, horrible)

Total Full GC pause time, and Total Pause time

Full GC to GC ratio (mine was 45%, pretty stiff)

Application throughput (percentage of time spent working, versus time spent in GC.... should be 95% or more)

The graphs are breathtaking...

All in all, they charge you a quite hefty (ANNUAL!) fee for this software... to perform analysis that doesn't seem all that complicated - I am sure I can figure out most of this stuff in Excel in a matter of hours!

However, if your time is precious, it seems a quite valuable tool...

WebLogic: check which patches are applied

Just throwing all available technology at this issue....

Method 1: grep BEA-141107 in the logs (I don't get anything)

Method 2: bsu

cd /opt3/oracle/fmw11_1_1_5/utils/bsu
./bsu.sh -report -bea_home=/opt3/oracle/fmw11_1_1_5/

Patch Report
============
  Report Info
    Report Options
      bea_home.................. /opt3/oracle/fmw11_1_1_5/
      product_mask.............. ### OPTION NOT SET
      release_mask.............. ### OPTION NOT SET
      profile_mask.............. ### OPTION NOT SET
      patch_id_mask............. ### OPTION NOT SET
    Report Messages
  BEA Home.................. /opt3/oracle/fmw11_1_1_5

  Product Description
  Product Name.............. Oracle Coherence
  Product Version........... 3.6.0.4
  Installed Components...... Coherence Product Files
  Product Install Directory. /opt3/oracle/fmw11_1_1_5/coherence_3.6
  Java Home................. null
  Jave Vendor............... null
  Java Version.............. null
  Patch Directory........... /opt3/oracle/fmw11_1_1_5/patch_ocp360

  Product Description
  Product Name.............. WebLogic Server
  Product Version........... 10.3.5.0
  Installed Components...... Core Application Server, Administration Console, Configuration Wizard and Upgrade
 Framework, Web 2.0 HTTP Pub-Sub Server, WebLogic SCA, WebLogic JDBC Drivers, Third Party JDBC Drivers, WebLogic Server
 Clients, WebLogic Web Server Plugins, UDDI and Xquery Support, Evaluation Database, Workshop Code Completion Support

  Product Install Directory. /opt3/oracle/fmw11_1_1_5/wlserver_10.3
  Java Home................. null
  Jave Vendor............... Sun
  Java Version.............. 1.6.0_24
  Patch Directory........... /opt3/oracle/fmw11_1_1_5/patch_wls1035

Method 3: opatch

export MW_HOME=/opt3/oracle/fmw11_1_1_5/
export ORACLE_HOME=/opt3/oracle/fmw11_1_1_5/osb
export JDK_HOME=/opt3/oracle/java
/opt3/oracle/fmw11_1_1_5/oracle_common/OPatch/opatch lsinv -all -jdk /opt3/oracle/java/ -invPtrLoc /opt/oracle/bin/software/oraInst_soa3.loc

Invoking OPatch 11.1.0.8.2

Oracle Interim Patch Installer version 11.1.0.8.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt3/oracle/fmw11_1_1_5/osb
Central Inventory : /opt3/oracle/orainventory
   from           : /opt/oracle/bin/software/oraInst_soa3.loc
OPatch version    : 11.1.0.8.2
OUI version       : 11.1.0.9.0
OUI location      : /opt3/oracle/fmw11_1_1_5/osb/oui
Log file location : /opt3/oracle/fmw11_1_1_5/osb/cfgtoollogs/opatch/opatch2014-10-02_12-06-03PM.log

Patch history file: /opt3/oracle/fmw11_1_1_5/osb/cfgtoollogs/opatch/opatch_history.txt


OPatch detects the Middleware Home as "/opt3/oracle/fmw11_1_1_5"

Lsinventory Output file location : /opt3/oracle/fmw11_1_1_5/osb/cfgtoollogs/opatch/lsinv/lsinventory2014-10-02_12-06-03PM.txt

--------------------------------------------------------------------------------
List of Oracle Homes:
  Name          Location
   OH827611496         /opt3/oracle/fmw11_1_1_5/oracle_common
   OH401177498         /opt3/oracle/fmw11_1_1_5/osb

Installed Top-level Products (1):

Oracle Service Bus                                                   11.1.1.5.0
There are 1 products installed in this Oracle Home.


Interim patches (1) :

Patch  12362492     : applied on Mon Sep 15 16:37:11 CEST 2014
Unique Patch ID:  14334178
   Created on 5 Dec 2011, 21:20:01 hrs PST8PDT
   Bugs fixed:
     12362492



--------------------------------------------------------------------------------

OPatch succeeded.

Method 4: look in WebLogic console, Monitoring tab, you should have a list of patches

Since the patch was applied with opatch, it can only be detected by opatch. Strange thought that no trace of this patch appears in the logs...

I assume that bsu is basically dead now, and all patches should be installed with opatch... I am not an expert though...

WebLogic: Registered more than one instance with the same objectName

We keps getting an error in the logs:

BEA-149500> (RuntimeMBeanDelegate.java:255)
        at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
        at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:193)
        at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:182)

until we restarted and it went away. No change was done in the configuration recently.

Looking in Oracle Support I found:

BEA-149500 Error Registered more than one instance with the same objectName using JDBC TLOG Store (Doc ID 1544879.1)

WebLogic Server tries to create a new MBean every time a PersistentStoreRuntimeMBean is called. This issue has been fixed in unpublished defect 16063328

Our MBean type is "JMSPooledConnectionRuntime" and not "PersistentStoreRuntime", but maybe the origin is the same... anyway I am not going to apply the patch, as this has occurred only once....