SiteScope and configuring a MDB for monitoring

Are you lost? That's normal.... SiteScope is designed with that purpose in mind.

When adding a counter, search for:

com.bea.YOURSERVERNAME.APPLICATION_NAME.MODULE_NAME.
EJBPoolRuntime.JAR_NAME.MODULE_NAME.BeansInUseCount

(or whatever other attribute)

practical example:

com.bea/ACMEappli2/InfoHostingProcessesMdbBean_jms/infoHostingProcessesQueue/InfoHostingProcessesMdbBean_jms/infoHostingProcessesQueue/infoHostingProcesses/EJBPoolRuntime/infoHostingProcesses.jar/BeansInUseCurrentCount


Easy, isn't it?

Anatomy of a MDB

EJBRuntimeMBean is the base Interface for all EJB-related MBeans.

subinterfaces are:

- MessageDrivenEJBRuntimeMBean, EntityEJBRuntimeMBean and StatelessEJBRuntimeMBean, which all contain a EJBPoolRuntimeMBean:

http://download.oracle.com/docs/cd/E12839_01/apirefs.1111/e13945/weblogic/management/runtime/EJBPoolRuntimeMBean.html


with attributes:

AccessTotalCount, BeansInUseCurrentCount, DestroyedTotalCount, MissTotalCount, PooledBeansCurrentCount, TimeoutTotalCount, WaiterCurrentCount

- StatefulEJBRuntimeMBean contains instead a EJBCacheRuntimeMBean (Stateful are cached, not pooled, since they contain stateful info which should not be discarded or shared) and EJBLockingRuntimeMBean

All types of Bean contain a EJBTransactionRuntimeMBean.


How do I retrieve a MDB RuntimeMBean?

From a DomainRuntimeMBean, retrieve a ServerRuntimeMBean, then its ApplicationRuntimeMBean[], then its ComponentRuntimeMBean[], and test if its type is EJBComponentRuntime, in this case you can cast it to EJBComponentRuntimeMBean

Painful, no? Here is the code:


public List getEBJs(String serverName) {
List result = new ArrayList();
ServerRuntimeMBean serverRuntimeMBean = getServerRuntimeMBean(serverName);
if (serverRuntimeMBean != null) {
ApplicationRuntimeMBean[] apps = serverRuntimeMBean.getApplicationRuntimes();
for (ApplicationRuntimeMBean app : apps) {
ComponentRuntimeMBean[] components = app.getComponentRuntimes();
for (ComponentRuntimeMBean bean : components) {
if (bean.getType().equals("EJBComponentRuntime")) {
result.add((EJBComponentRuntimeMBean)bean);
}
}
}
}
return result;
}

What to monitor in an Application Server

EJBCache (ActivationCount CacheAccessCount CachedBeansCurrentCount CacheHitCount PassivationCount)

EJBLocking (LockEntriesCurrentCount LockManagerAccessCount TimeoutTotalCount WaiterCurrentCount WaiterTotalCount)

EJBPool (BeansInUseCount IdleBeansCount TimeoutTotalCount WaiterTotalCount)

EJBTransactions (TransactionsCommittedTotalCount TransactionsRolledBackTotalCount TransactionsTimedOutTotalCount)

JDBC (ActiveConnectionsCurrentCount ActiveConnectionsHighCount ConnectionsTotalCount LeakedConnectionCount MaxCapacity WaitingForConnectionCurrentCount WaitingForConnectionHighCount WaitSecondsHighCount)

JMS (MessagesCurrentCount MessagesHighCount MessagesPendingCount MessagesReceivedCount BytesCurrentCount BytesHighCount BytesPendingCount BytesReceivedCount)

JRockit (JvmProcessorLoad AllProcessorsAverageLoad TotalGarbageCollectionTime)

ThreadPool (busyThrdCurrCnt idleThrdCurrCnt totalThrdCurrCnt standbyThrdCurrCnt stuckThrdCurrCnt queueLength)

kill -3 and JRockit

in your domain home, create a file ctrlhandler.act

set_filename filename=/tmp/output.txt
print_class_summary
heap_diagnostics
print_threads
timestamp
jrarecording nativesamples=false latency filename=myjrarecording time=120
stop


every time you do a kill -3 pid, the JRockit VM will execute the control handler.

http://download.oracle.com/docs/cd/E13188_01/jrockit/geninfo/diagnos/ctrlbreakhndlr.html

Which JMS Connection Factory shall I use?

unless a Connection Factory has been explicitely specified for a Queue, WebLogic will use a default connection factory named:

JMS_FACTORY="weblogic.jms.ConnectionFactory"

this will be in the JNDI_TREE for you to use:

Here is the (crazy and cumbersome) code:

Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory");
env.put(Context.PROVIDER_URL,"t3://localhost:7001");
InitialContext ctx = new InitialContext(env);


QueueConnectionFactory   qConFactory = (QueueConnectionFactory)ctx.lookup("weblogic.jms.ConnectionFactory");
QueueConnection     qCon        = qConFactory.createQueueConnection();
QueueSession     qSession    = qCon.createQueueSession(false,Session.AUTO_ACKNOWLEDGE);
Queue       queue2       = (Queue)ctx.lookup(jms_queue);
QueueSender        qSender     = qSession.createSender(queue2);
TextMessage      msg         = qSession.createTextMessage();

qCon.start();

JMS_BEA_DeliveryFailureReason

a property named JMS_BEA_DeliveryFailureReason is being used in error messages in a WLI application; its value is an integer, and it means that the delivery limit was reached. The number 2 has nothing to do with the delivery count. See weblogic.jms.backend.BEDestinationImpl.

JRMC JRockit mission control, unable to connect console

still investigating on this problem, most likely I have not set -Xmanagement when running the WebLogic server... Thanks to the developers for the immense clarity of the error message.

Oh yes of course, you should change your startWebLogic.cmd adding
set JAVA_OPTIONS=%JAVA_OPTIONS% -Xmanagement:ssl=false,authenticate=false,port=7099

once you restart weblogic, you should see this:
[INFO ][mgmnt  ] Remote JMX connector started at address esp50ws112345:7099
[INFO ][mgmnt  ] Local JMX connector started

Of course make sure you specify 7099 (and NOT 7001) in the JRMC client new connection parameters.


This was the original error message:

Could not open Management Console for frparmat01.acme.dns:8765.
  com.jrockit.mc.rjmx.ConnectionException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
    java.io.EOFException]
    com.jrockit.mc.rjmx.ConnectionException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.io.EOFException]
        at com.jrockit.mc.rjmx.ConnectionManager.connect(ConnectionManager.java:63)
        at com.jrockit.mc.console.ui.actions.StartConsole$1.preConnect(StartConsole.java:38)
        at com.jrockit.mc.browser.utils.PreConnectJob.run(PreConnectJob.java:74)
        at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
    Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.io.EOFException]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:317)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:229)
        at com.jrockit.mc.rjmx.internal.RJMXConnection.setupServer(RJMXConnection.java:537)
        at com.jrockit.mc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:144)
        at com.jrockit.mc.rjmx.internal.RJMXConnectorModel.establishConnection(RJMXConnectorModel.java:111)
        at com.jrockit.mc.rjmx.internal.RJMXConnectorModel.connect(RJMXConnectorModel.java:154)
        at com.jrockit.mc.rjmx.ConnectionManager.innerConnect(ConnectionManager.java:95)
        at com.jrockit.mc.rjmx.ConnectionManager.connect(ConnectionManager.java:61)
        ... 3 more
    Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.io.EOFException]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
        at javax.naming.InitialContext.lookup(InitialContext.java:351)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1817)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1787)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:259)
        ... 10 more
    Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.io.EOFException
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:273)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:306)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:93)
        ... 15 more
    Caused by: java.io.EOFException
        at java.io.DataInputStream.readByte(DataInputStream.java:243)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215)
        ... 19 more

JMS: difference between PENDING and CURRENT

MessagesPendingCount

The number of pending messages in the destination.
Pending messages are over and above the current number of messages. A pending message is one that has either been sent in a transaction and not committed, or that has been received and not committed or acknowledged.

MessagesCurrentCount

The current number of messages in the destination. This does not include the pending messages.


When you do "show messages",

The beauty of verbose:class

java -verbose:class is simply wonderful, when you are trying to troubleshoot classloader problems.

Here is a sample output:

[INFO ][class  ] created: #2859 org/apache/poi/poifs/filesystem/POIFSDocument (file:/acme/users/abraham/abrahammonitoring/lib/poi-3.5-FINAL-20090928.jar)
[INFO ][class  ] 4391 org/apache/poi/poifs/filesystem/POIFSDocument success (0.36 ms)
[INFO ][class  ] 4394 org/apache/poi/poifs/storage/DocumentBlock fail (0.00 ms)
[INFO ][class  ] 4396 org/apache/poi/poifs/storage/BigBlock fail (0.00 ms)
[INFO ][class  ] created: #2861 org/apache/poi/poifs/storage/BigBlock (file:/acme/users/abraham/abrahammonitoring/lib/poi-3.5-FINAL-20090928.jar)
[INFO ][class  ] 4395 org/apache/poi/poifs/storage/BigBlock success (0.12 ms)
[INFO ][class  ] created: #2860 org/apache/poi/poifs/storage/DocumentBlock (file:/acme/users/abraham/abrahammonitoring/lib/poi-3.5-FINAL-20090928.jar)
[INFO ][class  ] 4393 org/apache/poi/poifs/storage/DocumentBlock success (0.30 ms)
[INFO ][class  ] 4398 org/apache/poi/poifs/filesystem/POIFSDocument$BigBlockStore fail (0.00 ms)
[INFO ][class  ] created: #2862 org/apache/poi/poifs/filesystem/POIFSDocument$BigBlockStore (file:/acme/users/abraham/abrahammonitoring/lib/poi-3.5-FINAL-20090928.jar)

Grepper Monkey or Singe Greppeur

"Grepper Monkey" is a guy who spends his time grepping log files in search for errors, and reporting them to developers....
this is what I am doing right now... I am writing a tool to do this intelligently though....

WebLogic portal generating lot of info into the diagnostic framework store file WLS_DIAGNOSTICS000000.DAT

WLS_DIAGNOSTICS000000.DAT

try using:

-Dcom.bea.wlw.netui.disableInstrumentation=true
-D_Offline_FileDataArchive=true


in your startup command, especially of the admin

it seems to be a "best practice"
http://forums.oracle.com/forums/thread.jspa?messageID=3302556&#3302556

Connection Pools reaching their max capacity and over

http://docs.oracle.com/cd/E23943_01/apirefs.1111/e13952/taskhelp/jdbc/jdbc_datasources/ConfigureDiagnosticsForADataSource.html

to make sure you are not leaking connections (forgetting to close them),
in your DataSource configuration go to "configuration"/diagnostics and check the "Profile Connection Leak"
This will generate a "profile-type" element in the jdbc-config.xml and set it to 4.
then in your Connection Pool configuration, click on advanced, and enter some large value for "Inactive Connection Timeout" (should be greater than the JTA timeout IMHO)

For more details, see Oracle Support Doc "How to enable WLS Database Profiling to diagnose database connection problems in Webcenter Content (Doc ID 2071557.1) "

WLI: how to clean up a WLI environment

- TERMINATE all processes (using for instance the WLI Console)
- purge all tasks
- stop all processes and task applications
- uninstall all processes and task applications
- stop application servers
- delete all this:

¤ / pathToYourDomain / yourDomain /wliconfig /ProcessTypeProperties.xml

¤ / pathToYourDomain / yourDomain /wliconfig /AppliServers /ProcessTypeProperties.xml

¤ / pathToYourDomain / yourDomain /servers/ adminServer /cache/*

¤ / pathToYourDomain / yourDomain /servers/ adminServer /tmp/*

¤ / pathToYourDomain / yourDomain /servers/ appliServer /cache/*

¤ / pathToYourDomain / yourDomain /servers/ appliServer /tmp/*

¤ / pathToYourDomain / yourDomain /servers/ appliServer /stage


DB:
- you must disable constraints
- truncate all tables starting by JPD_* WLI_*, BUT WLI_CALENDAR and WLI_CALENDAR_SEQUENCE

restart servers
ProcessTypeProperties.xml will be recreated


verify with WLI Console that no process nor process instance exist

redeploy processes and tasks applications

recreate all resources

WLST useful tips and tricks

Run a .py script with:

java weblogic.WLST scriptname.py


How to define a function with exception handling:

def connect_to_weblogic(username,password,adminurl):
try :
connect(username,password,adminurl)
except :
print 'Cannot connect to server '
exit()


How to get the list of defined servers:

serverList = ls('Servers',returnMap='true')

(the "returnMap" options makes the serverList variable a HashMap with key=serverName)


How to get a list of RUNNING servers:

domainRuntime()
runningList = ls('ServerRuntimes',returnMap='true')

Windows XP crashing with message "missing or corrupt system32/hal.dll"

Welcome to the wonderful world of Bill Mafia and his treacherous connections with the DOD USA which allowed him to impose this horrible piece of crap as a world standard.

You can boot the PC with UBUNTU 9.10, retrieve the hal.dll file from the c:/windows directory and copy it to c:/windows/system32 directory! It works! Don't ask me why there are 2 copies of the same file!

JRockit crash resolved

if your dump file contains something like:

Thread Stack Trace:
at known_ld_opt+685()@0x2b36a098d091
at strength_reduction_generic+158()@0x2b36a098d287
at optStrengthReduction+152()@0x2b36a098d36b
at optmanOptimizeMIR+330()@0x2b36a0935cd7
at generateMethodWithStage+102()@0x2b36a08212b7
at cmgrGenerateMethodFromPhase+216()@0x2b36a0822126
at cmgrGenerateNormalMethod+91()@0x2b36a082122d
at cmgrGenerateCode+209()@0x2b36a08210c1
at generate_code2+205()@0x2b36a09048a3
at codegenThread+1006()@0x2b36a0904f45
at tsiCallStartFunction+67()@0x2b36a08dd5cb
at tsiThreadStub+308()@0x2b36a08de7d5
at ptiThreadStub+14()@0x2b36a09468a6
at start_thread+199()@0x3def806367


a workaround could be that you add the method reported by the
Method :
line to an exclusion file myOptfile, which tells JRockit which methods it should NOT optimize.

then start your application with the flags
-Xverbose:opt -Djrockit.optfile=myOptfile

WebLogic Performance checklist

Pinned-To-Thread on JDBC Connection Pool : to enable

"Set XA Transaction Timeout" and "XA Transaction Timeout" sets the timeout on this resource. It is advisable to define it explicitly and must be greater than the overall timeout of the transaction (and therefore higher than the overall timeout of JTA or of timeout of the EJB that initiated the transaction).

JMS bridges and Distributed Destinations

http://download.oracle.com/docs/cd/E11035_01/wls100/bridge/bridgefaq.html

(quote)
Can the messaging bridge use distributed destinations as source and target destinations?

Yes, the messaging bridge can send to and receive from distributed destinations. Bea recommends the following configurations:

* If the source is distributed destination, the bridge is pinned to one of the members when it connects to the destination. It stays connected only to that member until it reconnects. This means that the bridge will not receive messages from the other members of the distributed destination. Therefore, the best practice is to configure one bridge for each member of a distributed destinations using the member's JNDIName.

* If the target is a distributed destination, the best practice is to send to the distributed destination using the distributed destination’s JNDIName and disable server affinity. This allows the distributed destination to load balance incoming messages.
(unquote)

Querying and terminating WLI instances

see http://download-llnw.oracle.com/docs/cd/E13214_01/wli/docs92/wli.javadoc/com/bea/wli/management/runtime/ProcessRuntimeMBean.html#terminate(java.lang.String,%20java.lang.String)

package com.acme.wli;

import java.util.Iterator;
import java.util.Set;

import javax.naming.Context;
import javax.naming.NamingException;

import com.bea.wli.bpm.runtime.ProcessStatus;
import com.bea.wli.management.runtime.ProcessInstanceQuery;
import com.bea.wli.management.runtime.ProcessInstanceQueryResult;
import com.bea.wli.management.runtime.ProcessRuntimeMBean;

import weblogic.jndi.Environment;
import weblogic.management.MBeanHome;
/**
* Ajouter D:\beawliwlp103\wli_10.3\lib\jpdpublic.jar:
* @author vernetto
*
*/
public class WLITerminator {
//private static final String SERVICE_URI = "/ProtectionProcesses/com/acme/corp/protectionProcesses/iris/ReceiveCIUN_v5.jpd";
//private static final String SERVICE_URI = "/ProtectionProcesses/com/acme/corp/protectionProcesses/iris/CotationDataReady_v4.jpd";
private static final String SERVICE_URI = "/ProtectionProcesses/com/acme/corp/protectionProcesses/iris/HandleCIUN_v1.jpd";

public static void main(String[] args) throws NamingException {
Environment env = new Environment();
env.setSecurityPrincipal("weblogic");
env.setSecurityCredentials("weblogic");
env.setProviderUrl("t3://admmaintatlas.acme.dns:8501");
Context ctx = env.getInitialContext();
MBeanHome home = (MBeanHome)ctx.lookup(MBeanHome.ADMIN_JNDI_NAME);
System.out.println("Got the Server-specific MBeanHome: " + home);
Set s = home.getMBeansByType("ProcessRuntime");
Iterator it = s.iterator();
try {
            if (it.hasNext()) {
                ProcessRuntimeMBean bean = (ProcessRuntimeMBean)it.next();
                ProcessInstanceQuery query = new ProcessInstanceQuery();
                //query.setServiceURI(SERVICE_URI);
                query.setStatus(ProcessStatus.ABORTED);
                ProcessInstanceQueryResult info = bean.getProcessInstances(query);
                int count = info.getResultCard();
                String[] instances = info.getInstanceIds();
                System.out.println("found " + instances.length + " instances");
                for (int i = 0 ; i <  instances.length; i++) {
                    String instanceId = instances[i];
                    ProcessInstanceInfo pi = bean.getProcessInstanceInfo(instanceId, false);
                    System.out.println(pi.getServiceURI() + " " + instanceId);
                    //bean.terminate(pi.getServiceURI(), instanceId);
                }}
} catch (Exception ex) {
System.out.println(ex);
ex.printStackTrace();
}
}
}

WLI: configuring JMS queues for performance

The JMS queues AsyncDispatcher and AsyncDispatcher_error are needed for each project containing process.
All WLI process are based on these 2 queues, and it is therefore necessary to configure them properly.

For the queue AsyncDispatcher:
- Retry => 0 or a low value with a redelivery delay set. The retry may be useful in case of unavailability of external system.
- Error destination => AsyncDispatcher_error.

For the queue AsyncDispatcher_error:
- Retry => maximum value in order not to lose messages.
- Redelivery delay positioned so as not to saturate the server.

BEA-190032

<Warning> <Connector> <<WLS Kernel>> <BEA-190032> << eis/jms/WLSConnectionFactoryJNDIXA > ResourceAllocationException thr
own by resource adapter on call to ManagedConnectionFactory.createManagedConnection(): "javax.resource.ResourceException: Failed to start the connection ">


it's evidently a JMS Bridge problem.

This teaches me the following:

The bridge uses a JCA Connector packaged in this jar:

jms-xa-adp.rar

and its JNDI name is eis.jms.WLSConnectionFactoryJNDIXA

see more info here:

http://weblogic.sys-con.com/node/44439

.... more to come...

WebLogic: unable to log into console

if you can't login into your console and you get a message:

"It's likely that this condition is the result of a configuration problem"

and a "response has already been committed" in the logs,

search in your $DOMAIN_HOME/pending directory and remove what might be there. Also, clean your config.xml from any garbage, and copy it to any other nodes.

If this still fails, empty all "stage" and "tmp" directories.

Avoiding overload

http://download.oracle.com/docs/cd/E12839_01/web.1111/e13701/overload.htm


in a nutshell:

* limit the number of execute threads with JVM properties:
-Dweblogic.threadpool.MinPoolSize=x
-Dweblogic.threadpool.MaxPoolSize=y

* create a Network Channel and assign to it a "Maximum Connected Clients"

* in configuration / overload, set "Shared Capacity For Work Managers"

* for WebApps, set the max-in-memory-sessions in the deployment descriptor

* create a global Work Manager called default, and assign constraints to it

(see http://download.oracle.com/docs/cd/E11035_01/wls100/config_wls/self_tuned.html)


In post 8.1 WebLogic, the threads belong to a unique queue weblogic.kernel.Default.

Hogging Threads, Idle Threads...

See here for more info.

A thread in the pool can be: Standby, Active, Stuck or Idle

The Monitoring Threads console shows us:

Hogging Thread Count: not well defined. Seemingly, a hogging thread is a thread which has been busy for a "long" (how long?) time... the difference between a hogging thread and a stuck thread is not clear.



Standby Thread Count: the threads who are sitting idle in the standby pool

Queue Length:

Execute Thread Idle Count:

Execute Thread Total Count:

Active Execute Threads: the threads actually working now





Standby Thread Count + Active Execute Threads = Execute Thread Total Count

JRockit, how to enable a .core file (core dump)

first, check with "ulimit -a" that you have no limits on core file size:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 137215
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137215
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


if your file is limited, then issue a
"ulimit -c unlimited"


check that no "-XXdumpSize" is specified in your java command options
(default is "large")

Also, run df -h to make sure you have enough space on disk

JRockit crash and dump

When JRockit crashes, it produces a .dump file

for more info see:
http://download.oracle.com/docs/cd/E13188_01/jrockit/geninfo/diagnos/dumpfile.html


http://download.oracle.com/docs/cd/E13188_01/jrockit/geninfo/diagnos/crash.html

here is a sample .dump file:


===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 23:20:28 on Wed Nov 4 17:13:48 2009

**********************************************************
* If you see this dump, please go to *
* http://edocs.bea.com/jrockit/go2troubleshooting.html *
* for troubleshooting information. *
**********************************************************

Additional information is available in:
/acme/serveur/wlsapp/IRISDomain/jrockit.27444.dump
No snapshot file (core dump) will be created because core dumps have been
disabled. To enable core dumping, try "ulimit -c unlimited"
before starting JRockit again.

Error Message: Illegal memory access. [54]
Signal info : si_signo=11, si_code=1 si_addr=(nil)
Version : BEA JRockit(R) R27.6.3-40_o-112056-1.5.0_17-20090318-2103-linux-x86_64
CPU : Intel Core 2 SSE SSE2 SSE3 SSSE3 SSE4.1 Core Intel64
Number CPUs : 8
Tot Phys Mem : 16833388544 (16053 MB)
OS version : Red Hat Enterprise Linux Server release 5.3 (Tikanga)
Linux version 2.6.18-128.el5 (mockbuild@hs20-bc1-7.build.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Dec 17 11:41:38 EST 2008 (x86_64)
Thread System: NPTL
Java locking : Normal
State : JVM is running
Command Line : -Xms4096m -Xmx4096m -Dweblogic.security.SSL.trustedCAKeyStore=/opt/tuxedo/platform922/weblogic92/server/lib/cacerts -Xverify:none -XgcPrio:throughput -Xverbose:gc,gcreport -XverboseTimeStamp -XverboseLog:/serveur/wlsapp/logs/irisappli1/gc.log -da -Dplatform.home=/opt/tuxedo/platform922/weblogic92 -Dwls.home=/opt/tuxedo/platform922/weblogic92/server -Dwli.home=/opt/tuxedo/platform922/weblogic92/integration -Dweblogic.wsee.bind.suppressDeployErrorMessage=true -Dweblogic.http.descriptor.merge=true -Dlog4j.configuration=log4j.xml -Dweblogic.management.discover=false -Dweblogic.management.server=t3://admmaintiris.acme.dns:8501 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.Stdout=/serveur/wlsapp/logs/irisappli1/stdout.log -Dweblogic.Stderr=/serveur/wlsapp/logs/irisappli1/stderr.log -Dweblogic.ext.dirs=/opt/tuxedo/platform922/patch_weblogic922/profiles/default/sysext_manifest_classpath -Duser.language=fr -Duser.country=FR -Dacme.config.path=/serveur/wlsapp/IRISDomain/acmeConfig -Dacme.framework.server=true -Dlog4j.debug=true -Dacme.persistence.loginAtStartup=false -Dacme.server.standAlone=false -Dweblogic.Name=irisappli1 -Djava.security.policy=/opt/tuxedo/platform922/weblogic92/server/lib/weblogic.policy -Dsun.java.launcher=SUN_STANDARD weblogic.Server
java.home : /opt/jrmc-3.1.0-1.5.0/jre
j.class.path : :/opt/tuxedo/platform922/patch_weblogic922/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/jrockit/lib/tools.jar:/opt/tuxedo/platform922/weblogic92/server/lib/weblogic_sp.jar:/opt/tuxedo/platform922/weblogic92/server/lib/weblogic.jar:/opt/tuxedo/platform922/weblogic92/server/lib/webservices.jar::/opt/tuxedo/platform922/weblogic92/server/lib/jpd.jar:/opt/tuxedo/platform922/weblogic92/integration/L10N:/opt/tuxedo/platform922/weblogic92/integration/lib/worklist-system-required.jar:/opt/tuxedo/platform922/weblogic92/common/eval/pointbase/lib/pbclient51.jar:/opt/tuxedo/platform922/weblogic92/server/lib/xqrl.jar:/serveur/wlsapp/IRISDomain/acmeConfig::/serveur/wlsapp/IRISDomain/acmeLib/framework.jar:/serveur/wlsapp/IRISDomain/acmeLib/commons-beanutils-1.7.0.jar:/serveur/wlsapp/IRISDomain/acmeLib/toplink.jar:/serveur/wlsapp/IRISDomain/acmeLib/baseInfo.jar:/serveur/wlsapp/IRISDomain/acmeLib/commons-lang-2.1.jar:/serveur/wlsapp/IRISDomain/acmeLib/jmxtools-1.1.jar:/serveur/wlsapp/IRISDomain/acmeLib/wtc-internalCollaborate.jar:/serveur/wlsapp/IRISDomain/acmeLib/commons-logging-1.0.2.jar:/serveur/wlsapp/IRISDomain/acmeLib/commons-pool-1.3.jar:/serveur/wlsapp/IRISDomain/acmeLib/log4j-1.2.14.jar:/serveur/wlsapp/IRISDomain/acmeLib/xmlparserv2.jar:/serveur/wlsapp/IRISDomain/acmeLib/baseInfoXmlBeans.jar:/serveur/wlsapp/IRISDomain/acmeLib/jmxri-1.1.jar:/serveur/wlsapp/IRISDomain/acmeLib/ehcache-1.2.2.jar:/serveur/wlsapp/IRISDomain/acmeLib/baseXmlBeans.jar:/serveur/wlsapp/IRISDomain/acmeLib/externalCollaborateField32.jar:/serveur/wlsapp/IRISDomain/acmeLib/wrapperMgt.jar:/serveur/wlsapp/IRISDomain/acmeLib/commons-digester-1.7.jar:/serveur/wlsapp/IRISDomain/acmeLib/xercesImpl.jar:/serveur/wlsapp/IRISDomain/acmeLib/dbunit-2.2.jar:/serveur/wlsapp/IRISDomain/acmeLib/commons-collections-3.2.jar:/serveur/wlsapp/IRISDomain/acmeLib/xml.jar:/serveur/wlsapp/IRISDomain/acmeLib/base.jar:/serveur/wlsapp/IRISDomain/acmeLib/ojdbc14.jar:/serveur/wlsapp/IRISDomain/acmeLib/castor-1.0.5-xml.jar:/opt/tuxedo/platform922/weblogic92/server/lib/xquery.jar:/opt/tuxedo/platform922/weblogic92/server/lib/binxml.jar:/serveur/wlsapp/IRISDomain/configTemp/:
j.lib.path : /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/jrockit:/opt/jrmc-3.1.0-1.5.0/jre/lib/amd64:/opt/jrmc-3.1.0-1.5.0/jre/../lib/amd64:/opt/tuxedo/platform922/patch_weblogic922/profiles/default/native:/opt/tuxedo/platform922/patch_weblogic922/profiles/default/native:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64/oci920_8:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64/oci920_8
JAVA_HOME : /opt/jrockit
_JAVA_OPTIONS: <not set>
LD_LIBRARY_PATH: /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/jrockit:/opt/jrmc-3.1.0-1.5.0/jre/lib/amd64:/opt/jrmc-3.1.0-1.5.0/jre/../lib/amd64:/opt/tuxedo/platform922/patch_weblogic922/profiles/default/native:/opt/tuxedo/platform922/patch_weblogic922/profiles/default/native:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64/oci920_8:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64:/opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64/oci920_8
LD_ASSUME_KERNEL: <not set>
Method : com/acme/corp/request/RequestCtrlImpl.modifyRequest(Lcom/acme/x2002/bdt/RqtUpdtBDTDocument;)Lcom/acme/x2002/bdt/RqtBDTDocument;
StackOverFlow: 0 StackOverFlowErrors have occured
OutOfMemory : 0 OutOfMemoryErrors have occured
C Heap : Good; no memory allocations have failed
GC Strategy : Mode: throughput. Currently using strategy: genparpar
GC Status : OC is not running. Last finished OC was OC#393.
: YC is not running. Last finished YC was YC#84.
OC History : Strategy genparpar was used for OC#125.
: Strategy singleparpar was used for OC#126 to OC#127.
: Strategy genparpar was used for OC#128.
: Strategy singleparpar was used for OC#129.
: Strategy genparpar was used for OC#130 to OC#393.
YC History : Ran 3 YCs before OC#389.
: Ran 1 YCs before OC#390.
: Ran 2 YCs before OC#391.
: Ran 2 YCs before OC#392.
: Ran 3 YCs before OC#393.
: Ran 2 YCs since last OC.
YC Promotion : Last YC successfully promoted all objects
Heap : 0x2ad688312000 - 0x2ad788312000 (Size: 4096 MB)
Compaction : 0x2ad6c8312000 - 0x2ad6d8312000 (Current compaction type: external)
NurseryList : 0x2ad6c7eafa98 - 0x2ad752410450
KeepArea : 0x2ad70f0e22d0 - 0x2ad752410450
NurseryMarker: [ 0x2ad6f7450178, 0x2ad70f0e22d0 ]
CompRefs : References are uncompressed 64-bit.

Registers (from ThreadContext: 0x41aaa3b0 / OS context: 0x41aaa470):
rax = 00002ad68806f091 rcx = 0000000000000000
rdx = 00002ad6880d1fac rbx = 0000000000000000
rsp = 0000000041aaa8b0 rbp = 0000000041aaa8f0
rsi = 000000002000002f rdi = 0000000000000000
r8 = 0000000000000000 r9 = 0000000041aaa8c4
r10 = 0000000000000000 r11 = 00002aaaaeb0dbc8
r12 = 0000000041aaabb0 r13 = 00002aaaafcb8f70
r14 = 0000000000000001 r15 = 00002aaac50cb408
cs = 0000000000000033 fs = 0000000400000000
gs = 0004000000000000
rip = 00002ad68806f091 flags = 0000000000000213

Loaded modules:
(* denotes the module causing the exception)
0000000000400000-00000000004119eb /opt/jrmc-3.1.0-1.5.0/bin/java
0000003def800000-0000003def815507 /lib64/libpthread.so.0
0000003a6aa00000-0000003a6aa8122f /lib64/libm.so.6
0000003def000000-0000003def001f93 /lib64/libdl.so.2
0000003deec00000-0000003deed4b937 /lib64/libc.so.6
0000003dee800000-0000003dee81b99f /lib64/ld-linux-x86-64.so.2
00002ad687eae000-00002ad6880de56f */opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/jrockit/libjvm.so
0000003df0800000-0000003df0806e8b /lib64/librt.so.1
00002ad79131d000-00002ad79132664b /lib64/libnss_files.so.2
00002ad791587000-00002ad791593467 /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libverify.so
00002ad791696000-00002ad7916b921b /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libjava.so
0000003df2800000-0000003df28140c7 /lib64/libnsl.so.1
00002ad7d1850000-00002ad7d185663f /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/native_threads/libhpi.so
00002aaaae0b6000-00002aaaae0c3fa3 /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libzip.so
00002aaaae1fa000-00002aaaae20b5cf /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libnet.so
00002aaaae30e000-00002aaaae3145db /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libnio.so
00002aaaae41f000-00002aaaae422d73 /lib64/libnss_dns.so.2
0000003df3c00000-0000003df3c10823 /lib64/libresolv.so.2
00002aaaae624000-00002aaaae62918b /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libmanagement.so
00002aaaae72b000-00002aaaae73363f /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/libjmapi.so
00002aaaae845000-00002aaaae8472af /opt/tuxedo/platform922/weblogic92/server/native/linux/x86_64/libwlfileio2.so
00002aaaae9ff000-00002aaaae9ff96b /opt/jrmc-3.1.0-1.5.0/jre/lib/amd64/librmi.so

Stack:
(* marks the word pointed to by the stack pointer)
0000000041aaa8b0: 0000000041aaa920* 000000008805e841 0000000310000408 00002aaaafcb8f70
0000000041aaa8d0: 0000000041aaabb0 0000000000000018 0000000041aaa934 0000000041aaabb0
0000000041aaa8f0: 0000000041aaa920 00002ad68806f287 00000000000005dc 00002aaac50cb408
0000000041aaa910: 00000000000006d8 0000000000000000 0000000041aaa960 00002ad68806f36b
0000000041aaa930: 0000000000000000 0000000000000000 0000000000000001 0000000041aaabb0
0000000041aaa950: 0000000000000000 00002aaac50cb408 0000000041aaa9b0 00002ad688017cd7

Code:
(* marks the word pointed to by the instruction pointer)
00002ad68806f030: 49c08949fff1298f 6674c0854870458b 00000001b8063883 894858ebf0440f44
00002ad68806f050: f8e28102eac148f2 484057034903ffff e0c1481fe083f089 08488b4802034804
00002ad68806f070: 7708fe8341d1014c 29158d48f0894432 488204634800062f eb01634ce0ffd001
00002ad68806f090: 0f4c10eb018b4c15* 01b70f4c0aeb01bf 01bf01be0f4c04eb 415f74ff85000000
00002ad68806f0b0: 8944c2894c485d8b 000000b8ff894cf6 0f41ffee8f59e800 0f06e9c166044db7
00002ad68806f0d0: 048908245c89c9b6 4100000001b94124 0000ba00000001b8 4c0000001dbe0000


"(Code Optimization Thread 1)" id=5 idx=0x34 tid=27456 lastJavaFrame=0xfffffffffffffffc

Stack 0: start=0x41a8a000, end=0x41aac000, guards=0x41a8f000 (ok), forbidden=0x41a8d000
Thread Stack Trace:
at known_ld_opt+685()@0x2ad68806f091
at strength_reduction_generic+158()@0x2ad68806f287
at optStrengthReduction+152()@0x2ad68806f36b
at optmanOptimizeMIR+330()@0x2ad688017cd7
at generateMethodWithStage+102()@0x2ad687f032b7
at cmgrGenerateMethodFromPhase+216()@0x2ad687f04126
at cmgrGenerateNormalMethod+91()@0x2ad687f0322d
at cmgrGenerateCode+209()@0x2ad687f030c1
at generate_code2+205()@0x2ad687fe68a3
at codegenThread+1006()@0x2ad687fe6f45
at tsiCallStartFunction+67()@0x2ad687fbf5cb
at tsiThreadStub+308()@0x2ad687fc07d5
at ptiThreadStub+14()@0x2ad6880288a6
at start_thread+199()@0x3def806367
-- Java stack --

Extended, platform specific info:
libc release: 2.5-stable
Elf headers:
libc ehdrs: EI: 7f454c46020101000000000000000000 ET: 3 EM: 62 V: 1 ENTRY: 0000003deec1da70 PHOFF: 0000000000000040 SHOFF: 00000000001a1080 EF: 0x0 HS: 64 PS: 56 PHN; 10 SS: 64 SHN: 77 STIDX: 76
libpthread ehdrs: EI: 7f454c46020101000000000000000000 ET: 3 EM: 62 V: 1 ENTRY: 0000003def805780 PHOFF: 0000000000000040 SHOFF: 0000000000022ef8 EF: 0x0 HS: 64 PS: 56 PHN; 9 SS: 64 SHN: 39 STIDX: 38
libjvm ehdrs: EI: 7f454c46020101000000000000000000 ET: 3 EM: 62 V: 1 ENTRY: 000000000002de90 PHOFF: 0000000000000040 SHOFF: 000000000252ee50 EF: 0x0 HS: 64 PS: 56 PHN; 3 SS: 64 SHN: 30 STIDX: 27

**********************************************************
* If you see this dump, please go to *
* http://edocs.bea.com/jrockit/go2troubleshooting.html *
* for troubleshooting information. *
**********************************************************

===== END DUMP ===============================================================

Sitescope and Weblogic using JMX

a good introduction on the topic:

http://90kts.com/blog/2008/monitoring-weblogic-using-jmx-in-sitescope/

especially useful is the test using JConsole:

jconsole.exe -J-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:60000:500:10

and connecting to
service:jmx:rmi://jndi/iiop://yourserver:yourport/weblogic.management.mbeanservers.runtime

Anyway Sitescrotum UI is a POC (piece of crap), for instance if you enter the good JMX URL with the wrong weblogic username you still get "invalid URL" error...
frankly pathetic.

I could make Sitescrotum work only by adding
-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:30000:500:10

of course Sitescrotum doesn't give you any warning or error message...
also I didn't provide the optional domain name... also try hitting the "clear selection" and "reload configuration" many times...

Sitescrotum UI sucks!