Memory analysis on Linux

see also http://rimuhosting.com/howto/memory.jsp


to show the physical and swap memory:

free -m

total used free shared buffers cached
Mem: 7728 7131 597 0 377 1875
-/+ buffers/cache: 4878 2850
Swap: 32767 0 32767



to show the memory usage per process:

ps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
beasvbt 18204 0.1 6.9 981712 548220 ? Sl 2008 247:53 /opt/bea9/albpm6.0/j2eewl/jre/bin/java -Xms512m -Xmx512m
beasvbt 25100 0.0 11.1 1752064 884604 ? Ssl 2008 153:40 /opt/bea9/jdk1.5.0_14_amd64/bin/java -Dweblogic.Name=ofop
root 26350 0.0 0.2 20488 17052 ? S 2008 0:07 /opt/NAI/Linuxshield/libexec/scanner -e /opt/NAI/Linuxshi
beasvbt 28344 0.0 0.0 52776 1020 ? S Mar03 0:00 /bin/sh ./startWebLogic.sh
beasvbt 28345 0.0 0.0 52780 1160 ? S Mar03 0:00 /bin/sh /opt/bea9/user_projects/edu/ofop/ofop_bpm_edu/bin
beasvbt 28381 1.3 10.1 1122036 801376 ? Sl Mar03 1084:46 /opt/bea9/jdk1.5.0_14_amd64/bin/java -client -Xms256m -X



vmstat 1 2 shows you if you are swapping

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 1048 24096 332176 2739716 0 0 17 35 2 0 3 3 93 1
0 0 1048 24096 332180 2739716 0 0 0 60 1049 639 2 1 97 0

More on the deadlock, with code decompiled

these are the 3 threads:

this thread is waiting for the TransactionLogger to complete writing the transaction file:

"[ACTIVE] ExecuteThread: '402' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.transaction.internal.ResourceCheckpoint@4e5c2e7c WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
weblogic.transaction.internal.ResourceCheckpoint.blockingStore(ResourceCheckpoint.java:330)
weblogic.transaction.internal.XAResourceDescriptor.checkpointResourceDiscovery(XAResourceDescriptor.java:1102)
weblogic.transaction.internal.XAResourceDescriptor.refreshCheckpoint(XAResourceDescriptor.java:925)
weblogic.transaction.internal.ServerTransactionManagerImpl.checkpoint(ServerTransactionManagerImpl.java:1421)
weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1483)
weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:35)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)


synchronized void blockingStore(TransactionLogger transactionlogger)
{
transactionlogger.store(this);
try
{
if(!ioComplete)
wait();
}
catch(InterruptedException interruptedexception) { }
}




this thread is blocked waiting on a lock on a JDBCConnection, and there are 400 of these, all waiting on the same Connection:

"[ACTIVE] ExecuteThread: '401' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.jdbcx.base.BaseXAConnection@2923410e BLOCKED
weblogic.jdbcx.base.BaseXAResource.end(Unknown Source)
weblogic.jdbc.jta.DataSource.end(DataSource.java:803)
weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1232)
weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:404)
weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:326)
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1624)
weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2012)
weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3020)
weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:66)
weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3252)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)


public void end(Xid xid, int i)
throws XAException
{
synchronized(xaConnection)
{
checkXid(xid);
implXAResource.end(xid, i);
try
{
xaConnection.setGlobalTransaction(false);
if(implXAResource.requiredLocalAutoCommit gt 0)
xaConnection.unforceAutoCommitMode();
}
catch(SQLException sqlexception)
{
throw new XAException(sqlexception.getMessage());
}
}
}




"[STANDBY] ExecuteThread: '400' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.transaction.internal.ServerTransactionImpl@73937719 TIMED_WAITING
java.lang.Object.wait(Native Method)
weblogic.transaction.internal.ServerTransactionImpl.log(ServerTransactionImpl.java:1980)
weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2291)
weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:266)
weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:227)
weblogic.jms.backend.BEForwardingConsumer.processMessages(BEForwardingConsumer.java:478)
weblogic.jms.backend.BEForwardingConsumer.pushMessages(BEForwardingConsumer.java:285)
weblogic.messaging.util.DeliveryList.run(DeliveryList.java:256)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)


ServerTransactionImpl:
protected void log()
throws AbortRequestedException
{
label0:
{
synchronized(this)
{
if(!isCancelledUnsync())
break label0;
}
return;
}

WLST quirk with getServerRuntimes()

this:

connect(username,password,url)

domainRuntime()

cmo.getServerRuntimes()

will not work! Don't ask me why.

use instead:

domainRuntimeService.getServerRuntimes()

domainRuntimeService is a WLST predefined variable.
Strange, uh?