Unicast cluster tuning

there are 2 properties you can set:

weblogic.cluster.unicastHeartbeatTimeoutMillis (default 15000)
weblogic.cluster.unicastDiscoveryPeriodMillis (default 3000)

If you get a message like:
add Leader BLA to unreliableSet

(you get it only if UnicastMessaging debug flag is enabled)


and your cluster is divided in multiple groups each one with its own Group Leader, and this is annoying you,
you MIGHT consider changing the timeout value (do it at your own risk, side effects are still unknown by the author of this blog!).

In fact unicastHeartbeatTimeoutMillis determines how long after the last heartbeat (LAT=Last Arrival Time) the "slave" should declare that the Group Leader is unrealiable.


The cluster's GroupLeader sends heartbeat messages every LeaderHeartbeatPeriod seconds:

http://download.oracle.com/docs/cd/E11035_01/wls100/schemaref/security/http.www.bea.com.ns.weblogic.920.domain/types/database-less-leasing-basistype.leader-heartbeat-period.html

default value 10.


to view the value of the heartbeat period:



wlst.sh

connect('weblogic', 'welcome1', 't3://localhost:7011')
cd(‘/Clusters/myCluster/DatabaseLessLeasingBasis/myCluster’)
cmo.getLeaderHeartbeatPeriod()

and it’s 10



and to set it I tried:

cmo.setLeaderHeartbeatPeriod(20)

but I get

java.lang.RuntimeException: The requested attribute is not exposed through JMX: setLeaderHeartbeatPeriod

Caused by: javax.management.AttributeNotFoundException: Attribute is readonly. : com.bea:Name=myCluster,Type=DatabaseLessLeasingBasis,Cluster=myCluster:LeaderHeartbeatPeriod

So I guess you can only set it by hacking the the config.xml – I don’t see it in the WebLogic console.

weblogic.unicast.HttpPing in Unicast clusters

If you have the impression that your Cluster Members are vanishing from the cluster and have difficulty joining it back, read this:

http://download.oracle.com/docs/cd/E21764_01/doc.1111/e15731/weblogic_server_issues.htm#CIHEAFDH

9.4.1 Threads Are Blocked on Cluster Messaging in Unicast Mode

When using Unicast mode for cluster communication, many threads are blocked on cluster messaging, which may result in cluster members having difficulty sending heartbeat messages. In this situation, some cluster members drop out from the cluster and may take some time to rejoin the cluster.

Workaround

Set the following system property to resolve this issue:

-Dweblogic.unicast.HttpPing=true



If any member of the cluster is down, you will see:

####<21-Aug-2011 12:36:30 o'clock BST> <debug> <unicastmessaging> <pierrepc> <ms1> <Timer-2> <<anonymous>> <> <> <1313926590111> <BEA-000000> <[HttpPingRoutine] HttpPing Caught IOException: java.net.SocketException: Socket Closed>


These messages follow the ping attempts:

####<21-Aug-2011 12:39:27 o'clock BST> <Debug> <UnicastMessaging> <pierrepc> <ms1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1313926767511> <BEA-000000> <[UnicastFragmentSocket] sending ' server ms1, id=1313926767511' to local group>
####<21-Aug-2011 12:39:27 o'clock BST> <Debug> <UnicastMessaging> <pierrepc> <ms1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1313926767511> <BEA-000000> <[Group] [LocalGroup [[ms1]]] we are the seniormost. Send message to group>
####<21-Aug-2011 12:39:27 o'clock BST> <Debug> <UnicastMessaging> <pierrepc> <ms1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1313926767512> <BEA-000000> <[Group] [LocalGroup [[ms1]]] we are the seniormost. Send message to group>

The SocketException will go away once you start all members of the cluster.


if you get this:


####<21-Aug-2011 13:11:59 o'clock BST> <Warning> <Socket> <pierrepc> <ms1> <ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1313928719011> <BEA-000442> <Connection attempt was rejected because the incoming protocol is not enabled on channel "unicastChannel".>

here is what they say : enable the protocol on the channel



And if you get this:


####<21-Aug-2011 13:12:05 o'clock BST> <Debug> <UnicastMessaging> <pierrepc> <ms1> <Timer-2> <<anonymous>> <> <> <1313928725736> <BEA-000000> <[HttpPingRoutine] HttpPing Caught IOException: java.io.EOFException: Response contained no data>


then you might try this (as suggested here): -Dhttp.keepAlive=false (it didn't work for me :o( )

Here http://download.oracle.com/docs/cd/E12840_01/wls/docs103/issues/known_resolved.html they mention that in pre 10.3 versions of WebLogic the cluster gets in trouble when you suspend one of the Managed Servers. This is known as CR370084, yet the workaround of setting -Dweblogic.unicast.HttpPing=true is recommended only for test environments.