Coherence cluster interference

####<Dec 28, 2011 12:32:51 PM CET> <Warning> <Coherence> <acme102> <osbdv1ms1> <Logger@1780819408 3.6.0.4> <<anonymous>> <> <0000JI4RpfkE8Tk5ozx0iY1Eyjr6000002> <1325071971843> <BEA-000000> <Oracle Coherence 3.6.0.4 (member=n/a): Received a discovery message that indicates the presence of an existing cluster that does not respond to join requests; this is usually caused by a network layer failure:
Message "SeniorMemberHeartbeat"
{
FromMember=Member(Id=1, Timestamp=2011-11-24 15:56:30.477, Address=13.53.53.189:7890, MachineId=14269, Location=process:26431, Role=OSB-node)
FromMessageId=0
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=17
ToPollId=0
Poll=null
Packets
{
[000]=Broadcast{PacketType=0x0DDF00D2, ToId=0, FromId=1, Direction=Incoming, ReceivedMillis=12:32:51.842, MessageType=17, ServiceId=0, MessagePartCount=1, MessagePartIndex=0, Body=0}
}
Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.6}
ToMemberSet=null
NotifySent=false

LastRecvTimestamp=none
MemberSet=MemberSet(Size=1, BitSetCount=1, ids=[1])
}>


https://forums.oracle.com/forums/thread.jspa?threadID=1068440


we need either to add

-Dtangosol.coherence.wka=localhost -Dtangosol.coherence.ttl=0 -Dtangosol.coherence.localhost=127.0.0.1

to the JVM parameters, or make sure each installation has a unique Coherence cluster multicast address


If you get forever this message:
"Delaying formation of a new cluster; waiting for well-known nodes to respond"

read here: http://docs.oracle.com/cd/E24290_01/coh.371/e22837/cluster_setup.htm#CEGGHCAB

then remove the wka option and use an explicit address:
-Dtangosol.coherence.clusteraddress=224.3.6.0 -Dtangosol.coherence.clusterport=3059

JPA using Coherence as Level 2 Cache

quite cool this video on TopLink grid with COH in WL Cluster.



In a nutshell:

• Coherence can cache DB data in a cluster-wide manner
• Grid Cache mode uses Coherence to cache (read-through) objects by ID
• Grid Read mode turns JPQL queries into COH Grid queries
• Grid Entity mode supports read and writes directly into the Cache

Grid Entity can be tricky, because updates are asynchronous and they could fail, leaving the cache out of sync with the DB.

Recommended model:
Out-Of-Process, COH is in a different JVM as the Application Server.

In-Process: everything in the same JVM. Good only for development.

Mixed model (some data in near-cache, some in far-cache) can be useful in some applications.



Second part:





it's quite cool, it shows how to create a COH cluster and server, how to configure a WL server for COH. It gives an example of a Grid Entity mode to offload DB writes.


Here the example
http://blogs.oracle.com/jeffwest/entry/introducing_the_oracle_parcel_serviceexamplereference_application


and the code is on SVN at https://svn.java.net/svn/oracle-pacel-svc~ops-r3-ee6

COOL!

Connecting to OSB Coherence cache cluster

cp $ORACLE_HOME/coherence_3.6/coherence.sh $ORACLE_HOME/coherence_3.6/cohosb.sh

vi $ORACLE_HOME/coherence_3.6/cohosb.sh

change JAVA_OPTS this way:

JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.distributed.localstorage=$STORAGE_ENABLED -Dtangosol.coherence.localhost=localhost -Dtangosol.coherence.localport=7890 -Dtangosol.coherence.clusteraddress=224.3.6.0 -Dtangosol.coherence.clusterport=36000 -Dtangosol.coherence.cluster=OSB-cluster -Dtangosol.coherence.wka=127.0.0.1 -Dtangosol.coherence.wka.port=7890"


I have managed to discover all these parameters by:

cp $ORACLE_HOME/oracle_common/modules/oracle.coherence_3.6/coherence.jar .
cp "$ORACLE_HOME/user_projects/domains/a087_osb/servers/AdminServer/tmp/_WL_user/ALSB Coherence Cache Provider/sm027g/APP-INF/lib/com.bea.alsb.coherence-impl.jar" .

jar xvf coherence.jar tangosol-coherence.xml
jar xvf coherence.jar tangosol-coherence-override-dev.xml
jar xvf com.bea.alsb.coherence-impl.jar tangosol-coherence-override.xml
cp $ORACLE_HOME/user_projects/domains/a087_osb/config/osb/coherence/osb-coherence-override.xml .


and examining their settings (the only one that really matters is the tangosol-coherence.xml)


Once you run it, you should see that it is the member number 2 of the OSB-cluster.


Here http://coherence.oracle.com/display/COH35UG/Command+Line+Setting+Override+Feature the list of all thee available command line properties

Coherence Cache viewer

googling around I find:
http://sourceforge.net/projects/sibyl/develop

http://www.sl.com/products/coherenceviewer.shtml (you must require evaluation)

http://sourceforge.net/projects/coherenceviewer/develop


A great Coherence expert, Michael, tells me:

I'm not sure if there is any built in functionality that can help with it. The easiest way is to join the cluster with Coherence Console (coherence.sh) and use 'maps' and 'list' commands.

Read this:
http://coherence.oracle.com/pages/viewpage.action?pageId=16684

and this:
http://mazanatti.info/index.php?/archives/65-Connecting-a-console-to-an-OSB-Coherence-cluster.html

Another way would be to implement very simple java app to browse the cache... that shouldn't be to complicated ;)

Coherence cache: read-through, write-through....

http://wiki.tangosol.com/display/COH33UG/Read-Through,+Write-Through,+Refresh-Ahead+and+Write-Behind+Caching



the presentation is very complete but slightly complicated...

DS=datasource.

in simple words:

read-through: if hit -> return; if miss -> fetch from DS, put in cache and return


refresh-ahead: tries to improve read performance by reloading from the DS the frequently requested data which are about to expire (better performance in reads)


write-through: upon put, store in cache, store in datasource and only then return (bad performance)


write-behind: upon put, store in cache and return immediately; you will update the DS later
(better performance in updates)


I have the sensation that cache synchronization problems can become really nasty when one tries to optimize stuff in a frequently updated cache... this is why you need Coherence experts, you can't improvise.

ALSB (OSB) Coherence Cache Provider

From OSB logs:



Creating WorkManager from "weblogic.wsee.mdb.DispatchPolicy" WorkManagerMBean for application "ALSB Coherence Cache Provider"

Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational configuration from resource "zip:/Oracle/Middleware/coherence_3.5/lib/coherence.jar!/tangosol-coherence.xml

Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational overrides from resource "zip:/Oracle/Middleware/user_projects/domains/OSBDomainBasic/servers/AdminServer/tmp/_WL_user/ALSB Coherence Cache Provider/sm027g/APP-INF/lib/com.bea.alsb.coherence-impl.jar!/tangosol-coherence-override.xml


Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational overrides from resource "file:/Oracle/Middleware/user_projects/domains/OSBDomainBasic/config/osb/coherence/osb-coherence-override.xml




Oracle Coherence 3.5.3/465p2 (member=n/a): Created a new cluster "OSB-cluster" with Member(Id=1, Timestamp=2010-09-04 19:00:46.957, Address=127.0.0.1:7890, MachineId=32896, Location=site:localdomain,machine:localhost,process:9285, Role=OSB-node, Edition=Grid Edition, Mode=Production, CpuCount=2, SocketCount=2) UID=0x7F0000010000012ADDB16DED80801ED2




So I gather that OSB internally uses the "ALSB Coherence Cache Provider" Enterprise Application, deployed as /Oracle/Middleware/Oracle_OSB1/lib/coherence.ear and targeted to the cluster.

This ear contains:

jar xvf coherence.ear

created: META-INF/
inflated: META-INF/MANIFEST.MF
inflated: META-INF/application.xml
created: APP-INF/
created: APP-INF/classes/
created: APP-INF/classes/com/
created: APP-INF/classes/com/bea/
created: APP-INF/classes/com/bea/alsb/
created: APP-INF/classes/com/bea/alsb/coherence/
created: APP-INF/classes/com/bea/alsb/coherence/init/
created: APP-INF/lib/
inflated: APP-INF/classes/com/bea/alsb/coherence/init/CoherenceAppListener$1.class
inflated: APP-INF/classes/com/bea/alsb/coherence/init/CoherenceAppListener.class
inflated: APP-INF/lib/com.bea.alsb.coherence-impl.jar
inflated: META-INF/weblogic-application.xml



the META-INF/weblogic-application.xml is revealing, basically it says "disregard any classes com.tangosol...." already loaded in the System classpath and load them in my EAR classloader. THIS IS EXTREMELY IMPORTANT! For multiple clusters to work in the same JVM, the coherence framework classes must be loaded in separate classloaders!

        com.bea.alsb.coherence.init.CoherenceAppListener


        com.tangosol.*

Here is the content of the "client-side" library jar, with the strategic tangosol-coherence-override.xml:


jar xvf com.bea.alsb.coherence-impl.jar
created: META-INF/
inflated: META-INF/MANIFEST.MF
created: com/
created: com/bea/
created: com/bea/alsb/
created: com/bea/alsb/coherence/
created: com/bea/alsb/coherence/impl/
created: com/bea/alsb/coherence/impl/messages/
inflated: com/bea/alsb/coherence/impl/messages/CoherenceLogLogger$MessageLoggerInitializer.class
inflated: com/bea/alsb/coherence/impl/messages/CoherenceLogLogger.class
inflated: com/bea/alsb/coherence/impl/messages/CoherenceLogLogLocalizer.properties
inflated: com/bea/alsb/coherence/impl/messages/CoherenceLogLogLocalizerDetail.properties
inflated: tangosol-coherence-override.xml
inflated: com/bea/alsb/coherence/impl/CacheValue.class
inflated: com/bea/alsb/coherence/impl/CoherenceCache$1.class
inflated: com/bea/alsb/coherence/impl/CoherenceCache$RemoveProcessor.class
inflated: com/bea/alsb/coherence/impl/CoherenceCache.class
inflated: com/bea/alsb/coherence/impl/CoherenceNotInstalledException.class
inflated: com/bea/alsb/coherence/impl/CoherenceProvider.class
inflated: com/bea/alsb/coherence/impl/CoherenceProviderFactory.class
inflated: com/bea/alsb/coherence/impl/ConfigurationException.class
inflated: com/bea/alsb/coherence/impl/OwnerValueExtractor.class



in this instance, tangosol-coherence-override.xml doesn't override the OOTB settings, it simply defines the cluster-name:

    
        
            OSB-cluster
            OSB-node
        
    
    
        jdk
        Oracle Coherence {version} (member={member}): {text}
    

And the Initializing class does this:

package com.bea.alsb.coherence.init;

import com.bea.alsb.coherence.CacheProviderManager;
import com.bea.alsb.coherence.impl.*;
import com.bea.alsb.coherence.init.messages.CoherenceLogLogger;
import java.util.logging.*;
import weblogic.application.ApplicationLifecycleEvent;
import weblogic.application.ApplicationLifecycleListener;
import weblogic.logging.LoggingHelper;

public class CoherenceAppListener extends ApplicationLifecycleListener
{

    public CoherenceAppListener()
    {
    }

    public void preStart(ApplicationLifecycleEvent evt)
    {
        try
        {
            redirectCoherenceLogger();
            com.bea.alsb.coherence.CacheProvider provider = CoherenceProviderFactory.newInstance("/coherence/osb-coherence-cache-config.xml");
            CacheProviderManager.initialize(provider);
            CoherenceLogLogger.logCacheProviderInitialized(provider);
        }
        catch(CoherenceNotInstalledException ex)
        {
            CacheProviderManager.reset();
            CoherenceLogLogger.raiseCoherenceNotInstalled();
        }
        catch(ConfigurationException ex)
        {
            CacheProviderManager.reset();
            CoherenceLogLogger.raiseInvalidConfiguration(ex);
        }
    }

    public void postStop(ApplicationLifecycleEvent applicationLifecycleEvent)
    {
        try
        {
            CacheProviderManager.reset();
            CoherenceLogLogger.logCacheProviderReset();
        }
        catch(Exception ex)
        {
            CoherenceLogLogger.raiseShutdownError(ex);
        }
    }

    private static void redirectCoherenceLogger()
    {
        Handler h = new Handler() {

            public void close()
                throws SecurityException
            {
            }

            public void flush()
            {
            }

            public void publish(LogRecord record)
            {
                if(serverLogger != null)
                    serverLogger.log(record);
            }

            Logger serverLogger;

           
            {
                serverLogger = LoggingHelper.getServerLogger();
            }
        }
;
        Logger coherenceLogger = Logger.getLogger("Coherence");
        coherenceLogger.setLevel(Level.ALL);
        coherenceLogger.addHandler(h);
        coherenceLogger.setUseParentHandlers(false);
    }

    private static final String CONFIG_FILE = "/coherence/osb-coherence-cache-config.xml";
    private static final String COHERENCE_LOGGER_NAME = "Coherence";
}

This osb-coherence-cache-config.xml is actually in $DOMAIN_HOME/config/osb/coherence/osb-coherence-cache-config.xml, this file was shown above.



Incidentally, the content of $DOMAIN_HOME/config/osb/coherence/osb-coherence-override.xml contains the network parameters for this cluster; it is evident that each cluster must provide its configuration:

        
        
                
                        
                
                    
127.0.0.1
7890
                
                        
            
127.0.0.1
7890
                
                
                        0
                
        

But, wait a second, what is this com.bea.alsb.coherence.CacheProviderManager ?

It turns out that CacheProviderManager is in /Oracle/Middleware/Oracle_OSB1/lib/modules/com.bea.alsb.coherence-api.jar. It's a basic singleton holder class for a SINGLE instance of CacheProvider - which means that in OSB only 1 OSB cache provider is supported. So if we want to access to our "custom " cache provider we must find another path: it's necessary to instantiate the tangosol classes in the same EAR (or WAR) where the client resides; or otherwise find a mechanism where we register the classloader of the CacheProvider and we retrieve from somewhere else.

Coherence tutorial

The Coherence tutorial starts here, it's really good:

http://download.oracle.com/docs/cd/E14526_01/coh.350/e14527/installcoh.htm

Here is a list of points and concepts to retain:

cache servers

define COHERENCE_HOME

cache-server.cmd

coherence.jar!/tangosol-coherence.xml
coherence.jar!/tangosol-coherence-override-dev.xml

custom-mbeans.xml

coherence.cmd = cache client
Map(?)
enter help
enter cache mycache

-Dtangosol.coherence.distributed.localstorage=true
-Dtangosol.coherence.cacheconfig=contacts-cache-config.xml




NamedCache interface
mycache is defined in coherence-cache-config.xml

in tangosol-coherence.xml you find tangosol.coherence.clusteraddress and port

NamedCache
CacheFactory
CacheService

PortableObject (POF: PofReader PofWriter, optimized serialization algorythm)



More on Coherence:
http://www.oracle.com/technology/pub/articles/vohra-coherence.html

replicated cache (fast read, slow write)
distributed (partitioned) cache

OSB, Coherence threads

I am getting familiar now with the new Threads which appear in a Thread Dump when the built-in Coherence clustering is activated:

Here are the Thread names... they are a LOT!

Logger@9257178 3.5.3/465p2 (com/tangosol/coherence/component/util/Daemon.onWait(Daemon.CDB:18))

PacketListener1 (com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20))

PacketListenerN (com/tangosol/coherence/component/net/socket/UdpSocket.receive(UdpSocket.CDB:20))


PacketReceiver (com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketReceiver.onWait(PacketReceiver.CDB:2))

PacketPublisher (com/tangosol/coherence/component/util/daemon/queueProcessor/packetProcessor/PacketPublisher.onWait(PacketPublisher.CDB:2))


PacketSpeaker (com/tangosol/coherence/component/util/queue/ConcurrentQueue.remove(ConcurrentQueue.CDB:7))


Cluster|Member(Id=1....) (com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:6))


TcpRingListener  (com/tangosol/coherence/component/net/socket/TcpSocketAccepter.accept(TcpSocketAccepter.CDB:18))


Invocation:Management:EventDispatcher (com/tangosol/coherence/component/util/daemon/queueProcessor/Service$EventDispatcher.onWait(Service.CDB:7))


Invocation:Management (com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:6))


DistributedCache:ORA-OSB-deployments  (com/tangosol/coherence/component/util/daemon/queueProcessor/service/Grid.onWait(Grid.CDB:6))

OSB and Coherence, a real life scenario with performance data

Here http://javatoolsforweblogic.googlecode.com/svn/trunk/OSBArtifacts/OSBCachingTests.jar you have a sample project that you can explore and play with. I also log if the cache was hit or missed.


Make sure you enable Caching in OSB: "Operations/Global Settings/Enable Result Caching".

You can configure caching only at Business Service level:

and make sure you extract the Cache Token (a unique ID for the request) from the payload.

Make sure you read this:

in order to test result caching, do not invoke the business service directly from the test console

You should then create a PS with a routing node pointing to the BS, and using inbound operation for outbound.

The configuration is:

GeoOsbServicePS is the actual service
GeoOsbServiceBS is the BS with caching enabled, and pointing to GeoOsbServicePS
GeoOsbServiceWithCachePS is simply a facade to expose the GeoOsbServiceBS

Test with cache:

Test without cache:

Using SoapUI I am injecting a 5 Threads load, first on the GeoOsbServicePS Proxy Service (no cache) and then on the GeoOsbServiceWithCachePS. These are the average response times measured:

GeoOsbServicePS : 47 ms
GeoOsbServiceWithCachePS : 13 ms

I was requesting over and over the same data.... for a more realistic test we should try requesting a variety of data.... the improvement of course will heavily depend on the hit/miss ratio.... if the overhead introduced by the extra PS and the caching layer is not justified by the percentage of hits, then performance will be deteriorated...


It is particurarly fascinating to observe the outbound message to the BS:

cache token in this case is the unique identifier extracted from the payload, and the TTL in cache is set to 30 minutes


<tran:cache-token xmlns:tran="http://www.bea.com/wli/sb/transports">2</tran:cache-token>
<tran:cache-ttl xmlns:tran="http://www.bea.com/wli/sb/transports">PT30M</tran:cache-ttl>

the outbound response contains:

<tran:cache-originated xmlns:tran="http://www.bea.com/wli/sb/transports">false</tran:cache-originated>

the first time, and for subsequent invocations it becomes true.


Which Cache Token Expression are you allowed to use? Not any user defined expression, but only one dependent on:
$body, $header, $operation or $metadata.

otherwise you will get a OSB Kernel BEA-398311 error
Cache Token expression is using invalid variable variableName.

The good thing is that you can use a single BS to cache the result of multiple operations on an external service, since the actual token used in Coherence will be made by your Cache Token + Operation + BS Identifier, so there is no chance that two different operations conflict with each other because they provide the same token.





When you start your OSB server, you should find this:


<13-06-2010 16:05:25 CEST> <Info> <Coherence> <BEA-000000> <Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational configuration from resource "zip:C:/bea11/coherence_3.5/lib/coherence.jar!/tangosol-coherence.xml">

<13-06-2010 16:05:25 CEST> <Info> <Coherence> <BEA-000000> <Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational overrides from resource "zip:C:/bea11/coherence_3.5/lib/coherence.jar!/tangosol-coherence-override-dev.xml">

<13-06-2010 16:05:25 CEST> <Info> <Coherence> <BEA-000000> <Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational overrides from resource "zip:C:/bea11/user_projects/domains/osbdomain/servers/AdminServer/tmp/_WL_user/ALSB Coherence Cache Provider/sm027g/APP-INF/lib/com.bea.alsb.coherence-impl.jar!/tangosol-coherence-override.xml">

<13-06-2010 16:05:25 CEST> <Info> <Coherence> <BEA-000000> <Oracle Coherence 3.5.3/465p2 (member=n/a): Loaded operational overrides from resource "file:/C:/bea11/user_projects/domains/osbdomain/config/osb/coherence/osb-coherence-override.xml">

<13-06-2010 16:05:26 CEST> <Warning> <Coherence> <BEA-000000> <Oracle Coherence 3.5.3/465p2 (member=n/a): Local address "127.0.0.1
" is a loopback address; this cluster node will not connect to nodes located on different machines>

<13-06-2010 16:05:29 CEST> <Info> <Coherence> <BEA-000000> <Oracle Coherence 3.5.3/465p2 (member=n/a): Created a new cluster "OSB-
cluster" with Member(Id=1, Timestamp=2010-06-13 16:05:26.258, Address=127.0.0.1:7890, MachineId=50080, Location=machine:localhost,
process:4468, Role=OSB-node, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0x7F0000010000012931A11172C3A0
1ED2>
<13-06-2010 16:05:30 CEST> <Info> <OSB Coherence WebLogic> <BEA-2032482> <The OSB Coherence Cache Provider has been initialized wi
th com.bea.alsb.coherence.impl.CoherenceProvider: SafeCluster: Name=OSB-cluster

WellKnownAddressList(Size=1,
WKA{Address=127.0.0.1, Port=7890}
)

MasterMemberSet
(
ThisMember=Member(Id=1, Timestamp=2010-06-13 16:05:26.258, Address=127.0.0.1:7890, MachineId=50080, Location=machine:localhost,p
rocess:4468, Role=OSB-node)
OldestMember=Member(Id=1, Timestamp=2010-06-13 16:05:26.258, Address=127.0.0.1:7890, MachineId=50080, Location=machine:localhost
,process:4468, Role=OSB-node)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-06-13 16:05:26.258, Address=127.0.0.1:7890, MachineId=50080, Location=machine:localhost,process:44
68, Role=OSB-node)
)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
)
)

Services
(
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=127.0.0.1:7890}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
).>



OSB: coherence and caching

this
http://www.oracle.com/technology/architect/enterprise_solution_cookbook/data_tier_caching_for_soa.html

makes a good reading just to get an idea of how to make a CacheService invoking Coherence. The service can be exposed with a LOCAL transport protocol to avoid marshalling/unmarshalling.

In a nutshell it says: you can cache either Service State (e.g. for a BPM the serialized state of a Process Instance) or Service Result (the result of a DB query).

Cache-aside means: 1) readFromCache 2) miss 3) retrieveFromDB 4) storeInCache
next time you read the same key, you have a hit.

This is very illuminating:

Oracle Coherence offers a Java API; a native service call from BPEL to Coherence was a preference over a standard web service interface. The performance overhead of invoking web service operations is several orders of magnitude larger than that of invoking native Java classes. That's because marshaling and un-marshaling XML, processing SOAP envelopes, etc. are expensive operations.

Note the use of

thread.setContextClassLoader(com.tangosol.net.NamedCache.class.getClassLoader());

to avoid creating a different cache per each classloader in the application! Well done!
(don't forget to set the classloader again to its previous value!)