Showing posts with label OracleRAC. Show all posts
Showing posts with label OracleRAC. Show all posts

Thursday, November 12, 2015

SCAN and REDIRECT with Oracle RAC

If you access a ORACLE RAC, be aware that you must open a firewall rule not only for the SCAN listener, but also to all the RAC instances. Connections are NOT routed via the SCAN listeners, they are merely client-side redirected.

Failure to do so will result in the connection fail with a Socket Timeout Exception. You will be puzzled because the telnet to the listener works... the problem is that your JDBC driver is trying to connect ALSO to the instance on a different IP (possibly also PORT...).... so you should test with telnet also the RAC instance!

import java.sql.DriverManager;
import java.sql.SQLException;

public class DBPing {
        public static void main(String[] args) throws ClassNotFoundException, SQLException {
                Class.forName("oracle.jdbc.driver.OracleDriver");
                System.out.println("length " + args.length);
                String user = args[0];
                String password = args[1];
                String url = args[2];
                String now = new java.util.Date().toString();
                System.out.println(now + " user= " + user + " password=" + password + " url=" + url);
                java.sql.Connection conn = DriverManager.getConnection(url, user, password);
                System.out.println("ok");
        }
}


java -cp /acme/appsrv/bin/wl12.1a/oracle_common/modules/oracle.jdbc_12.1.0/ojdbc6.jar:/home/userpippo DBPing dbusername dbpassword "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbhostname-scan)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservicename)))" &^

The program will wait for connection for a while, then timeout. If you netstat BEFORE the timeout:

netstat -an | grep 1522

you will discover which IP corresponds to the RAC INSTANCE:

tcp 0 1 10.11.12.13:59482 44.55.66.222:1522 SYN_SENT

What is 44.55.66.222 ? It's NOT the dbhostname-scan IP, but rather the IP of a RAC instance.... the SCAN listener tells the Oracle JDBC driver to do a redirect to a new IP/PORT where the RAC instance is running.

What is SYN_SENT ? http://serverfault.com/questions/328361/tcptrack-shows-syn-sent-connections-does-that-mean-the-syn-package-reached-the .... which means that the JDBC driver sent a SYN, but the server didn't reply (because the firewall blocked the communication).

Here some doc on SCAN.
See also Oracle Support note "Port 1521 Open on Firewall But Unable to Connect Due to Errors: ORA-12535,TNS-12203 (Doc ID 361284.1)"


Tuesday, October 20, 2015

ORA-24756: transaction does not exist

Here is the stacktrace:

java.sql.SQLException: 
ORA-24756: transaction does not exist at 
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450) at 
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:392) at 
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:385) at 
oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:1018) at 
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522) at 
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257) at 
oracle.jdbc.driver.T4CTTIOtxen.doOTXEN(T4CTTIOtxen.java:166) at 
oracle.jdbc.driver.T4CXAResource.doTransaction(T4CXAResource.java:757) at 
oracle.jdbc.driver.T4CXAResource.doPrepare(T4CXAResource.java:518) at 
oracle.jdbc.xa.client.OracleXAResource.prepare(OracleXAResource.java:713)


If you get this error, you must be using Oracle RAC and XA.

You must carefully craft the various timeouts in WebLogic and in Oracle DB.

See Oracle note "Troubleshooting ORA-24756 while Running an XA Program or MSDTC with the Oracle RDBMS (Doc ID 1076242.6)". THE rule is: global transaction timeout < session timeout < distributed_lock_timeout . See also "Recommended Timeout Configuration When Using an External Transaction Processing Monitor and XA with Oracle (Doc ID 338880.1)" .

Each MDB has a specific timeout, but this should not be involved in the issue (this in the weblogic.ejb.jar.xml DD https://docs.oracle.com/cd/E13222_01/wls/docs103/ejb/DDreference-ejb-jar.html )

<transaction-descriptor>
 <trans-timeout-seconds>2400</trans-timeout-seconds>
 </transaction-descriptor>


So you should end up with this configuration in config.xml and the "$XA_DATASOURCE"jdbc.xml

> config/jdbc/<datasource>-jdbc.xml
 <jdbc-xa-params>
 <keep-xa-conn-till-tx-complete>true</keep-xa-conn-till-tx-complete>
 <xa-end-only-once>true</xa-end-only-once>
 <xa-set-transaction-timeout>true</xa-set-transaction-timeout>
 <xa-transaction-timeout>3300</xa-transaction-timeout>
 <xa-retry-duration-seconds>3600</xa-retry-duration-seconds>
 </jdbc-xa-params>

> config/config.xml
 <jta>
 <timeout-seconds>3000</timeout-seconds>
 <forget-heuristics>false</forget-heuristics>
 <max-resource-requests-on-server>1000</max-resource-requests-on-server>
 </jta>




Wednesday, October 2, 2013

oracle.repackaged.ucp.jdbc.oracle.RACCallbackGuard

today we saw this in the logs:

####<Oct 2, 2013 8:47:23 AM CEST> <Warning>
 <oracle.repackaged.ucp.jdbc.oracle.RACCallbackGuard> <acme111>
 <osbpr1ms4> <Thread-100> <<anonymous>> <> 
<0000K5o7QgvE4Uk5ozS4yY1IIgWK000003> <1380696443473> <BEA-000000> 
<RAC callback: guarded method threw exception>



no idea what happened.... it could be a RAC instance dead and TX being switched to another instance... still under investigation...

Ok, what happened is that the DBA team started the second node of the RAC cluster, and the JDBCDrivers somehow got the notification, but something must have gone wrong.


Monday, May 27, 2013

BEA-000150, Server failed to get a connection to the database in the past 30 seconds for lease renewal. Server will shut itself down.

We are using a Database-based lease renewal for automated server migration (I personally think it's a bad idea), we occasionally (once a month) have a server restarting with

BEA-000150 Server failed to get a connection to the database in the past 30 seconds for lease renewal. Server will shut itself down.



An Oracle DOC "With Database Leasing Server Migration WebLogic Servers Part Of Cluster Restarting Intermittently With Unable Renew Lease Errors [ID 1550164.1]" suggests that we should check for time skew in the different RAC nodes, which makes perfectly sense. However this is not our case. We shall keep investigating.

Monday, April 2, 2012

Connecting to a RAC instance with Oracle SQL Developer

New Connection
Connection Type: Advanced
in Custom JDBC URL enter:


jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)(FAILOVER=ON)(ENABLE=BROKEN)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=myhost1-vip.acme.com)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=myhost2-vip.acme.com)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=srv_osb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(DELAY=5)(RETRIES=36))))



and.... it works! No need to set any special classpath.

WebLogic 11g and Oracle RAC with GridLink DataSource

At the origin there was the OCI Driver, natively supporting TAF (transparent application failover)

In the old days people used to use MultiDataSources to connect to RAC.

This had several disadvantages which are explained in the links below.

The new way seems to be the GridLink data source.

Overall description here:

http://docs.oracle.com/cd/E17904_01/web.1111/e13737/gridlink_datasources.htm#JDBCA373

Detailed configuration instructions here:
http://www.oracle.com/technetwork/middleware/weblogic/wls-jdbc-gridlink-howto-333331.html

When creating a GridLink DS, this page is a bit puzzling:




ONS means Oracle Notification Service, which is part of the Oracle DB installation (NOT of WebLogic).

FAN means Fast Application Notification

Tuesday, March 31, 2009

Weblogic and Oracle RAC

The suggested solution for high availability is using a Multi Data Source with load balancing.

It turns out that Oracle RAC is very sensitive to the version of JDBC driver you are using:


Automatic database connection failover and load balancing with global transactions (XA) in a highly-available (HA) DBMS architecture is supported with the following Oracle RAC versions and drivers:

Oracle Thin/XA Driver 10g

(which is oracle.jdbc.xa.client.OracleXADataSource, versus weblogic.jdbcx.oracle.OracleDataSource which is the Weblogic Oracle driver)

If you try using the Weblogic Type 4 JDBC driver (weblogic.jdbcx.oracle.OracleDataSource) you will encounter deadlock problems:

"[ACTIVE] ExecuteThread: '398' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.jdbcx.base.BaseXAConnection@2923410e BLOCKED
weblogic.jdbcx.base.BaseXAResource.end(Unknown Source)
weblogic.jdbc.jta.DataSource.end(DataSource.java:803)
weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1232)
weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:404)
weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:326)
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1624)
weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2012)
weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3020)
weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:66)
weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3252)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)


Highly recommended is this:

http://e-docs.bea.com/wls/docs100/jdbc_admin/oracle_rac.html

More specifically for ALBPM:
http://edocs.bea.com/albsi/docs60/installguide/index.html?t=installation/c_Required_Databases.html