Sunday, April 29, 2012

Tomcat, WebApplication using a DataSource

http://wiki.apache.org/tomcat/UsingDataSources

http://tomcat.apache.org/tomcat-5.5-doc/jndi-datasource-examples-howto.html

copy ojdbc5.jar to $TOMCAT_HOME/lib directory

create your "cmdb" webapp folder under $TOMCAT_HOME/webapps/cmdb

create $TOMCAT_HOME/webapps/cmdb/META-INF/context.xml and put this in it:

<?xml version="1.0" encoding="UTF-8"?>

<Context>

 <Resource name="jdbc/cmdb" auth="Container"
   type="javax.sql.DataSource" driverClassName="oracle.jdbc.OracleDriver"
   url="jdbc:oracle:thin:@127.0.0.1:1521:xe"
   username="cmdb" password="cmdb" maxActive="20" maxIdle="10"
   maxWait="-1"/>
</Context>


in WEB-INF/web.xml put


<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
    version="2.4">
  <description>CMDB</description>
  <resource-ref>
      <description>DB Connection</description>
      <res-ref-name>jdbc/cmdb</res-ref-name>
      <res-type>javax.sql.DataSource</res-type>
      <res-auth>Container</res-auth>
  </resource-ref>
</web-app>



in WEB-INF\lib put standard.jar and jstl.jar

and your test.jsp is

<%@ taglib uri="http://java.sun.com/jsp/jstl/sql" prefix="sql" %>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>

<sql:query var="rs" dataSource="jdbc/cmdb">
select JMSMODULENAME from NESOA2_JMSMODULES
</sql:query>

<html>
  <head>
    <title>Machines</title>
  </head>
  <body>

  <h2>Results</h2>
  
<c:forEach var="row" items="${rs.rows}">
    JMSMODULENAME ${row.JMSMODULENAME}<br/>
</c:forEach>

  </body>
</html>


If you get errors like

"javax.servlet.jsp.JspException: Unable to get connection, DataSource invalid: "org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)""

and "tnsping" works, it's very possible that you need to ipconfig /renew or something like that...


On another instance of Tomcat, the same WebApp works only if I add

<Context path="/cmdb">
     <ResourceLink name="jdbc/cmdb" type="javax.sql.DataSource" global="jdbc/cmdb"/>
</Context>


Saturday, April 28, 2012

Sending and receiving emails in WLST / Python

http://docs.python.org/library/smtplib.html

import smtplib
s = smtplib.SMTP('smtp.acme.com')
#SMTP.sendmail(from_addr, to_addrs, msg[, mail_options, rcpt_options])
s.sendmail('pippo@gmail.com', 'pluto@gmail.com', 'ciao come stai?')
s.quit()

s.login('pierluigi.vernetto@acme.com', 'mypw')

Traceback (innermost last):
File "", line 1, in ?
File "/opt/oracle/fmw11_1_1_5/wlserver_10.3/common/wlst/modules/jython-modules.jar/Lib/smtplib.py", line 540, in login
SMTPException: SMTP AUTH extension not supported by server.

s.connect()



see
http://www.oracle.com/technetwork/java/javamail/index.html

WLST to delete JMS messages from queues, based on Selector

from javax.naming import Context, InitialContext
from weblogic.jndi import WLInitialContextFactory
from javax.jms import QueueSession, Queue, Session

msservers=["myhost1:8001", "myhost2:8001", "myhost3:8001", "myhost4:8001"]
jmsservers=["CommonJmsServer1", "CommonJmsServer2", "CommonJmsServer3", "CommonJmsServer4"]

jmsQueueJNDIs=["jms.jndi.dq.AQ", "jms.jndi.dq.BQ"];

for i in range(len(msservers)):
    msserver = msservers[i] 
    jmsserver = jmsservers[i]
    properties = Properties()
    properties[Context.PROVIDER_URL] = "t3://" + msserver
    properties[Context.INITIAL_CONTEXT_FACTORY] = WLInitialContextFactory.name
    print "connecting to " + msserver + "..."
    ctx = InitialContext(properties)
    print "successfully connected to ", msserver
    
    connectionFactory = ctx.lookup("weblogic/jms/XAConnectionFactory" )
    queueCon = connectionFactory.createQueueConnection();
    queueCon.start()
    queueSession = queueCon.createQueueSession( false, Session.AUTO_ACKNOWLEDGE );
    for jmsqueue in jmsQueueJNDIs:
        theJNDI = jmsserver + "@" + jmsqueue
        queue = ctx.lookup(theJNDI)
        
        queueReceiver = queueSession.createReceiver(queue) # "ISTEST='true'"
        condition = True
        print "deleting messages from " + theJNDI
        while (condition):
            message = queueReceiver.receiveNoWait()
            if (message != None):
                print 'ack on message'
                message.acknowledge();
            condition = (message != None)
          





See also


http://docs.oracle.com/javaee/6/api/javax/jms/Message.html
http://docs.oracle.com/javaee/6/api/javax/jms/Queue.html
http://docs.oracle.com/javaee/6/api/javax/jms/Session.html
http://docs.oracle.com/javaee/6/api/javax/jms/QueueSession.html
http://docs.oracle.com/javaee/6/api/javax/jms/MessageConsumer.html

Testing Physical Destinations belonging to Distributed Queues

I need to send a JMS message to a specific Queue part of a Distributed Queue.
If I send a message to a Distributed Queue, I have no control on which specific instance of the Queue, on which JMSServer, the message will go.

When you target a Distributed Queue "MyQ" to a cluster, this gets created in the JNDI tree of each Managed Server in the cluster:

The Distributed Queue:
Class: weblogic.jms.common.DistributedDestinationImpl
Binding Name: jms.jndi.dq.MyQ

The actual instance in the JMSServer running in the Managed Server:
Class: weblogic.jms.common.DestinationImpl
Binding Name = CommonJmsServer1@jms.jndi.dq.MyQ


So, you must bind to "CommonJmsServer1@jms.jndi.dq.MyQ" to send to only that JMSServer.

Example of WLST script to send to multiple physical destinations:

from javax.naming import Context, InitialContext
from weblogic.jndi import WLInitialContextFactory
from javax.jms import QueueSession, Queue, Session

msservers=["myhost1:8001", "myhost2:8001", "myhost3:8001", "myhost4:8001"]
jmsservers=["CommonJmsServer1", "CommonJmsServer2", "CommonJmsServer3", "CommonJmsServer4"]

jmsQueueJNDIs=["jms.jndi.dq.AQ", "jms.jndi.dq.BQ"];

for i in range(len(msservers)):
    msserver = msservers[i] 
    jmsserver = jmsservers[i]
    properties = Properties()
    properties[Context.PROVIDER_URL] = "t3://" + msserver
    properties[Context.INITIAL_CONTEXT_FACTORY] = WLInitialContextFactory.name
    print "connecting to " + msserver + "..."
    ctx = InitialContext(properties)
    print "successfully connected to ", msserver
    
    connectionFactory = ctx.lookup("weblogic/jms/XAConnectionFactory" )
    queueCon = connectionFactory.createQueueConnection();
    queueSession = queueCon.createQueueSession( false, Session.AUTO_ACKNOWLEDGE );
    for jmsqueue in jmsQueueJNDIs:
        theJNDI = jmsserver + "@" + jmsqueue
        queue = ctx.lookup(theJNDI)
        sender = queueSession.createSender( queue );
        
        msg = queueSession.createTextMessage( "I am a pig" );
        msg.setStringProperty("ISTEST", "true");
        print "sending message to " + theJNDI
        sender.send( msg );



See also:
http://www.javamonamour.org/2011/09/creating-durable-subscribers-for.html

Setting up HAFileAdapter with Oracle RAC DB

Create these DataSources:

the first is :

dsname=SOADataSource (local, non XA)
dsjndi=jdbc/SOALocalTxDataSource
dstype=GridLink
dsSupportsGlobalTransactions=true
dsXAType=OnePhaseCommit
dsDriverName=oracle.jdbc.OracleDriver
dsFanEnabled=true
dsOnsNodeList=myrachost.com:6200

<?xml version='1.0' encoding='UTF-8'?>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/jdbc-data-source http://xmlns.oracle.com/weblogic/jdbc-data-source/1.0/jdbc-data-source.xsd">
  <name>SOADataSource</name>
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=myrachost)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=srv_osb)))</url>
    <driver-name>oracle.jdbc.OracleDriver</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>pp1_soainfra</value>
      </property>
    </properties>
    <password-encrypted>{AES}rnUdIazVyMYKqcFXg55eXdPGyxefK6Mr9fqUglWFhm8=</password-encrypted>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1  FROM DUAL</test-table-name>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbc/SOALocalTxDataSource</jndi-name>
  </jdbc-data-source-params>
  <jdbc-oracle-params>
    <fan-enabled>true</fan-enabled>
    <ons-node-list>myrachost:6200</ons-node-list>
    <ons-wallet-file></ons-wallet-file>
  </jdbc-oracle-params>
</jdbc-data-source>


and the second is:

dsname=SOAXADataSource (global (XA))
dsjndi= jdbc/SOADataSource
dstype=GridLink
dsXAType=TwoPhaseCommit
dsDriverName=oracle.jdbc.xa.client.OracleXADataSource
dsFanEnabled=true
dsOnsNodeList=myrachost.com:6200

<?xml version='1.0' encoding='UTF-8'?>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/jdbc-data-source http://xmlns.oracle.com/weblogic/jdbc-data-source/1.0/jdbc-data-source.xsd">
  <name>SOAXADataSource</name>
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=myrachost)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=srv_osb)))</url>
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>pp1_soainfra</value>
      </property>
    </properties>
    <password-encrypted>{AES}LjuMmp4BvBf23878ydOfK2XcBMyZTjGDf2lP1NTn2raU=</password-encrypted>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1  FROM DUAL</test-table-name>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbc/SOADataSource</jndi-name>
    <global-transactions-protocol>TwoPhaseCommit</global-transactions-protocol>
  </jdbc-data-source-params>
  <jdbc-oracle-params>
    <fan-enabled>true</fan-enabled>
    <ons-node-list>myrachost:6200</ons-node-list>
    <ons-wallet-file></ons-wallet-file>
  </jdbc-oracle-params>
</jdbc-data-source>




now you can move to the eis/HAFileAdapter and configure it like this:

controlDir=/opt/oracle/domains/myydomain/shared/apps/fileadapter/controlDir (this is on a shared drive)

inboundDataSource=jdbc/SOADataSource
outboundDataSource=jdbc/SOADataSource
outboundDataSourceLocal=jdbc/SOALocalTxDataSource
outboundLockTypeForWrite=oracle


The mutex - allowing the HAFileAdapters to coordinate with each other and avoid picking up the same file - is based on 2 DB tables:

describe FILEADAPTER_IN
Name Null Type
------------------ -------- --------------
FULL_PATH NOT NULL VARCHAR2(4000)
ROOT_DIRECTORY NOT NULL VARCHAR2(3000)
FILE_DIRECTORY NOT NULL VARCHAR2(3000)
FILE_NAME NOT NULL VARCHAR2(1000)
FILE_ENDPOINT_GUID NOT NULL VARCHAR2(2000)
FILE_LAST_MODIFIED NUMBER
FILE_READONLY CHAR(1)
FILE_PROCESSED CHAR(1)
CREATED NOT NULL NUMBER
UPDATED NUMBER


describe FILEADAPTER_MUTEX
Name Null Type
------------------ -------- --------------
MUTEX_ID NOT NULL VARCHAR2(4000)
MUTEX_CREATED TIMESTAMP(6)
MUTEX_LAST_UPDATED TIMESTAMP(6)
MUTEX_SEQUENCE NUMBER



It's all, folks! Don't forget to test!

See also here for first approach

WebLogic TemporaryQueue0 JMS Destination

https://forums.oracle.com/forums/thread.jspa?threadID=1115400

apparently this queue is for "weblogic internal use only", so do not bother to monitor it...

Xlib: PuTTY X11 proxy: wrong authorisation protocol attempted

I log into myserver as user “myuser”, then “sudo su – soa”; I have X11 forwarding enable on Putty, and XMing running on my laptop.
I run
/opt/oracle/java/bin/jvisualvm
and I get:

Xlib: connection to "localhost:10.0" refused by server
Xlib: PuTTY X11 proxy: wrong authorisation protocol attempted

googling for the message I find:

http://froebe.net/blog/2008/11/14/getting-xlib-putty-x11-proxy-wrong-authentication-protocol-attempted-i-have-the-answer/

I try to
cat /etc/ssh/sshd_config
but I get
cat: /etc/ssh/sshd_config: Permission denied

I go back as myuser and I do
xauth list
myserver/unix:10 MIT-MAGIC-COOKIE-1 8ea792a81074c62518d53789faf9558d

then I “sudo su – soa” again and I do
xauth add myserver/unix:10 MIT-MAGIC-COOKIE-1 8ea792a81074c62518d53789faf9558d
xauth: creating new authority file /home/soa/.Xauthority

I run again
/opt/oracle/java/bin/jvisualvm

and it works! Thank you Jason!!!

Useful links:

For info on .Xauthority

Sample EntityManager usage in Java and Python WLST

In Eclipse, create a JDBC Connection
show view Data Source Explorer
new Connection Profile (enter parameters for your schema)

New JPA project
JPA Tools, Generate Entities from Tables

you should have a persistence.xml which contains

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
 <persistence-unit name="ToplinkTest" transaction-type="RESOURCE_LOCAL">
  <class>com.osb.reporting.WliQsReportAttribute</class>
  <class>com.osb.reporting.WliQsReportData</class>
  <properties>
   <property name="javax.persistence.jdbc.url" value="jdbc:oracle:thin:@localhost:1521:xe"/>
   <property name="javax.persistence.jdbc.user" value="DEV_SOAINFRA"/>
   <property name="javax.persistence.jdbc.password" value="DEV_SOAINFRA"/>
   <property name="javax.persistence.jdbc.driver" value="oracle.jdbc.OracleDriver"/>
  </properties>
 </persistence-unit>
</persistence>



You can now code the client:

package com.osb.reporting.client;

import java.util.List;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
import javax.persistence.Query;

import com.osb.reporting.WliQsReportAttribute;

public class ReportingClient {
 public static void main(String[] args) throws ClassNotFoundException {
  EntityManagerFactory emf = Persistence.createEntityManagerFactory("ToplinkTest");
  EntityManager em = emf.createEntityManager();
  WliQsReportAttribute result = em.find(WliQsReportAttribute.class, "uuid:0e6fb58fc7c49289:5378b92d:134416bc5f5:-7fbe");
  System.out.println(result.getMsgLabels());
  Query fa = em.createQuery("select a from WliQsReportAttribute a");
  List res = fa.getResultList();
  for (Object el : res) {
   System.out.println( ((WliQsReportAttribute)el).getMsgLabels());
  }
 }
}



you will need to add the JDBC driver to the classpath, like
C:\bea1035\wlserver_10.3\server\ext\jdbc\oracle\11g\ojdbc5_g.jar

In WLST, just add to the classpath the location from which you can access classes and META-INF (where persistence.xml is located).... like the build\classes directory.


package com.osb.reporting.client;

from javax.persistence import *
emf = Persistence.createEntityManagerFactory("CMBDModeling")
em = emf.createEntityManager()
q = em.createQuery("select envs from Nesoa2Env envs")
res = q.getResultList()
for i in res:
   print i.envname

so the message is: you can easily use JPA in WLST and get rid of all that horrible pain of reading property files, by accessing directly a DB

Friday, April 27, 2012

Bill of rights of technical human resources

Meetings

A meeting must have objectives. All relevant information should be circulated in advance to all participants of the meeting.
In every meeting there should be a minute taker, minutes should be circulated after the meeting. The meeting is not over
until all attendees have expressed their opinions, and all the objectives have been reached. Otherwise, pending points should be rescheduled for another meeting.




A meeting should not be called to read an email to all attendees. Email should be sent in advance, then eventually ask people what they think.

All management stuff should be discussed in management meeting ,to which technical resources do not need to attend.

Meetings are meant for people to exchange information and reach agreement,
they should not be the monopoly of a single people who takes all the bandwidth and prevents communication amongst other attendees.
As a rule of thumb, in a meeting with 10 people, no more than 30% of the time should be allocated to a single attendee.
If a single person talks for more than 3 minutes in a row in a meeting, this is a communication antipattern.

Long lasting verbal communication should be avoided, and if really needed backed up by schemas, writing etc. Not all people have the same receptivity
to verbal communication, and visual communication is to be preferred over pure verbal conceptualization.



Emails
These emails should be concise, and with precise topics. Complex stuff should not be treated in emails, they have invented voice conversations for that.


Tasks

It’s the management responsibility to identify tasks – which should be as much as possible atomic –prioritize them, and assign them to the technical resources.
Technical people can help in this process, but “task management” is strictly the management responsibility.
It’s the duty of the technical resource to provide regular (once a day, your mileage might vary) progress report to management.
Management has always the right to review priorities and reassign tasks, but bearing in mind that in general this should be avoided.
Excessive task switching is a known factor of burnout and poor work quality.



WLST to send a JMS message

Here the Java version.

from javax.naming import Context, InitialContext
from weblogic.jndi import WLInitialContextFactory
from javax.jms import QueueSession, Queue, Session

properties = Properties()
properties[Context.PROVIDER_URL] = "t3://localhost:7001"
properties[Context.INITIAL_CONTEXT_FACTORY] = WLInitialContextFactory.name
ctx = InitialContext(properties)

connectionFactory = ctx.lookup("weblogic/jms/XAConnectionFactory" )
queueCon = connectionFactory.createQueueConnection();
queueSession = queueCon.createQueueSession( false, Session.AUTO_ACKNOWLEDGE );
queue = ctx.lookup( "jms/testme")
sender = queueSession.createSender( queue );

msg = queueSession.createTextMessage( "I am a pig" );

sender.send( msg );


If you want to resubmit to a JMS queue all the files in a folder, use this:

from javax.naming import Context, InitialContext
from weblogic.jndi import WLInitialContextFactory
from javax.jms import QueueSession, Queue, Session
import os

cfjndi="PVOSBTESTCF"
queuejndi="PV_OSB_TESTQ"
PROVIDER_URL="t3://acme.com:10001,acme.com:10003"
filespath="/opt/oracle/scripts/retrieveErrors/thefilesfailed"

properties = Properties()
properties[Context.PROVIDER_URL] = PROVIDER_URL
properties[Context.INITIAL_CONTEXT_FACTORY] = WLInitialContextFactory.name
properties[Context.SECURITY_PRINCIPAL] = "weblogic"
properties[Context.SECURITY_CREDENTIALS] = "mypassword"
ctx = InitialContext(properties)

connectionFactory = ctx.lookup(cfjndi)
queueCon = connectionFactory.createQueueConnection();
queueSession = queueCon.createQueueSession( false, Session.AUTO_ACKNOWLEDGE );
queue = ctx.lookup(queuejndi)
sender = queueSession.createSender( queue );

for file in os.listdir(filespath):
    current_file = os.path.join(filespath, file)
    data = open(current_file, "rb")
    msg = queueSession.createTextMessage( data.read());
    print "processing ", current_file
    sender.send( msg );
    print "done processing ", current_file





Error untargetting and targetting again a JMSServer

We had an interesting case where WebLogic was keeping in JNDI tree a reference to a JMS Connection Factory (CF) even after untargetting the JMSServer on which the CF was deployed.
When I retarget the CF, I get this error:


An error occurred during activation of changes, please see the log for details.
Message icon - Error Error preparing connection factory GM_CommonOsbJMSModule!CommonOsbCF
Message icon - Error The proposed JNDI name "jms.jndi.cf.CommonOsb.CommonOsbCF" for connection factory "GM_CommonOsbJMSModule!CommonOsbCF" is already bound by another object of type "weblogic.jms.frontend.FEConnectionFactoryImpl_1035_WLStub"


The only solution was a server restart.

Minimalistic file load generator and HAFileAdapter tester

I need to copy over and over the same file (with different file names) to a shared SAMBA folder, to test HAFileAdater in a cluster.

Minimalistic approach, a shell script:

for i in {1..10}
do
    cp myfile.xml target/myfile${i}.xml
done

but when it comes to sleeping, sleep in shell takes only a parameter in SECONDS (no milliseconds). Pathetic.

Why some people use Shell script to do anything more than really basic stuff, I will never understand. Probably they started their career 20 years ago with Shell scripting and they never weaned off.

So I do the same in Python:
import shutil
import time
i = 1
while i < 100:
    shutil.copyfile('myfile.xml', 'target/myfile' + str(i) + '.xml'
    i = i + 1
    time.sleep(0.01)
Here is a JCA file poller.... just change FileAdapter into HAFileAdapter in the JCA file, and the source folder, and add a Report action to trace the file name ($inbound/ctx:transport/ctx:request/tp:headers/jca:jca.file.FileName ) here is $inbound :

  
    Read
  
  
    jca://eis/FileAdapter
    request
    best-effort
    
      
        
        Read
        New Text Document.txt
        C:\pierre\infiles\in3
        0
        IG1OEdhYeSMWsjUbSnFe1NnrVx-2v0YafavtRTvgZGs.
        1
        1335498161556
      
      UTF-8
    
    
      
      0
    
  
  
    
      <anonymous>
    
  




WLST and DBEXTS

wls:/offline> from dbexts import dbexts
wls:/offline> d = dbexts()
Traceback (innermost last):
File "", line 1, in ?
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/dbexts.py", line 188, in __init__
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/javaos$py.class", line 256, in __getitem__
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/UserDict$py.class", line 14, in __getitem__
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/javaos$py.class", line 256, in __getitem__
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/UserDict$py.class", line 14, in __getitem__
KeyError: HOME


we need a dbexts.ini
Here it says:
"dbexts will default to looking for a file named 'dbexts.ini' in the same directory as dbexts.py but can optionally be passed a filename to the cfg attribute."


I find out many "dbexts.py", one looking promising is:
C:\bea1035\wlserver_10.3\common\wlst\modules\jython-modules\Lib\dbexts.py

I try putting a dbexts.ini in
C:\bea1035\wlserver_10.3\common\wlst\modules\jython-modules\Lib\dbexts.ini

my dbexts.ini is:
[default]
name=ora

[jdbc]
name=mysql
url=jdbc:mysql://localhost/ziclix
user=
pwd=
driver=org.gjt.mm.mysql.Driver
datahandler=com.ziclix.python.sql.handler.MySQLDataHandler

[jdbc]
name=ora
url=jdbc:oracle:thin:@localhost:1521:xe
user=DEV_SOAINFRA
pwd=DEV_SOAINFRA
driver=oracle.jdbc.driver.OracleDriver
datahandler=com.ziclix.python.sql.handler.OracleDataHandler

and still get the same "KeyError: HOME" error.

Looking in the code I find:

fn = os.path.join(os.path.split(__file__)[0], "dbexts.ini")
if not os.path.exists(fn):
fn = os.path.join(os.environ['HOME'], ".dbexts")


So I need to declare a HOME System property:
set HOME=C:\bea1035\wlserver_10.3\common\wlst\modules\jython-modules\Lib\


This time around I get:

File "", line 1, in ?
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/dbexts.py", line 189, in __init__
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/dbexts.py", line 661, in __init__
File "C:\bea1035\WLSERV~1.3\common\wlst\modules\jython-modules.jar\Lib/dbexts.py", line 664, in parse
IOError: No such file or directory: C:\bea1035\wlserver_10.3\common\wlst\modules\jython-modules\Lib\.dbexts

then I realize the file should be called ".dbexts" rather than dbexts.ini

and this time around IT WORKS!

The mystery is how to pass the dbexts.ini file location in the cfg parameter... really confusing and poorly documented...

Thursday, April 26, 2012

WLST and Oracle DB, zxJDBC, Jython...

the zxJDBC doc:
http://www.jython.org/jythonbook/en/1.0/DatabasesAndJython.html
and also here
http://www.jython.org/archive/21/docs/zxjdbc.html

it seems that WLS 10.3.5 incorporates this library (enter zxJDBC in Everything Search)


java weblogic.WLST

from com.ziclix.python.sql import zxJDBC
params = {}
params['serverName'] = 'localhost'
params['databaseName'] = 'xe'
params['user'] = 'DEV_SOAINFRA'
params['password'] = 'DEV_SOAINFRA'
params['port'] = 1521
db = apply(zxJDBC.connectx, ("oracle.jdbc.xa.client.OracleXADataSource",), params)

this fails with "Error: no such method [setPort] using arg type [class java.lang.Integer], value [1521]"


then I try

jdbc_url = "jdbc:oracle:thin:@pierrepc:1521:XE"
username = "DEV_SOAINFRA"
password = "DEV_SOAINFRA"
driver = "oracle.jdbc.xa.client.OracleXADataSource"

conn = zxJDBC.connect(jdbc_url, username, password, driver)

cursor = conn.cursor(1)
cursor.execute("select count(*) from WLI_QS_REPORT_ATTRIBUTE")
print cursor.rowcount
0
cursor.fetchone()
(4.0,)
print cursor.rowcount
1


it seems to work!

cursor.execute("select MSG_LABELS from WLI_QS_REPORT_ATTRIBUTE")
for a in cursor.description:
print a

('MSG_LABELS', 12, 2048, None, None, None, 1)

try also

print cursor.fetchall()
print cursor.fetchmany()

cursor.execute("select MSG_GUID from WLI_QS_REPORT_ATTRIBUTE order by MSG_GUID")
res = cursor.fetchall()
print res

for a in res:
   print a

for a in res:
   print a[0]



To access a specific column without guessing its position:

cursor.execute("select * from SOMETABLE where ID = '29'")

columnNames = []
for item in cursor.description:
 columnNames.append(item[0])
 
for a in cursor.fetchall():
 ID = a[columnNames.index('ID')]
 DSNAME = a[columnNames.index('DSNAME')]
 print 'ID=' + ID
 print 'DSNAME=' + DSNAME


where ID and DSNAME are column names

Wednesday, April 25, 2012

Hyperic, exploring (and exporting) autodiscovered resources with HQApi

http://support.hyperic.com/display/DOC/HQApi+3.2+resource+command


List details of a platform (machine)

cd C:\pierre\workspace\Hyperic\lib\hqapi1-client-5.0.0\bin
hqapi resource list -platform myhost.acme.com

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ResourcesResponse>
    <Status>Success</Status>
    <Resource id="11328" name="myhost.acme.com" description="Red Hat Enterprise Linux 5"
location="" instanceId="10003" typeId="1">
        <ResourcePrototype resourceTypeId="1" instanceId="10001" id="10001" name="Linux"/>
        <Agent id="10003" address="10.58.25.132" port="2144" version="4.6.5" unidirectional="false"/>

        <Ip address="10.58.25.132" mac="34:8E:99:27:FD:80" netmask="255.255.255.0"/>
        <Ip address="127.0.0.1" mac="00:00:00:00:00:00" netmask="255.0.0.0"/>
        <ResourceInfo key="fqdn" value="myhost.acme.com"/>
    </Resource>
</ResourcesResponse>



List all details of a all Linux platforms (machine)

hqapi resource list -prototype Linux

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ResourcesResponse>
    <Status>Success</Status>
    <Resource id="10882" name="myhost1.acme.com" description="Red Hat Enterprise Linux 5"
location="" instanceId="10001" typeId="1">
        <ResourcePrototype resourceTypeId="1" instanceId="10001" id="10001" name="Linux"/>
        <Agent id="10001" address="10.56.5.191" port="2144" version="4.5.3" unidirectional="false"/>

        <Ip address="10.56.53.191" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="10.56.53.174" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="127.0.0.1" mac="00:00:00:00:00:00" netmask="255.0.0.0"/>
        <ResourceInfo key="fqdn" value="myhost1.acme.com"/>
    </Resource>

    <Resource id="10883" name="myhost2.acme.com" description="Red Hat Enterprise Linux 5"
location="" instanceId="10002" typeId="1">
        <ResourcePrototype resourceTypeId="1" instanceId="10001" id="10001" name="Linux"/>
        <Agent id="10001" address="10.56.5.192" port="2144" version="4.5.3" unidirectional="false"/>

        <Ip address="10.56.53.191" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="10.56.53.174" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="127.0.0.1" mac="00:00:00:00:00:00" netmask="255.0.0.0"/>
        <ResourceInfo key="fqdn" value="myhost.acme.com"/>
    </Resource>
</ResourcesResponse>



Something really fantastic is when you do:

hqapi resource list -prototype Linux --children > myresources.xml

this will dump the ENTIRE database of resources!

And the real great thing is that you can
resource sync --file myresources.xml

to import into a HQ Server the resources exported by "resource list"

Platform MBean Server Used

In the logs I find:


The JMX MBean PlatformMBeanServerUsed attribute is true, but the Platform MBeanServer was created without the hooks for the WLS security infrastructure. The Platform MBeanServer will NOT be used and Platform MBeans will NOT be available via the WLS Runtime or Domain Runtime MBeanServers. This can occur if you have defined Platform MBeanServer system properties or JVM options (-Dcom.sun.management.jmxremote or JRockit -XManagement).
To allow the Platform MBeanServer to be used, you must either remove the system properties/JVM options or start WLS with the following system property:
-Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder
If you want to eliminate this log error and do not need Platform MBeans to be available via WLS, then set the PlatformMBeanUsed attribute in the JMXMBean to false.



In the WL Console, Domain, Configuration, General, we have:

Platform MBean Server Enabled

Specifies whether each server instance initializes the JDK's platform MBean server. Enabling it, along with isPlatformMBeanServerUsed, causes WebLogic Server to use the platform MBean server as its Runtime MBean Server


and also
Platform MBean Server Used


Specifies whether WebLogic Server will use the platform MBean server for its Runtime MBean Server. Previously, WebLogic Server used the platform MBean server by default if it was enabled. This attribute provides a separation between enabling the platform MBean server and using it for WebLogic Server MBeans


So.... in order to USE actually the Platform MBean Server, you not only must set to true both flags (Enabled and Used), but also pass the -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder property.... hard work!

More doc here:
http://docs.oracle.com/cd/E12840_01/wls/docs103/jmxinst/designapp.html#UsingJVMPlatformMBeanServer



"If it is essential that JMX clients be able to monitor your custom MBeans, WebLogic Server MBeans, and the JVM’s platform MBeans through a single MBean server, you can configure the runtime MBean server to be the JVM platform MBean server"


Tuesday, April 24, 2012

How choosing the wrong RAR adapter can screw you big time

In a rush to redeploy a malfunctioning DbAdapter.rar, a colleague has incorrectly chosen the one in the "was" directory

${ORACLE_HOME}/osb/soa/connectors/was/DbAdapter.rar
${ORACLE_HOME}/osb/soa/connectors/DbAdapter.rar

To avoid this problem in future, I just want to remove the was folder from any WebLogic installation. Honestly naming two completely different adapters with the same name is only a call for trouble.

Public Key Authentication With PuTTY

http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/pka-putty.html


I run PuTTYgen

provide passphrase
generate key

save public key ssh-rsa in a file ssh-rsa.txt
save private key to myuserpprd.ppk


I log to remote server as myuser

mkdir .ssh
cd .ssh
vi authorized_keys
paste the public key
save

chmod 600 authorized_keys*
run putty
edit the "Private key file for authentication" to provide myuserpprd.ppk

the first login I get

"Authenticating with public key "rsa-key-20120424"
Passphrase for key "rsa-key-20120424":"

Monday, April 23, 2012

WLST list all JDBC resources (datasources) in WebLogic

allJDBCResources = cmo.getJDBCSystemResources()
for jdbcResource in allJDBCResources:
   dsname = jdbcResource.getName()
   print dsname, jdbcResource.getJDBCResource().getJDBCDataSourceParams().getJNDINames()[0], jdbcResource.getJDBCResource().getJDBCDriverParams().getUrl() 



DbAdapter.rar: weblogic.application.ModuleException: weblogic.connector.exception.RAException: Jndi Name is null or empty.

I have tried everything:
- deleted the deployment and redeployed again
- restarted the Managed Server
- visually checked the plan.xml


I am persuaded that the problem stems from Connection Factories referring to non -existing Datasources... and that the error message should actually be "unable to locate object bla in JNDI tree"..... and surely the error message is really poorly fomulated and gives you no clue of what went wrong.

WebLogic HTTP access.log not being written

(aka How To Change the Buffer Size for the WebLogic Server Http Access Log [ID 1113583.1] )

HTTP access logs are buffered for performance reasons.
By default logs are buffered until reaching 8kb when they are flushed to the file.
You can configure this buffer - if set to zero then logs will be written immediately.


The HTTPAccessLog could sometimes not be viewed from the console.
Solution:
The LogFileMBean now includes a buffer-size-kb parameter. This parameter has a default value of 8 KB; when set to zero or less, it will not buffer the log messages.
For example, to set the buffer size for access.log in config.xml:

<server>
...
<web-server>
<web-server-log>
<buffer-size-kb>0</buffer-size-kb>
<web-server-log> See also LogFileFlushSecs, which is deprecated and no longer available: http://docs.oracle.com/cd/E17904_01/apirefs.1111/e13945/weblogic/management/configuration/WebServerMBean.html#setLogFileFlushSecs%28int%29 so your only chance it to change the BufferSize

edit()
startEdit()
cd /Servers/osbdv2ms1/WebServer/osbdv2ms1/WebServerLog/osbdv2ms1
set('BufferSizeKB',1)
save()
activate()



Sunday, April 22, 2012

Hyperic HQApi and the Groovy Console

HQApi documentation here

"HQApi allows you to circumvent the Hyperic user interface and directly access Hyperic Server functionality. " ... great, I love it!

Let's get our hands dirty....

On HQ Console, Click on Administration, Groovy Console...

type:
print 'ciao'
return 'pippo'

and click "execute"... great, in the "result" tab I see "pippo".... where is "ciao" though? I can't find the stdout of "Server for Hyperic HQ" Windows service.

Scripts are stored in a temporary file C:\pierre\hypericserver\server-4.6.5-EE\hq-engine\hq-server\temp\gcon3792879954783360219.tmp

I click on "HQ Web Services Api" and download the hqapi1-client-5.0.0.tar.gz

The most common useful classes to run the examples in Hyperic site are in

<classpathentry kind="lib" path="C:/pierre/workspace/Hyperic/lib/hqapi1-client-5.0.0/hqapi1-5.0.0.jar"/>
<classpathentry kind="lib" path="C:/pierre/workspace/Hyperic/lib/hqapi1-client-5.0.0/hqapi1-tools-5.0.0.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/hq-rendit-4.6.5.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/hq-server-4.6.5.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/httpclient-4.1.1.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/httpcore-4.1.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericagent/agent-4.6.5-EE/bundles/agent-4.6.5/pdk/lib/httpmime-4.1.1.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/activemq-all-5.3.0.jar"/>
<classpathentry kind="lib" path="C:/pierre/workspace/Hyperic/lib/hqapi1-client-4.3.0/lib/spring-core-3.0.1.RELEASE.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/lib/commons-codec-1.1.jar"/>



Some Groovy Console samples are here:

http://support.hyperic.com/display/hyperforge/Groovy+Snippets

First steps with HQApi command line interface

here instructions on how to create the file.

I do this (my user is pvernet):
cd C:\Users\pvernet\
mkdir ".hq"

I create a client.properties file containing this:

host=localhost
port=7443
secure=true
user=hqadmin
password=hqadmin


and when I do
cd C:\pierre\workspace\Hyperic\lib\hqapi1-client-5.0.0\bin
hqapi agent list

I get a fantastic

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AgentsResponse>
<Status>Success
<Agent id="10003" address="10.16.15.112" port="2144" version="4.6.5" unidirectional="false"/>
</AgentsResponse>


Awesome! This is like the HQApi Hello World!

ImportError: no module named hyperic

I have added all the good classes to CLASSPATH (using full paths),
I run weblogic.WLST and I do

import org.hyperic.hq.hqapi1.HQApi

this gives me ImportError: no module named hyperic

Here it reminds you that there is a

C:\Oracle\Middleware\wlserver_10.3\common\wlst directory

but this is a place only for Python modules...

Well the reality is that, if you set the classpath right (make sure you use full path names), you should not need to make any extra step to use a Java class...

WebLogic CMDB

Definition of a CMDB here

I am starting to model a WebLogic environments management DB here.

It will be a long exercise, but the fountation of a sound management tool for WebLogic administration.


SVG file available here for ER model.


Thursday, April 19, 2012

Deployment descriptors of a OSB MDB

Inside the /opt/oracle/domains/osbdv2do/sbgen/_ALSB_1329599716508.ear you will find:

META-INF/application.xml

<?xml version='1.0' encoding='UTF-8'?>
<jav:application xmlns:jav="http://java.sun.com/xml/ns/javaee">
  <jav:display-name>ProxyService GM_ReqPackMat/ProxyServices/ReqPackMat_ReqPackMatQ_to_WMOS_PS</jav:display-name>
  <jav:module>
    <jav:ejb>ejb.jar</jav:ejb>
  </jav:module>
</jav:application>


META-INF/weblogic-application.xml

<?xml version='1.0' encoding='UTF-8'?>
<web:weblogic-application xmlns:web="http://xmlns.oracle.com/weblogic/weblogic-application">
  <web:ejb>
    <web:start-mdbs-with-application>false</web:start-mdbs-with-application>
  </web:ejb>
  <web:application-param>
    <web:param-name>service-ref</web:param-name>
    <web:param-value>ProxyService$GM_ReqPackMat$ProxyServices$ReqPackMat_ReqPackMatQ_to_WMOS_PS</web:param-value>
  </web:application-param>
  <web:listener>
    <web:listener-class>com.bea.wli.sb.transports.jms.JmsEndPointAppListener</web:listener-class>
  </web:listener>
</web:weblogic-application>


The plan.xml in /opt/oracle/domains/osbdv2do/osb/config/plan/Plan-_ALSB_1335183510889.ear.xml is:

<?xml version='1.0' encoding='UTF-8'?>
<deployment-plan xmlns="http://xmlns.oracle.com/weblogic/deployment-plan" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/deployment-plan http://xmlns.oracle.com/weblogic/deployment-plan/1.0/deployment-plan.xsd">
  <application-name>_ALSB_1335183510889</application-name>
  <module-override>
    <module-name>_ALSB_1335183510889.ear</module-name>
    <module-type>ear</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-application</root-element>
      <uri>META-INF/weblogic-application.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>application</root-element>
      <uri>META-INF/application.xml</uri>
    </module-descriptor>
    <module-descriptor external="true">
      <root-element>wldf-resource</root-element>
      <uri>META-INF/weblogic-diagnostics.xml</uri>
    </module-descriptor>
  </module-override>
  <module-override>
    <module-name>ejb.jar</module-name>
    <module-type>ejb</module-type>
    <module-descriptor external="false">
      <root-element>weblogic-ejb-jar</root-element>
      <uri>META-INF/weblogic-ejb-jar.xml</uri>
    </module-descriptor>
    <module-descriptor external="false">
      <root-element>ejb-jar</root-element>
      <uri>META-INF/ejb-jar.xml</uri>
    </module-descriptor>
  </module-override>
  <config-root>/opt/oracle/domains/osbdv2do/osb/config/plan</config-root>
</deployment-plan>


And in the ejb.jar you can find:

META-INF/ejb-jar.xml

<?xml version='1.0' encoding='UTF-8'?>
<jav:ejb-jar xmlns:jav="http://java.sun.com/xml/ns/javaee">
  <jav:display-name>BEA ALSB JMS Inbound Endpoint</jav:display-name>
  <jav:enterprise-beans>
    <jav:message-driven>
      <jav:display-name>BEA ALSB JMS Inbound Endpoint (Request)</jav:display-name>
      <jav:ejb-name>RequestEJB-2972318866359532858-249b028e.136ddc2e3c6.-7fb5</jav:ejb-name>
      <jav:ejb-class>com.bea.wli.sb.transports.jms.JmsInboundMDB</jav:ejb-class>
      <jav:transaction-type>Container</jav:transaction-type>
      <jav:message-destination-type>javax.jms.Queue</jav:message-destination-type>
      <jav:env-entry>
        <jav:env-entry-name>service-ref</jav:env-entry-name>
        <jav:env-entry-type>java.lang.String</jav:env-entry-type>
        <jav:env-entry-value>ProxyService$PVReadStockDownloadErrorQueue$PVReadStockDownloadErrorQueue_PS</jav:env-entry-value>
      </jav:env-entry>
      <jav:env-entry>
        <jav:env-entry-name>XA-supported</jav:env-entry-name>
        <jav:env-entry-type>java.lang.Boolean</jav:env-entry-type>
        <jav:env-entry-value>true</jav:env-entry-value>
      </jav:env-entry>
      <jav:env-entry>
        <jav:env-entry-name>checksum</jav:env-entry-name>
        <jav:env-entry-type>java.lang.Integer</jav:env-entry-type>
        <jav:env-entry-value>-2061456480</jav:env-entry-value>
      </jav:env-entry>
    </jav:message-driven>
  </jav:enterprise-beans>
  <jav:assembly-descriptor>
    <jav:container-transaction>
      <jav:method>
        <jav:ejb-name>RequestEJB-2972318866359532858-249b028e.136ddc2e3c6.-7fb5</jav:ejb-name>
        <jav:method-name>*</jav:method-name>
      </jav:method>
      <jav:trans-attribute>Required</jav:trans-attribute>
    </jav:container-transaction>
  </jav:assembly-descriptor>
</jav:ejb-jar>

weblogic-ejb-jar.xml



  
    RequestEJB-2972318866359532858-249b028e.136ddc2e3c6.-7fb5
    
      
        1000
        1
      
      jms.jndi.dq.GM_StockDownload.StockDownloadReprocessQ
      t3://myhost.acme.com:8245
      weblogic.jms.XAConnectionFactory
    
    
      600
    
    
      jms/ConnectionFactory
      weblogic.jms.XAConnectionFactory
    
    
      jms/QueueName
      jms.jndi.dq.GM_StockDownload.StockDownloadReprocessQ
    
  
  BEA-010001
  BEA-010054
  BEA-010200
  BEA-010202




Hyperic will discover this MDB as "Weblogic 10.3 Message Driven EJB"

Weblogic osbdv1do osbdv1ms1 _ALSB_1335183510889 RequestEJB-6533237974024524838-be3fafc.135ee630c77.-7f0a_GM_ReqPackMatJMSModule!AcmeCommonJmsServer1@ReqPackMatQ Message Driven EJB

but also as a "Weblogic 10.3 Application"
Weblogic osbdv2do osbdv2ms1 _ALSB_1335183510889 Application


http://www.javamonamour.org/2012/04/wlst-browsing-deployed-applications.html

http://www.javamonamour.org/2011/10/osb-how-mdb-proxy-service-in.html

WLST browsing deployed Applications : OSB EJBs

connect('weblogic', 'weblogic1', 't3://acme102.acme.com:7101')
easeSyntax()
cd AppDeployments
ls



dr-- ALDSP Transport Provider
dr-- ALSB Cluster Singleton Marker Application
dr-- ALSB Coherence Cache Provider
dr-- ALSB Domain Singleton Marker Application
dr-- ALSB Framework Starter Application
dr-- ALSB Logging
dr-- ALSB Publish
dr-- ALSB Resource
dr-- ALSB Routing
dr-- ALSB Subscription Listener
dr-- ALSB Test Framework
dr-- ALSB Transform
dr-- ALSB UDDI Manager
dr-- ALSB WSIL
dr-- AqAdapter
dr-- BPEL 10g Transport Provider
dr-- DMS Application#11.1.1.1.0
dr-- DatalinksPocs-Project1-context-root
dr-- DbAdapter
dr-- EJB Transport Provider
dr-- Email Transport Provider
dr-- FLOW Transport Provider
dr-- FMW Welcome Page Application#11.1.0.0.0
dr-- File Transport Provider
dr-- FileAdapter
dr-- Ftp Transport Provider
dr-- FtpAdapter
dr-- JCA Transport Provider
dr-- JEJB Transport Provider
dr-- JMS Reporting Provider
dr-- JmsAdapter
dr-- MQ Transport Provider
dr-- MQSeriesAdapter
dr-- Message Reporting Purger
dr-- OracleAppsAdapter
dr-- OracleBamAdapter
dr-- SB Transport Provider
dr-- SFTP Transport Provider
dr-- SOA-DIRECT Transport Provider
dr-- ServiceBus_Console
dr-- SocketAdapter
dr-- Tuxedo Transport Provider
dr-- WS Transport Async Applcation
dr-- WS Transport Provider
dr-- XBus Kernel
dr-- _ALSB_1329599716508
dr-- _ALSB_1329599717744
dr-- _ALSB_1329599717858
dr-- _ALSB_1329599717944
dr-- _ALSB_1329754384502
dr-- _ALSB_1329824947023
dr-- _ALSB_1329888150254
dr-- _ALSB_1330332828977
dr-- _ALSB_1330332829084
dr-- _ALSB_1330333576280
dr-- _ALSB_1330333576386
dr-- _ALSB_1330463663134
dr-- _ALSB_1331647120095
dr-- _ALSB_1331708084330
dr-- _ALSB_1332091161130
dr-- _ALSB_1333024471355
dr-- _ALSB_1333969392746
dr-- _ALSB_1334249766371
dr-- _ALSB_1334333217358
dr-- _ALSB_1334608617083
dr-- _ALSB_1334670601508
dr-- wsil-wls


cd _ALSB_1329599716508
ls



dr-- SubDeployments
dr-- Targets

-r-- AbsoluteInstallDir null
-r-- AbsolutePlanDir /opt/oracle/domains/osbdv2do/osb/config/plan
-r-- AbsolutePlanPath /opt/oracle/domains/osbdv2do/osb/config/plan/Pla
n-_ALSB_1329599716508.ear.xml
-r-- AbsoluteSourcePath /opt/oracle/domains/osbdv2do/sbgen/_ALSB_1329599
716508.ear
-r-- ApplicationIdentifier _ALSB_1329599716508
-r-- ApplicationName _ALSB_1329599716508
-r-- CompatibilityName null
-r-- DeploymentOrder 100
-r-- DeploymentPlan [B@185b08b
-r-- DeploymentPlanExternalDescriptors null
-r-- DeploymentPrincipalName null
-r-- InstallDir null
-r-- ModuleType ear
-r-- Name _ALSB_1329599716508
-r-- Notes null
-r-- PlanDir osb/config/plan
-r-- PlanPath Plan-_ALSB_1329599716508.ear.xml
-r-- SecurityDDModel DDOnly
-r-- SourcePath sbgen/_ALSB_1329599716508.ear
-r-- StagingMode stage
-r-- Type AppDeployment
-r-- ValidateDDSecurityData false
-r-- VersionIdentifier null

-r-x freezeCurrentValue Void : String(attributeName)
-r-x isSet Boolean : String(propertyName)
-r-x unSet Void : String(propertyName)



see also http://www.javamonamour.org/2011/10/osb-how-mdb-proxy-service-in.html

Hyperic, installing Server on a Linux machine

requires root access if you want to use the internal PostGREs Database:

this script:
/opt/hyperic/installer/data/hqdb/tune-os.sh

must be run as root.

If you are not root, you can try to fool him by doing

touch /tmp/_hq-was-tuned

and hope your system will be fool enough to believe you... do at your own risk!!!

do
cd /opt/hyperic/server/server-4.6.5-EE/bin/
./hq-server.sh start
cd ../logs/
less bootstrap.log

I find this :o( :


2012-04-22 11:50:20,511 ERROR [main] [org.apache.tools.ant.UnknownElement@135] Task "dbupgrade" finished with error.
/opt/hyperic/server/server-4.6.5-EE/data/db-upgrade.xml:43: DBUpgrader: sql error: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.hyperic.tools.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:244)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




In fact I do

cd /opt/hyperic/installer/bin
./db-start.sh

and I find

./db-start.sh: line 7: cd: hqdb: No such file or directory
./db-start.sh: line 16: /opt/hyperic/installer/bin/pg_ctl: No such file or directory

:o(



Otherwise you must configure the Server to use an external db, such as Oracle

Wednesday, April 18, 2012

Hyperic, how to change HQ Server IP for an Agent

I have been torturing myself for ages about this simple question...

just run
hq-agent.sh setup

but at the end of the process I get

"Error updating agent: Unable to invoke 'updateAgent': Permission denied"

When I do this, it works:

Would you like to re-setup the auth tokens [default=no]: yes

Would you like to re-setup the auth tokens [default=no]: yes
- Received temporary auth token from agent
- Registering agent with HQ
- HQ gave us the following agent token
1334082450388-1632010252278915721-1721183443318423049
- Informing agent of new HQ server
- Validating
- Successfully setup agent

Tuesday, April 17, 2012

wrapper | The Hyperic HQ Agent service is not installed

First of all, Run As Administrator.


hq-agent.bat install
wrapper | Hyperic HQ Agent service installed.


hq-agent.bat start

wrapper | Starting the Hyperic HQ Agent service...
wrapper | Waiting to start...
wrapper | Hyperic HQ Agent started.
- No token file found, waiting for Agent to initialize
[ Running agent setup ]
Should Agent communications to HQ be unidirectional [default=no]:



What is the HQ server IP address: 10.60.188.193
Should Agent communications to HQ always be secure [default=yes]:
What is the HQ server SSL port [default=7443]:
- Testing secure connection ... Success
What is your HQ login [default=hqadmin]:
What is your HQ password:

What IP should HQ use to contact the agent [default=10.60.188.193]:
What port should HQ use to contact the agent [default=2144]:
- Received temporary auth token from agent
- Registering agent with HQ
The server to agent communication channel is using a self-signed certificate and could not be verifi
ed
Are you sure you want to continue connecting? [default=no]: yes
- HQ gave us the following agent token
1334696759417-1107302852316448320-217456217152137655
- Informing agent of new HQ server
- Validating
- Successfully setup agent



It works like a breeze, and the HQ server immediately discovers:

Microsoft Windows 2008
Weblogic Admin 1...main AdminServer
PostgreSQL 8.2
Tomcat 6.0
Oracle 10g
Agent 4.6.5

China rising steady and fast in the Software Industry

I have noticed a steady trend in the last few months: on the blog I am getting more and more hits from China




Of course, they do everything, why not Software?

So, I am quietly waiting for the day a good WebLogic administrator from China will bill remotely 100 USD a day, and get ready to go fishing.... actually no, I am vegetarian... to go picking berries in the woods.

Extended Logging Format Fields (ELFF) in WebLogic

By default WebLogic logs this fields in the access.log:
date time cs-method cs-uri sc-status

Here is the MBean WebServerLogMBean.ELFFields

and here the entire documentation on the supported field identifiers.

c-ip

The IP address of the client.

s-ip

The IP address of the server.


Don't forget that in case of need you can define a Java class to provide extra ELF fields.

Sunday, April 15, 2012

Eclipse not responding

Having spent most of my current project watching an hourglass hovering on Eclipse,
I have at last decided to do something about that.

Eclipse executable is in: C:\Oracle\Middleware\oepe_11gR1PS4\eclipse.exe

C:/Oracle/Middleware/jrockit_160_24_D1.1.2-4/jre/bin/javaw.exe

In the same directory, the eclipse.ini contains:

-vm
C:/Oracle/Middleware/jrockit_160_24_D1.1.2-4/jre/bin/javaw.exe
-startup
plugins/org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.1.2.R36x_v20101222
-showsplash
org.eclipse.platform
--launcher.defaultAction
openFile
-vmargs
-Xms256m
-Xmx768m
-XX:MaxPermSize=256m
-Dsun.lang.ClassLoader.allowArraySyntax=true
-Dweblogic.home=C:\Oracle\Middleware\wlserver_10.3
-Dharvester.home=C:\Oracle\Middleware\Oracle_OSB1\harvester
-Dosb.home=C:\Oracle\Middleware\Oracle_OSB1
-Dosgi.bundlefile.limit=750
-Dosgi.nl=en_US

I run Task Manager, and View Columns PID, so that I know the PID of this JVM

I install VisualVM to connect to the Eclipse JVM (anyway the Eclipse process is properly named in VisualVM, no need to use Task Manager to view the PID)

I click on the Threads tab and I am ready to monitor the next "Not Responding" event...I also click on "Sampler", it hangs for a while to determine if Eclipse supports CPU and Memory profiling.... it does!

So I monitor the CPU while Eclipse seems to be hanging, and I notice that the jrockit.net.SocketNativeIO.readBytesPinned is eating my CPU alive...

Oh, but why JRockit? Because it's the vm option in eclipse.ini.

What if I switch to Sun JVM?
I try with C:/Oracle/Middleware/jdk160_24/bin/javaw.exe

It seems to be more responsive now...

Will keep an eye on it.... next time I will do a
netstat -ano
(see here)
to find out on Windows which sockets are open by a PID


I have tried using:
Window/Preferences/Java/Editor/Content Assist/ Uncheck Auto Activation

Saturday, April 14, 2012

JMS Server bases on a JDBCStore on RAC

First create a GridLink DataSource, but BEWARE, it should not be based on a XA driver, nor have LLR or Emulate 2PC:

http://docs.oracle.com/cd/E17904_01/web.1111/e13701/store.htm#i1143431

You cannot specify a JDBC data source that is configured to support global (XA) transactions. Therefore, the specified JDBC data source must use a non-XA JDBC driver. In addition, you cannot enable Logging Last Resource or Emulate Two-Phase Commit in the data source. This limitation does not remove the XA capabilities of layered subsystems that use JDBC stores. For example, WebLogic JMS is fully XA-capable regardless of whether it uses a file store or any JDBC store.

This is all the WLST needed:


To run the tests:
#Variable declaration

username='weblogic'
password='weblogic1'
serverurl='t3://myserver.acme.com:7101'

clustername='osbdv2cl'
#Connect with WLST to Admin

cd /opt/oracle/domains/osbdv2do
. ./bin/setDomainEnv.sh
java weblogic.WLST
connect(username, password, serverurl)
#create DataSource (non-RAC, no XA, no LLR)

datasource='JMSPerfTestDataSource'
datasource_jndi='jdbc.JMSPerfTestDataSource'
datasource_url='jdbc:oracle:thin:@myhost2.acme.com:1551:dosb01'
datasource_driver='oracle.jdbc.OracleDriver'
datasource_username='dv1_soainfra'
datasource_password='dv1_acme'

edit()
startEdit()

cmo.createJDBCSystemResource(datasource)
cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource)
cmo.setName(datasource)
cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource + '/JDBCDataSourceParams/' + datasource)
set('JNDINames',jarray.array([String(datasource_jndi)], String))
cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource + '/JDBCDriverParams/' + datasource)
cmo.setUrl(datasource_url)
cmo.setDriverName(datasource_driver)
cmo.setPassword(datasource_password)

cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource + '/JDBCConnectionPoolParams/' + datasource)
cmo.setTestTableName('SQL SELECT 1 FROM DUAL')
cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource + '/JDBCDriverParams/' + datasource + '/Properties/' + datasource + '')

cmo.createProperty('user')
cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource + '/JDBCDriverParams/' + datasource + '/Properties/' + datasource + '/Properties/user')
cmo.setValue(datasource_username)
cd('/JDBCSystemResources/' + datasource + '/JDBCResource/' + datasource + '/JDBCDataSourceParams/' + datasource)
cmo.setGlobalTransactionsProtocol('None')
cd('/SystemResources/' + datasource)
set('Targets',jarray.array([ObjectName('com.bea:Name=' + clustername + ',Type=Cluster')], ObjectName))

save()
activate()
#create File Store

filestore_name='LocalJMSPerfTestFileStore'
filestore_target='osbdv2ms1 (migratable)'
filestore_directory='/opt/oracle/domains/osbdv2do/servers/osbdv2ms1/data/store'

edit()
startEdit()

cd('/')
cmo.createFileStore(filestore_name)
cd('/FileStores/' + filestore_name)
cmo.setDirectory(filestore_directory)
set('Targets',jarray.array([ObjectName('com.bea:Name=' + filestore_target + ',Type=MigratableTarget')], ObjectName))

save()
activate()
#create JDBC Store

jdbcstore_name='JDBCStoreJMSPerfTest'
jdbcstore_datasource='JMSPerfTestDataSource'
jdbcstore_prefix='JMSPerfTest'
jdbcstore_target='osbdv2ms1 (migratable)'

edit()
startEdit()

cd('/')
cmo.createJDBCStore(jdbcstore_name)
cd('/JDBCStores/'+ jdbcstore_name)
cmo.setDataSource(getMBean('/SystemResources/' + jdbcstore_datasource))
cmo.setPrefixName(jdbcstore_prefix)
set('Targets',jarray.array([ObjectName('com.bea:Name=' + jdbcstore_target + ',Type=MigratableTarget')], ObjectName))

save()
activate()
#create JMS Server

cmo.createJMSServer('JMSServerPerfTest')
cd('/Deployments/JMSServerPerfTest')
cmo.setPersistentStore(getMBean('/FileStores/LocalPerfTestFileStore'))
set('Targets',jarray.array([ObjectName('com.bea:Name=osbdv2ms1,Type=Server')], ObjectName))
#create JMS Module

cd('/')
cmo.createJMSSystemResource('JMSPerfTestModule')
cd('/SystemResources/JMSPerfTestModule')
set('Targets',jarray.array([ObjectName('com.bea:Name=osbdv2cl,Type=Cluster')], ObjectName))
#create JMS SubDeployment

cmo.createSubDeployment('JMSPerfTestSD')
cd('/SystemResources/JMSPerfTestModule/SubDeployments/JMSPerfTestSD')

set('Targets',jarray.array([ObjectName('com.bea:Name=ACMECommonJmsServer1,Type=JMSServer')], ObjectName))
#create JMS Distributed Queue

cd('/JMSSystemResources/JMSPerfTestModule/JMSResource/JMSPerfTestModule')
cmo.createUniformDistributedQueue('JMSPerfDQueue')
cd('/JMSSystemResources/JMSPerfTestModule/JMSResource/JMSPerfTestModule/UniformDistributedQueues/JMSPerfDQueue')
cmo.setJNDIName('jms.JMSPerfDQueue')
cmo.setDefaultTargetingEnabled(true)
cmo.unSet('Template')
cmo.setLoadBalancingPolicy('Round-Robin')
cmo.setResetDeliveryCountOnForward(true)
cmo.setIncompleteWorkExpirationTime(-1)
cmo.setForwardDelay(-1)
cmo.setAttachSender('supports')
cmo.setSAFExportPolicy('All')
cmo.setProductionPausedAtStartup(false)
cmo.setDefaultUnitOfOrder(false)
cmo.setDefaultTargetingEnabled(false)
cmo.setUnitOfOrderRouting('Hash')
cmo.setUnitOfWorkHandlingPolicy('PassThrough')
cmo.setInsertionPausedAtStartup(false)
cmo.setMessagingPerformancePreference(25)
cmo.setConsumptionPausedAtStartup(false)
#assign SubDeployment

cmo.setSubDeploymentName('JMSPerfTestSD')

save()
activate()

WebLogic and NFS, the odd couple...

It's a sort of metropolitan legend that NFS is "not fully supported" with WebLogic (I guess "not fully supported" means "it works unless you do something naughty").


Here is the Official Oracle Reference:

http://docs.oracle.com/cd/E14571_01/web.1111/e13814/storetune.htm#CACFDGCA

In fact I had heard in the past that SAN is recommended over NFS.




Some salient excerpts from the above document:

“Oracle strongly recommends verifying the behavior of a server restart after abrupt machine failures when the JMS messages and transaction logs are stored on an NFS mounted directory.”

“The NFS storage device does not become aware of machine failure in a timely manner and the locks are not released by the storage device. As a result, after abrupt machine failure, followed by a restart, any subsequent attempt by WebLogic Server to acquire locks on the previously locked files may fail.”


ONE workaround they suggest is to “disable file locking” on the FileStores of any kind (diagnostics, default, custom file stores, JMS logs...)

Centos drbd nfs

We are using a dual machine mounting Centos and the drbd NFS package (it's free!) to ensure a shared NFS storage with a mirror, sinchronized in case 1 disk crashes.

On any server mounting this device, if I do "mount" I see:

/dev/mapper/rootvg-rootlv on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/rootvg-optlv on /opt type ext3 (rw)
/dev/mapper/rootvg-homelv on /home type ext3 (rw)
/dev/mapper/rootvg-tmplv on /tmp type ext3 (rw)
/dev/mapper/rootvg-varlv on /var type ext3 (rw)
/dev/cciss/c0d0p1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
//blablabla.acme.com/PIPPOPPRD on /data/bo/bu type cifs (rw,mand,noexec,nosuid,nodev)
/dev/mapper/rootvg-hypericlv on /home/hyperic type ext3 (rw)
peppettone-vip2:/drbd/main/shared/ on /opt/oracle/domains/osbdomain/shared type nfs (rw,user=root,noexec,nosuid,nodev,user,addr=10.46.38.45)


As you can see, the last entry peppettone is of type nfs (the cifs is the Samba share).


This is a quite interesting article on how a Highly Available NFS share is setup (it's 5 pages long...).

Notice that the NFS share is represented by a Virtual IP address (the 10.46.38.45), actually shielding the 2 (or more?) instances of physical servers.


More in DRDB here.
More on NFS here.

Friday, April 13, 2012

Linux disk performance benchmark

Suppose you have local and shared disks mounted on your box. You want to find out their relative performance, to assess for instance the performance loss in case you locate your WebLogic files (logs, filestores...) on one or the other.

$ df -kP

Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/rootvg-rootlv 8125880 1953072 5753380 26% /
/dev/mapper/rootvg-optlv 30472188 24894656 4005004 87% /opt
/dev/mapper/rootvg-homelv 2031440 1939192 0 100% /home
/dev/mapper/rootvg-tmplv 3047184 161068 2728832 6% /tmp
/dev/mapper/rootvg-varlv 9141624 687588 7982244 8% /var
/dev/cciss/c0d0p1 497829 31487 440640 7% /boot
tmpfs 12342596 0 12342596 0% /dev/shm



$ /sbin/hdparm /dev/mapper/rootvg-optlv

/dev/mapper/rootvg-optlv: Permission denied

(hdparm has to be run with root privileges!)


iostat
Linux 2.6.18-308.1.1.el5 (myhostname)         04/13/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.03    0.01    0.20    0.01    0.00   97.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0       10.14         8.85       305.48   16119821  556372354
cciss/c0d0p1      0.00         0.00         0.00       2351        156
cciss/c0d0p2     10.14         8.85       305.48   16117198  556372198
dm-0              4.94         0.63        38.96    1148082   70960475
dm-1             13.75         6.45       108.96   11742082  198439411
dm-2              5.78         1.29        45.76    2341482   83332872
dm-3              2.51         0.01        20.04      12186   36492920
dm-4             11.49         0.48        91.77     871650  167146504
dm-5              0.00         0.00         0.00       1072          0


You can also use Bonnie++ but I don't want to install any module.

REGEX and string manipulation with SQL

REGEXP_SUBSTR : search a string for a regular expression pattern


REGEXP_REPLACE the function returns source_char with every occurrence of the regular expression pattern replaced with replace_string

REPLACE REPLACE returns char with every occurrence of search_string replaced with replacement_string

Querying the OSB Reporting Provider table

We are using the OSB JMS Reporting Provider to track messages.
We must build custom reports to track the activity within a range of time.

The only problem is that the key-value pairs are stored as SEMICOLON (;) separated lists key1=value1;key2=value2....

Nothing is easier that writing SQL queries supported by PL/SQL functions.

Here how to create the function and types needed:
http://stackoverflow.com/questions/1089508/how-to-best-split-csv-strings-in-oracle-9i

if you run
select tokenize('key1=value1;key2=value2', ';') from dual;

you get the list
GM.TOKEN_LIST('key1=value1','key2=value2')

Since my PL/SQL skills are very basic, I will follow a pure SQL Functions approach:


WITH csv_data AS (SELECT '&your_csv_string' AS CSV ,'[^ |;]+' REGEX FROM DUAL)

SELECT regexp_replace(TRIM(REGEXP_SUBSTR(A.CSV, A.REGEX, 1, LEVEL)),'[^[:print:]]', '') STR FROM csv_data a CONNECT BY LEVEL <= REGEXP_COUNT(A.CSV, A.REGEX) ORDER BY 1 Honestly I prefer a "poor man's" approach, that I will explain in another post.

Extracting data with SQL from WLI_QS_REPORT_DATA and WLI_QS_REPORT_ATTRIBUTE

describe wli_qs_report_data
Name       Null     Type         
---------- -------- ------------ 
MSG_GUID   NOT NULL VARCHAR2(64) 
DATA_TYPE           NUMBER(38)   
ENCODING            VARCHAR2(24) 
DATA_VALUE          BLOB         

describe wli_qs_report_attribute
Name                  Null     Type           
--------------------- -------- -------------- 
MSG_GUID              NOT NULL VARCHAR2(64)   
DB_TIMESTAMP          NOT NULL DATE           
LOCALHOST_TIMESTAMP   NOT NULL DATE           
HOST_NAME             NOT NULL VARCHAR2(50)   
STATE                 NOT NULL VARCHAR2(8)    
NODE                           VARCHAR2(128)  
PIPELINE_NAME                  VARCHAR2(128)  
STAGE_NAME                     VARCHAR2(128)  
INBOUND_SERVICE_NAME  NOT NULL VARCHAR2(256)  
INBOUND_SERVICE_URI   NOT NULL VARCHAR2(128)  
INBOUND_OPERATION              VARCHAR2(64)   
OUTBOUND_SERVICE_NAME          VARCHAR2(256)  
OUTBOUND_SERVICE_URI           VARCHAR2(256)  
OUTBOUND_OPERATION             VARCHAR2(64)   
MSG_LABELS                     VARCHAR2(2048) 
ERROR_CODE                     VARCHAR2(64)   
ERROR_REASON                   VARCHAR2(1024) 
ERROR_DETAILS                  VARCHAR2(2048) 



if WLI_QS_REPORT_DATA.DATA_TYPE == 2 then DATA_VALUE contains XML

if WLI_QS_REPORT_DATA.DATA_TYPE == 1 then DATA_VALUE contains String

MSG_GUID is unique, and it joins the 2 tables (there is a PK_WLI_QS_REPORT_ATTRIBUTE and IX_WLI_QS_REPORT_DATA index on the tables, UNIQUE on MSG_GUID).

If one of your labels is "InterfaceID=BLA", then this:

select REGEXP_SUBSTR(MSG_LABELS, 'InterfaceID=[^;]+') from wli_qs_report_attribute where MSG_GUID = 'uuid:7f9b72b69446518a:6207823d:13595beb079:-7e59';

will return InterfaceID=BLA, and

select REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'InterfaceID=[^;]+'), 'InterfaceID=', '') from wli_qs_report_attribute where MSG_GUID = 'uuid:7f9b72b69446518a:6207823d:13595beb079:-7e59';

will return BLA


So, putting it all together:

select * from wli_qs_report_attribute where REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'InterfaceID=[^;]+'), 'InterfaceID=', '') = 'BLA';

will return all the records of a given InterfaceID

(of course
select * from wli_qs_report_attribute where MSG_LABELS like '%InterfaceID=BLA%';
would do the same)

In our Reporting we have defined several labels:
InterfaceID=BLA
TechnicalMessageID=BLA^InputFileName.xml^ORIGIN^1330463665790
EventType=FileConsumed
PathOfService=GM_BLA/ProxyServices/BLA_File_PS
EventOccuredOn=2012-02-28T22:14:25.837+01:00
BusinessID=12345

If I want to find all events of a given interface and of a given EventType within a given time range:


select * from wli_qs_report_attribute where REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'InterfaceID=[^;]+'), 'InterfaceID=', '') = 'BLA' and MSG_LABELS like '%EventType=FileConsumed%' and REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'EventOccuredOn=[^;]+'), 'EventOccuredOn=', '') >= '2012-02-28T22:14' and REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'EventOccuredOn=[^;]+'), 'EventOccuredOn=', '') <= '2012-02-28T23:14'; and if I need to provide the value of each label as a different column: select REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'InterfaceID=[^;]+'), 'InterfaceID=', '') as InterfaceID, REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'EventType=[^;]+'), 'EventType=', '') as EventType, REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'BusinessID=[^;]+'), 'BusinessID=', '') as BusinessID, REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'EventOccuredOn=[^;]+'), 'EventOccuredOn=', '') as EventOccuredOn, REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'TechnicalMessageID=[^;]+'), 'TechnicalMessageID=', '') as TechnicalMessageID from wli_qs_report_attribute where REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'InterfaceID=[^;]+'), 'InterfaceID=', '') = 'BLA' and MSG_LABELS like '%EventType=FileConsumed%' and REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'EventOccuredOn=[^;]+'), 'EventOccuredOn=', '') >= '2012-02-28T22:14' and REPLACE(REGEXP_SUBSTR(MSG_LABELS, 'EventOccuredOn=[^;]+'), 'EventOccuredOn=', '') <= '2012-02-28T23:14'; creating a function will help you clean up the code:

create or replace FUNCTION findLabelValue
(labels IN VARCHAR2, labelname IN VARCHAR2 ) return VARCHAR2
is
begin
return REPLACE(REGEXP_SUBSTR(labels, labelname || '=[^;]+'), labelname || '=', '');
end;



With this, a sample query joining to events to determine the first and last even of a message processing is:

select
findlabelvalue(A.MSG_LABELS, 'InterfaceID') as InterfaceID,
findlabelvalue(A.MSG_LABELS, 'BusinessID') as BusinessID,
findlabelvalue(A.MSG_LABELS, 'EventType') as EventType1,
findlabelvalue(A.MSG_LABELS, 'EventOccuredOn') as Event1OccuredOn,
findlabelvalue(A.MSG_LABELS, 'EventType') as EventType2,
findlabelvalue(A.MSG_LABELS, 'EventOccuredOn') as Event2OccuredOn,
findlabelvalue(A.MSG_LABELS, 'TechnicalMessageID') as TechnicalMessageID

from wli_qs_report_attribute A, wli_qs_report_attribute B
where findlabelvalue(A.MSG_LABELS, 'InterfaceID') = 'Pippo'
and A.MSG_LABELS like '%EventType=FileConsumed%'
and findlabelvalue(A.MSG_LABELS, 'EventOccuredOn') >= '2012-02-28T22:14'
and findlabelvalue(A.MSG_LABELS, 'EventOccuredOn') <= '2012-05-28T23:14'
and findlabelvalue(A.MSG_LABELS, 'TechnicalMessageID') = findlabelvalue(B.MSG_LABELS, 'TechnicalMessageID')
and B.MSG_LABELS like '%EventType=FTPFilePut%'
;



Selling software entails much more than manufacturing a good product

A software producer tells me:



Software business is a _big_ business. It is big in that you need loooots of money to get they notice you.

Most funny is that you don't even need to have a very good product.

What you do need is:

1). A well-designed web site

2). Money to buzz about yourself on every corner. This includes not only various banners, but most important: having various computer magazines to write articles about you (that's all for payment); publish a couple of books about your product (but do not forget to hire a tech-writer for this, as well as to pay the publisher); participate in various exhibition (also pay for your place there); and the last: issue a press-release about every new "nail" in your software (but do not forget to subscribe to "media-kits" everywhere you want to press-release yourself)

3). At last, hire a lot of "account-managers" to push your software to the local businesses.

When you calculate how much it will cost, you'll get a figure of a several millions euros. That's it.
The development of the software itself will be the least part of your expenses.

Why all that system is needed to get people use that very software in the end, I don't really understand myself, particularly when everybody can reach every website in Internet and download/try any software out

Our customers do include some very top companies (or rather some people working there), like: (omissis) .... But all of them purchased mostly a single license (or set of licenses) and that was all.
If I was told from the very start that this would happen, I wouldn't belief...

Also, making all those things, I listed, would take some number of people. You wouldn't be able to do it alone.
And once you've got some tens of people, you obviously need a few more ordinary professionals, like a secretary,
a bookkeeper (or "CFO", as they say).
Possibly, also a human-resource manager and a lawyer -- but those would be required for a bit bigger business.

If your company is very small yet (e.g. 2-4 guys), the whole item N2 (that is, buzzing about yourself on every corner), could be done by a special "public relation" company. There are quite a lot of them, and they will arrange everything (all those sub-items). But you have to pay them, of course!

Well, whatever you do, a few thousands euros (or dollars) would be never enough. This is just called a "seed capital".
If you want to go any further, a true investor (with those millions) must be found somewhere!

That's theory, of course. But, lots of this is written in various books, even academic ones.
It is a big science, in fact, how to create and run company.

For instance, here you can read who those "account manages" are: http://en.wikipedia.org/wiki/Account_manager
I knew a few of them. Quite modest people with not so large salaries.
But if you have an army of them, you will be making millions! :)

But some other people, I guess, may think something different about all this...

For instance, how the "open source" model works still remains a mystery to me.
A typical open source project is posed as a team of geeks.
But on the other hand (from my experience), to make really complex things you need very coordinated and dedicated people.
That means, it is a full-time job. You've got to feed them (and not only feed, by the way)!
And the promotion is also needed, less of course, but anyway.




So.... if you dream of being the next Bill Gates, now you know it's not only about writing good code... am I saying that Microsoft produces good products? I must be ill...

Tuesday, April 10, 2012

Getting started with Hyperic - demo



I am really impressed by what you can do - like running "top" on a remote box within your Hyperic console....

This one is also very good:



Installing Hyperic Server on Windows and Agent on Linux to monitor WebLogic

Downloading:

https://www.vmware.com/tryvmware/p/activate.php?p=hyperic&lp=1

VMware vFabric Hyperic 4.6.5 Server and Agent - Linux (64 bit)
03/15/12 | 4.6.5 | 396M | Binary (.tar.gz)

It gives me a hyperic-hqee-installer-4.6.5-x86-64-linux.tar.gz
I unzip it into a hyperic-hqee-installer-4.6.5-x86-64-linux.tar
Which untarred gives a hyperic-hqee-installer-4.6.5 folder

which contains

86,371,609 hyperic-hqee-agent-4.6.5-x86-64-linux.tar.gz
DIR installer
49,455 open_source_licenses-SpringSource_tc_Server.txt
796,537 open_source_licenses.txt
241,153,046 server-4.6.5.tar.gz
108 setup.bat
147 setup.sh

Here the installation instructions.

I also download the Windows version.
Apparently no need for DB (using internal PostgreSQL DB)
Run setup.bat. Create a C:\pierre\hypericserver directory first.
Provide SMTP server.

Admin user: hqadmin


At the end I get:


--------------------------------------------------------------------------------
Installation Complete:
Server successfully installed to: C:\pierre\hypericserver/server-4.6.5-EE
--------------------------------------------------------------------------------

You should now install the HQ server as a Windows Service using this command:

C:\pierre\hypericserver\server-4.6.5-EE\bin\hq-server.bat install

You can then use the Service Control Manager (Control Panel->Services) to
start the HQ server. Note that the first time the HQ server starts up it may
take several minutes to initialize. Subsequent startups will be much faster.

Once the HQ server reports that it has successfully started, you can log in
to your HQ server at:

http://mymachine.euc.acme.com:7080/
username: hqadmin
password: pippo0

To change your password, log in to the HQ server, click the "Administration"
link, choose "List Users", then click on the "hqadmin" user.


Setup completed.
A copy of the output shown above has been saved to:
C:\pierre\downloads\hyperic-hqee-installer-4.6.5-win32\hyperic-hqee-installer-4.6.5\installer\logs\hq-install.log

Press the Enter key to exit setup.




I run a Command Prompt as Administrator, and execute

C:\pierre\hypericserver\server-4.6.5-EE\bin\hq-server.bat install

I get the message

wrapper | Hyperic HQ Server service installed.

I run services.msc and I can see a "Hyperic HQ Server" service: I start it.
The executable is C:\pierre\hypericserver\server-4.6.5-EE\wrapper\sbin\wrapper-windows-x86-32.exe -s C:\pierre\hypericserver\server-4.6.5-EE\conf\wrapper.conf set.SERVER_INSTALL_HOME=C:\pierre\hypericserver\server-4.6.5-EE\bin\.. set.JAVA_HOME=C:\pierre\hypericserver\server-4.6.5-EE\bin\..\jre

I access in Firefox the url http://mymachine.euc.acme.com:7080/
It takes a long time. But it works. I get this lovely screen:



Now let's install the agent on Linux:


Now I install the agent on Linux.
Copy all the Linux stuff to /opt/hyperic/.
chmod 755 *.sh
chmod 755 /opt/hyperic/installer/bin/*
mkdir /opt/hyperic/agent
./setup.sh

choose 2 (agent)

when asked
HQ agent installation path [default '/home/hyperic']: say:
/opt/hyperic/agent

It will tell you:


Loading install configuration...
Install configuration loaded.
Preparing to install...
Validating agent install configuration...
Installing the agent...
Looking for previous installation
Unpacking /opt/hyperic/hyperic-hqee-agent-4.6.5-x86-64-linux.tar.gz to: /opt/hyperic/agent/agent-4.6.5-EE...
Setting permissions on /opt/hyperic/agent/agent-4.6.5-EE...
Setting permissions on agent binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Agent successfully installed to: /opt/hyperic/agent
--------------------------------------------------------------------------------
You can now start your HQ agent by running this command:

/opt/hyperic/agent/agent-4.6.5-EE/bin/hq-agent.sh start

Setup completed.
A copy of the output shown above has been saved to:
/opt/hyperic/installer/logs/hq-install.log

Deleting temporary JRE



I run
/opt/hyperic/agent/agent-4.6.5-EE/bin/hq-agent.sh start


Starting HQ Agent...
- No token file found, waiting for Agent to initialize
[ Running agent setup ]
Should Agent communications to HQ be unidirectional [default=no]:
What is the HQ server IP address: 10.16.14.140
Should Agent communications to HQ always be secure [default=yes]:
What is the HQ server SSL port [default=7443]:
- Testing secure connection ... Success
What is your HQ login [default=hqadmin]:
What is your HQ password:
What IP should HQ use to contact the agent [default=10.16.5.122]:
What port should HQ use to contact the agent [default=2144]:
- Received temporary auth token from agent
- Registering agent with HQ
The server to agent communication channel is using a self-signed certificate and could not be verified
Are you sure you want to continue connecting? [default=no]: yes
- HQ gave us the following agent token
13340820388-163201022278915721-172183443318423049
- Informing agent of new HQ server
- Validating
- Successfully setup agent



At this point, I go to the Hyperic console Home Page and in the Auto-Discovery tab I find the 2 WebLogic domains and the Apache Tomcat running on the Linux box.... pretty cool!


On the Linux box I see 2 processes:

/opt/hyperic/agent/agent-4.6.5-EE/wrapper/sbin/../../wrapper/sbin/wrapper-linux-x86-64


/opt/hyperic/agent/agent-4.6.5-EE/jre/bin/java -Djava.security.auth.login.config=../../bundles/agent-4.6.5/jaas.config etc etc


To stop the agent:
/opt/hyperic//agent/agent-4.6.5-EE/bundles/agent-4.6.5/bin/hq-agent.sh stop