Starting a Managed Server with WLST and NodeManager

1) factor out commonly used constants into a setenv.sh file:

#!/bin/sh

#prefix for domain 
PREF=osbpr1
Interface=bond0
AdminIP=10.56.5.119
NetMask=255.255.255.0

JAVA_HOME="/opt/oracle/java"
MW_HOME="/opt/oracle/fmw"
WL_HOME="${MW_HOME}/wlserver_10.3"

DOM_HOME=/opt/oracle/domains

LOG_HOME=/var/log/weblogic
# home of certificates
CERT_HOME=/opt/oracle/certs

JAVA_VERSION=java-1.6.0-sun-1.6.0.29.x86_64

2) prepare a NodeManager userConfigFile:

/opt/oracle/domains/osbdv1do/nmuserconfigfile.secure

containing

weblogic.management.username={AES}hU8qRGjiFmqK6kHqG8yZlpXTD+KZGjld85q7sIgMP4w\=
weblogic.management.password={AES}xF+6rgG5NqNYBxDzqR/MlLlR1iYSqQezJrJ+Mi52gTc\=

and a userKeyFile /opt/oracle/domains/osbdv1do/nmuserkeyfile.secure



3) Prepare a startAdmin.py file (in reality it can start ANY server, not only the admin):

adminName = sys.argv[1]
domainName = sys.argv[2]
nmHost = sys.argv[3]

nmPort = '5556'
domHome='/opt/oracle/domains/' + domainName

print nmHost, nmPort, domainName, adminName

# -----------------------------------------------------------------------------
# connect to NodeManager
# -----------------------------------------------------------------------------
nmConnect( userConfigFile=domHome + '/nmuserconfigfile.secure', userKeyFile=domHome + '/nmuserkeyfile.secure', host=nmHost, port=nmPort, domainName=domainName, domainDir=domHome, nmType='plain' )

# -----------------------------------------------------------------------------
# start the server
# -----------------------------------------------------------------------------
try:

    nmStart( adminName, domHome )

except:
    nmDisconnect()
    exit('y', 4)


nmDisconnect()

Configuration error while reading domain directory

Error occured while performing nmConnect : Cannot connect to Node Manager. : Configuration error while reading domain directory

Solution: locate all nm_password.properties in your domains, and enter

username=bla
password=bla

where bla/bla correspond to your nodemanager username - typically weblogic_n - not to WLS username

Also, make sure your nodemanager.domains file contain the entry

domainname=domainpath


see also http://www.javamonamour.org/2011/08/nmconnect-weblogicnodemanagernmexceptio.html

NodeManager inactive

On a 2 nodes cluster, for one node (machine) I was getting that NodeManager was Inactive/Unreachable:

Could not execute command "getVersion" on the node manager. Reason: "Access to domain for user BLA denied

I have done the usual check on nodemanager.domains file. then googling I found this post http://jvzoggel.wordpress.com/2011/08/19/oracle-weblogic-nodemanager-bea-300033/ so I have edited:

vi /opt/oracle/domains/osbstg1do/config/nodemanager/nm_password.properties

entering:

username=weblogic_nm
password=weblogic_pw


and restarted the NodeManager. After restart the file contains only a line hashed=blablabla.

The same username/password must be entered in Domain/Security/General in "NodeManager Username" and "NodeManager Password"

See also http://www.javamonamour.org/2011/09/nmconnect-nmstart-nmkill.html

Crash Recovery with NodeManager

http://docs.oracle.com/cd/E12839_01/web.1111/e13740/overview.htm

Node Manager and System Crash Recovery

To ensure that Node Manager properly restarts servers after a system crash, you must perform the following:

    Ensure that CrashRecoveryEnabled is set to true.

    The CrashRecoveryEnabled configuration property allows Node Manager to restart servers after a system crash. The property is not enabled by default.
    You should start the Administration Server via Node Manager.
    All managed servers should be started via the Administration Server. You can accomplish this via WLST or the Administration Console.

After the system is restarted, Node Manager checks each managed domain specified in the nodemanager.domains file to determine if there are any server instances that were not cleanly shutdown. This is determined by the presence of any lock files which are created by Node Manager when a WebLogic Server process is created. This lock file contains the process identifier for WebLogic Server startup script. If the lock file exists, but the process ID is not running, Node Manager will attempt to automatically restart the server.

If the process is running, Node Manager performs an additional check to access the management servlet running in the process to verify that the process corresponding to the process ID is a WebLogic Server instance.
Note:  When Node Manager performs a check to access the management servlet, an alert may appear in the server log regarding improper credentials.

When you run startNodeManager.sh you will see "Automatically restarting server process as part of crash recovery"
and then it fails with:

Fatal error in node manager server
java.lang.NullPointerException
        at weblogic.nodemanager.server.ServerManager.getStartCallbacks(ServerManager.java:187)
        at weblogic.nodemanager.server.AbstractServerManager.startServer(AbstractServerManager.java:211)
        at weblogic.nodemanager.server.ServerManager.isCrashRecoveryNeeded(ServerManager.java:157)
        at weblogic.nodemanager.server.AbstractServerManager.initialize(AbstractServerManager.java:99)
        at weblogic.nodemanager.server.AbstractServerManager.(AbstractServerManager.java:63)
        at weblogic.nodemanager.server.ServerManager.(ServerManager.java:38)
        at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:96)
        at weblogic.nodemanager.server.DomainManager.(DomainManager.java:60)
        at weblogic.nodemanager.server.NMServer.initDomains(NMServer.java:220)
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:197)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:377)
        at weblogic.NodeManager.main(NodeManager.java:31)

Possibly the status of a server is determined by the servername.state (not sure):

cd /opt/oracle/domains/osbpr1do/servers/osbpr1ms2/data/nodemanager
cat osbpr1ms2.state
FORCE_SHUTTING_DOWN:Y:N


cat osbpr1ms2.pid
24767

Restart that NodeManager

I had a domain with admin listening on port 7013. NodeManager was running.
I have deleted the $DOMAIN_HOME, and recreated the domain from scratch, with the Admin listening at port 7003.
Much to my despair, restarting the ManagedServers through the NodeManager would still bring up instances referring to the admin running at 7013:

"ps -ef" was showing:

-Dweblogic.management.server=http://myserver.acme.com:7013

 Desperate greps everywhere brought no evidence of a 7013 still hiding somewhere.

At last, restarting the NodeManager fixed the issue. Evidently, NodeManager caches the information relative to a given domain, and if you change relevant stuff in the domain you better restart also the NodeManager.
Sad, but true.

FAILED_NOT_RESTARTABLE

While MS2 was STARTING (from WLS console using the NodeManager) I decided to shut it down (always from the console). TOo bad for me. MS2 went into FAILED_NOT_RESTARTABLE and there was nothing more I could do with it.
I had finally to restart the NodeManager to make it go back to "SHUTDOWN". Weird-oh.

OSB and NodeManager

I have created a OSB 10.3 (doing some archeology work) domain,
when I start ms1 from startManagedWebLogic.sh everything is fine,
when I start with node manager I get a bunch of

"weblogic java.lang.IllegalStateException: zip file closed"

for each deployment, and server fails horribly.

Going up a little in the logs, I find:

weblogic.common.ResourceException: Unknown Data Source cgDataSource

and

weblogic.common.resourcepool.ResourceSystemException: Cannot load driver class: com.pointbase.jdbc.jdbcUniversalDriver

I have made sure I was setting JAVA_VENDOR=Sun and JAVA_HOME to Sun jdk in commEnv.sh

I have cleared the tmp, stage and cache directories

In https://kr.forums.oracle.com/forums/thread.jspa?threadID=906756 it suggests to use the hostname rather than localhost in the JDBC configuration XML

Then of course I notice that the pbclient57.jar is not in the classpath as it should be (see the output of the startManagedWebLogic.sh:

CLASSPATH=:/home/orauser/bea/patch_wlw1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/orauser/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/orauser/bea/patch_cie670/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/orauser/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/orauser/bea/patch_alsb1031/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/orauser/bea/jdk160_05/lib/tools.jar:/home/orauser/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/home/orauser/bea/wlserver_10.3/server/lib/weblogic.jar:/home/orauser/bea/modules/features/weblogic.server.modules_10.3.0.0.jar:/home/orauser/bea/wlserver_10.3/server/lib/webservices.jar:/home/orauser/bea/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/home/orauser/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar::/home/orauser/bea/wlserver_10.3/../modules/features/osb.server.modules_10.3.1.0.jar:/home/orauser/bea/osb_10.3/lib/flow/sb-flow.jar:/home/orauser/bea/wlserver_10.3/../modules/com.bea.core.jaxen_1.0.1.0_1-1-1.jar:/home/orauser/bea/osb_10.3/lib/version.jar:/home/orauser/bea/osb_10.3/lib/alsb.jar:/home/orauser/bea/osb_10.3/3rdparty/lib/j2ssh-ant.jar:/home/orauser/bea/osb_10.3/3rdparty/lib/j2ssh-common.jar:/home/orauser/bea/osb_10.3/3rdparty/lib/j2ssh-core.jar:/home/orauser/bea/osb_10.3/3rdparty/lib/j2ssh-dameon.jar:/home/orauser/bea/osb_10.3/3rdparty/classes:/home/orauser/bea/wlserver_10.3/../modules/com.bea.core.apache.commons.logging_1.1.0.jar:/home/orauser/bea/osb_10.3/lib/transports/jca/jca-binding-api.jar:/home/orauser/bea/osb_10.3/lib/transports/jca/bpm-infra.jar:/home/orauser/bea/osb_10.3/lib/transports/jca/log4j_1.2.8.jar:/home/orauser/bea/osb_10.3/lib/transports/jca/xmlparserv2.jar:%WL_HOME/server/ext/jdbc/oracle/orai18n.jar:/home/orauser/bea/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/home/orauser/bea/wlserver_10.3/server/lib/xqrl.jar::

)


So really the only way to properly start OSB MS is to set the
"startscriptenabled=true" in the nodemanager.properties.

I have done it and it works.

Paolo Uccello, San Giorgio e il Drago

See also here http://blogs.oracle.com/jamesbayer/entry/weblogic_nodemanager_quick_sta

java.lang.ClassNotFoundException: com.bea.wli.config.deployment.server.ServerVersion

java.lang.ClassNotFoundException: com.bea.wli.config.deployment.server.ServerVersion

This error could indicate that a component was deployed on a cluster member but not other members of that cluster. Make sure that any component deployed on a server that is part of a cluster is also deployed on all other members of that cluster

Solution:

add
C:\Oracle2\Middleware\Oracle_OSB1\modules\com.bea.common.configfwk.wlinterop_10.3.1.0.jar

to the Server Start/ Classpath parameters of the Managed Server

If you get

FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)

then you must set
set ALSB_DEBUG_FLAG=false
and
set debugFlag=false

in setDomainEnv.cmd (sh)

WLST and NodeManager

wls:/offline> nm()
Not connected to Node Manager

wls:/offline> startNodeManager()
Launching NodeManager ...
blablabla
Node Manager starting in the background


wls:/offline> nmConnect('weblogic','weblogic1','localhost','5556','soadev','C:/Oracle2/Middleware/user_projects/domains/soadev','ssl')
Connecting to Node Manager ...
Successfully Connected to Node Manager.

wls:/offline>nmServerStatus('AdminServer')

wls:/offline>nmKill('AdminServer')


start a server WITHOUT NodeManager:
wls:/offline>startServer('AdminServer','soadev','t3://localhost:7001','weblogic','weblogic1','C:/Oracle2/Middleware/user_projects/domains/soadev','true',60000,'false')


see here http://download.oracle.com/docs/cd/E12840_01/wls/docs103/config_scripting/manage_servers.html



connect('weblogic', 'weblogic1', 't3://localhost:7001')

serverRuntime()

wls:/soadev/serverRuntime> ls()
dr-- ApplicationRuntimes
dr-- AsyncReplicationRuntime
dr-- ClusterRuntime
dr-- ConnectorServiceRuntime
dr-- DefaultExecuteQueueRuntime
dr-- EntityCacheCumulativeRuntime
dr-- EntityCacheCurrentStateRuntime
dr-- EntityCacheHistoricalRuntime
dr-- ExecuteQueueRuntimes
dr-- JDBCServiceRuntime
dr-- JMSRuntime
dr-- JTARuntime
dr-- JVMRuntime
dr-- JoltRuntime
dr-- LibraryRuntimes
dr-- LogBroadcasterRuntime
dr-- LogRuntime
dr-- MANAsyncReplicationRuntime
dr-- MANReplicationRuntime
dr-- MailSessionRuntimes
dr-- MaxThreadsConstraintRuntimes
dr-- MinThreadsConstraintRuntimes
dr-- PathServiceRuntime
dr-- PersistentStoreRuntimes
dr-- RequestClassRuntimes
dr-- SAFRuntime
dr-- SNMPAgentRuntime
dr-- ServerChannelRuntimes
dr-- ServerSecurityRuntime
dr-- ServerServices
dr-- SingleSignOnServicesRuntime
dr-- ThreadPoolRuntime
dr-- TimerRuntime
dr-- WANReplicationRuntime
dr-- WLDFRuntime
dr-- WTCRuntime
dr-- WebServerRuntimes
dr-- WorkManagerRuntimes
dr-- WseeClusterFrontEndRuntime
dr-- WseeWsrmRuntime

-r-- ActivationTime 1305484322463
-r-- AdminServer true
-r-- AdminServerHost [2001:0:5ef5:79fd:1c05:d557:ab21:abcd]
-r-- AdminServerListenPort 7001
-r-- AdminServerListenPortSecure false
-r-- AdministrationPort 9002
-r-- AdministrationPortEnabled false
-r-- AdministrationURL t3://[2001:0:5ef5:79fd:1c05:d557:ab21:abcd]:7001
-r-- AsyncReplicationRuntime null
-r-- ClusterRuntime null
-r-- CurrentDirectory C:\Oracle2\Middleware\user_p
rojects\domains\soadev\.
-rw- CurrentMachine
-r-- DefaultExecuteQueueRuntime null
-r-- DefaultURL t3://[2001:0:5ef5:79fd:1c05:d557:ab21:abcd]:7001
-r-- EntityCacheCumulativeRuntime null
-r-- EntityCacheCurrentStateRuntime null
-r-- EntityCacheHistoricalRuntime null
-r-- HealthState Component:ServerRuntime,State:HEALTH_OK,MBean:AdminServer,ReasonCode:[]
-r-- JoltRuntime null
-r-- ListenAddress pierrepc/2001:0:5ef5:79fd:1c05:d557:ab21:abcd
-r-- ListenPort 7001
-r-- ListenPortEnabled true
-r-- MANAsyncReplicationRuntime null
-r-- MANReplicationRuntime null
-r-- MiddlewareHome C:\Oracle2\Middleware
-r-- Name AdminServer
-r-- OpenSocketsCurrentCount 1
-r-- OracleHome C:\Oracle2\Middleware
-rw- Parent null
-r-- PathServiceRuntime null
-r-- RestartRequired false
-r-- RestartsTotalCount 0
-r-- SSLListenAddress null
-r-- SSLListenPort 7002
-r-- SSLListenPortEnabled false
-r-- SocketsOpenedTotalCount 1
-r-- State RUNNING
-r-- StateVal 2
-r-- Type ServerRuntime
-r-- WANReplicationRuntime null
-r-- WLECConnectionServiceRuntime null
-r-- WeblogicHome C:\Oracle2\Middleware\wlserver_10.3
-r-- WeblogicVersion WebLogic Server 10.3.4.0 Fri Dec 17 20:47:33 PST 2010 1384255
-r-- WseeClusterFrontEndRuntime null

-r-x addRequestClassRuntime Boolean : WebLogicMBean(weblogic.management.runtime.RequestClassRuntimeMBean)
-r-x forceShutdown Void :
-r-x forceSuspend Void :
-r-x getIPv4URL String : String(protocol)
-r-x getIPv6URL String : String(protocol)
-r-x getServerChannel java.net.InetSocketAddress : String(protocol)
-r-x getURL String : String(protocol)
-r-x preDeregister Void :
-r-x restartSSLChannels Void :
-r-x resume Void :
-r-x shutdown Void :
-r-x shutdown Void : Integer(timeout),Boolean(ignoreSessions)
-r-x start Void :
-r-x suspend Void :
-r-x suspend Void : Integer(timeout),Boolean(ignoreSessions)

NodeManager: domain salt file not found

See here the full story.

startNodeManager.cmd is in C:\bea11\wlserver_10.3\server\bin
in the same dir is also nodemanager.properties WHICH IS NOT BEING USED!

This file is configured to have:
DomainsFile=C\:\\bea11\\wlserver_10.3\\server\\bin\\nodemanager.domains


for some reason the nodemanager.domains is missing and this other is loaded:
C:\bea11\wlserver_10.3\common\nodemanager\nodemanager.domains


because NODEMGR_HOME=%WL_HOME%\common\nodemanager
where WL_HOME=C:\bea11\wlserver_10.3\

Verify the Domain name mappings:

if you are pointing to the wrong directory, where no valid domain exists, you will get this error emssage:

java.io.FileNotFoundException: Domain directory 'C:\bea11\wlserver_10.3\common\nodemanager' invalid (domain salt file not found)