Tuesday, September 29, 2015

Testing OVAB 12

I download OVAB_12. from http://www.oracle.com/technetwork/middleware/ovab/downloads/index.html?ssSourceSiteId=ocomen . Attempts to install the product from its installer failed, because I could not install the dependencies.... so I would rather use the preinstalled VM.

Form VirtualBox I do Import Appliance.

and agree the EULA
I get this message:
Could not start the machine OVAB_12. because the following physical network interfaces were not found:

en0: Ethernet (adapter 1)

You can either change the machine's network settings or stop the machine.

the issue is easily fixed by validating the proposed settings change.
I setup the Linux OS installation (create a "oracle"user). Read the OVAB appliance instructions http://download.oracle.com/otn/nt/middleware/12c/121200/OVAB_12.
Predefined users are ovab/oracle and root/oracle. I do cd /home/ovab/Oracle/Middleware/Oracle_Home and check that the installation is there.
Login as ovab/oracle, cd /home/ovab/bin/, nohup ./startDB.sh &, nohup ./startWLSAdminServer.sh & , ./startOVABStudio.sh 1

<Mar 8, 2015 8:16:51 AM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: Could not obtain the localhost address. The most likely cause is an error in the network configuration of this machine.
java.lang.AssertionError: Could not obtain the localhost address. The most likely cause is an error in the network configuration of this machine.
        at weblogic.server.channels.AddressUtils$AddressMaker.getLocalHost(AddressUtils.java:38)
        at weblogic.server.channels.AddressUtils$AddressMaker.<clinit>(AddressUtils.java:33)
        at weblogic.server.channels.AddressUtils.getIPAny(AddressUtils.java:154)
        at weblogic.protocol.configuration.ChannelHelper.checkConsistency(ChannelHelper.java:61)
        at weblogic.server.channels.ChannelService.start(ChannelService.java:207)
        Truncated. see log file for complete stacktrace
Caused By: java.net.UnknownHostException: dhcppc4: dhcppc4
        at java.net.InetAddress.getLocalHost(InetAddress.java:1360)
        at weblogic.server.channels.AddressUtils$AddressMaker.getLocalHost(AddressUtils.java:36)
        at weblogic.server.channels.AddressUtils$AddressMaker.<clinit>(AddressUtils.java:33)
        at weblogic.server.channels.AddressUtils.getIPAny(AddressUtils.java:154)
        at weblogic.protocol.configuration.ChannelHelper.checkConsistency(ChannelHelper.java:62)
        Truncated. see log file for complete stacktrace

hostname is dhcppc4

su - root (enter oracle), vi /etc/hosts, add line " dhcppc4", exit

now run: startWLSClusterServer.sh 1, startOPMN.sh, startOHS.sh, webtierStatus.sh,


Excellent presentation by the excellent Kai Yu:

go to minute 33 for OVAB-related stuff

here is the paper:


The server name bla is unknown to the administration server

If you get this:

weblogic.management.ManagementException: The server name bla is unknown to the administration server. Check if restart is required.
        at weblogic.management.provider.internal.RuntimeAccessImpl.initialize(RuntimeAccessImpl.java:447)
        at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:49)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:462)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:167)
        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
        at weblogic.Server.main(Server.java:71)

oracle documentation says:


Cause The -Dweblogic.Name command line property specifies a server that does not exist in the configuration that is active on the administration server. This can occur if the managed server was added to the configuration along with non-dynamic changes that affect the administration server. In this case, the administration server must be restarted.

Action Check to see if the administration server is marked as needing a restart. If it is, then restart the administration server before trying to boot this managed server. Check the configuration for the list of configured servers.

Yet there is another explanation: that your managed server is connecting to an admin belonging to a different domain! Check that there are no processing listening on the ip/port that you have given as admin url to the weblogic server.

Tuesday, September 22, 2015

JVisualVM MBeans plugin fails with "Data not available because JMX connection to the JMX agent could not be established"

If you get a "Data not available because JMX connection to the JMX agent could not be established" message on MBeans, try starting the monitored JVM with -Dcom.sun.management.jmxremote

Monday, September 21, 2015

Visual GC plugin for JVisualVM says "Not supported for this JVM"

For some local JVMs the Visual GC plugin (version 2.2.1) works, but for instance when I connect JVisualVM to itself I get "Not supported for this JVM" while all the rest works, also the MBean plugin
It's of course the same JVM...
I have tried running jstatd, same story...

I am very sorry, Visual GC plugin is one of my favorite tools... I only wish it failed with some diagnostics... frustrating...

Sunday, September 20, 2015

Playing with CAT and classloaders

tutorial video here:

I was reading the excellent Oracle document http://docs.oracle.com/middleware/1212/wls/WLPRG/classloading.htm#WLPRG320 on classloaders - very nice but it misses pictures and real data.

So I have created a EAR PVEar, and inside a Stateless EJB, an EJBClient (and a PVTimer, ignore it) and a WebApp

I open the cat tool, analyze the PVEar, click on "detailed", and I get this:
System Classloaders

 Type: sun.misc.Launcher$ExtClassLoader
HashCode: 22293724


Type: sun.misc.Launcher$AppClassLoader
HashCode: 27035333


Type: weblogic.utils.classloaders.GenericClassLoader
HashCode: 21751181

Application Classloaders

Type: weblogic.utils.classloaders.FilteringClassLoader
HashCode: 10170360
Filter: []
Classpath: empty
Type: weblogic.utils.classloaders.GenericClassLoader
HashCode: 1771581


clicking on the EJB module I get exactly the same picture. But on the WebApp module I get an extra FilteringClassLoader:
Application Classloaders

(this one is same as in the EJB)

Type: weblogic.utils.classloaders.FilteringClassLoader
HashCode: 10170360
Filter: []
Classpath: empty
Type: weblogic.utils.classloaders.GenericClassLoader
HashCode: 1771581


(this one is new)

Type: weblogic.utils.classloaders.FilteringClassLoader
HashCode: 6976334
Filter: []
Classpath: empty
Type: weblogic.utils.classloaders.ChangeAwareClassLoader
HashCode: 12727142


So, the FilteringClassLoader 10170360 separates from the System Classloader ANYTHING inside the EAR module.
The FilteringClassLoader 6976334 separates the WebApp from the EAR.WebApp and EJB share the same Application Classloaders GenericClassLoader 1771581.

gchisto- analyzing GC patterns

git clone https://github.com/jewes/gchisto.git
cd gchisto
set JAVA_HOME=d:\pierre\Java\jdk1.7.0_79
d:\apps\apache-maven-3.3.3\bin\mvn clean install
cd target
java -jar gchisto-1.0.1-SNAPSHOT.jar
beware that it requires also the jars in the lib folder - they are included in the MANIFEST.MF file...
I have run the tool on some PROD gc.... I was quite pleased with the result, you get almost all the information you need to assess how bad the situation is - of course you will never know WHAT caused your trouble, but you know when and how much CPU and wait time you are paying for Minor GC and Full GC.

Friday, September 18, 2015

jmap and userid

Reading on StackOverflow some people report issues when you take a heap profiling with jmap (jmap -dump or jmap -histo... NOT with jmap -heap) using root rather than actual user who started the JVM process.
Specifically you might get a:
/cs/java/jdk160_85-64b/bin/jmap -F -dump:format=b,file=/tmp/PIPPOheapdump20150918.hdump 10166
10166: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
At this point you do a ps -ef to find out who started that JVM.... and you get a NUMBER (uid) rather than a username...!!! unfortunately you cant "sudo su - ".... so you must do a reverse lookup of the UID...using the "getent passwd UID" command:

id jcqpc
uid=5010012(jcqpc) gid=5008728(jcqpc) groups=5008728(jcqpc)

getent passwd 5010012
jcqpc:x:5010012:5008728:Created by CentrifyEx Automation:/home/jcqpc:/bin/centrifyda

One would expect "id" to be able to do the reverse lookup, but this would be too easy.

Tuesday, September 15, 2015

com/sun/xml/ws/client/WSServiceDelegate$DelegatingLoader filling up PERMGEN

If you invoke WS using SEI through jaxws-rt, you might encounter an issue where your PERM space is filled with instances of com/sun/xml/ws/client/WSServiceDelegate$DelegatingLoader and you get a wonderful OutOfMemory PERMGEN error.
Looking at the code in http://grepcode.com/file/repo1.maven.org/maven2/com.sun.xml.ws/jaxws-rt/2.2.8/com/sun/xml/ws/client/WSServiceDelegate.java?av=f we see that each invokation to createEndpointIFBaseProxy will instantiate a new DelegatingLoader(loader1, loader2); (which is an inner class).... createEndpointIFBaseProxy is called for each getPort call... This inner class was not there in previous versions of the library.

So you might want to try an older version of the library ... or to apply some caching trick so as to avoid a getPort call for every WS invokation (if the WS is always the same, it seems to me that you don't need to go through all the work of creating a new Proxy for each call.... )

Installing JAD on Eclipse

GREAT instructions are here http://jadclipse.sourceforge.net/wiki/index.php/Main_Page

.... more to come...

Since I am behind corporate proxy, Eclipse Marketplace is blocked (as usual....)... so I had to download the jar and put in in ECLIPSE_HOME\plugin folder... and restart Eclipse.

In Eclipse, check that JadClipse menu appears in Windows/Preferences/Java. Remember to set path to jad (you will have to download it separately).

Saturday, September 12, 2015

Java Mission Control, the Swiss Knife of Monitoring

I have extensively used JRockit Mission Control and really missed when I moved to Sun-JVM projects.... now that we use WebLogic 12 we have Java 1.7 (> u40) and so we can use Java Mission Control (JMC), which is exactly like the JRockit version..... with its fantastic Flight Recorder...

Awesome presentation here ( a bit blabla in the first few minutes.... just be patient...)

Here the code used in the example:
package com.pierre;

public class DeadlockGenerator {
    private static class AllocThread extends Thread {
 public void run() {
     while (true) {
  try {
      sleep(20 * 1000);
  } catch (Exception e) {

  for (int i = 0; i < 40000; i++) {
      char[] tmp = new char[1024 * 1024];
      tmp[1] = 'a';

    private static class LockerThread extends Thread {
 Object l1;
 Object l2;

 public void init(Object lock1, Object lock2) {
     l1 = lock1;
     l2 = lock2;

 public void run() {
     while (true) {
  synchronized (l1) {
      try {
      } catch (InterruptedException e) {
  synchronized (l2) {
      try {
      } catch (InterruptedException e) {
  System.out.println("Got one!");

    public static void main(String[] args) {
 AllocThread allocThread = new AllocThread();
 Object lock1 = new Object();
 Object lock2 = new Object();
 LockerThread first = new LockerThread();
 LockerThread second = new LockerThread();
 first.init(lock1, lock2);
 second.init(lock2, lock1);


This video is much shorter and at the end it shows how to solve locking issues

monitoring remotely with jvisualvm and jstatd

If you run jstatd on a machine, you will get:

could not create remote object
access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
        at java.security.AccessController.checkPermission(AccessController.java:559)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at java.lang.System.setProperty(System.java:783)
        at sun.tools.jstatd.Jstatd.main(Jstatd.java:139)

So, create a jstatd.policy file and enter:
grant codebase "file:D:/pierre/Java/jdk1.7.0_79/lib/tools.jar" {
permission java.security.AllPermission;

(full documentation of policy files here )

then run "jstatd -J-Djava.security.policy=jstatd.policy"

Now in jvisualvm you can add remote host, then right-click on the host, "add jstatd connection" and this will connect to the default 1099 port on remote host (it's a RMI server, so it uses the default RMI port)

use "jps " from your local machine to check that the jstatd is actually reachable.

See also http://www.nljug.org/nieuws/blog-profiling-remote-jvm-using-visualvm/ for a more complete list of options. Particularly useful the -p option and the -J-Djava.rmi.server.hostname=x.x.x.x .

Useful JVM flags, revisited

This guy Patrick Peschlow (possibly a superhuman, a hero.... most likely a German, only Germans can love so much classifications) did an amazing job at illustrating all (most of) the JVM flags.
He explained it all in such a clear and precise way that I was about to cry for the emotion.

  • server and -client
  • version and -showversion
  • -Xint, -Xcomp, and -Xmixed

  • JVM flag categories
  • -XX:+PrintCompilation and -XX:+CITime
  • -XX:+UnlockExperimentalVMOptions
  • -XX:+LogCompilation and -XX:+PrintOptoAssembly

  • -XX:+PrintFlagsFinal and -XX:+PrintFlagsInitial
  • -XX:+PrintCommandLineFlags

  • -Xms and -Xmx (or: -XX:InitialHeapSize and -XX:MaxHeapSize)
  • -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath
  • -XX:OnOutOfMemoryError
  • -XX:PermSize and -XX:MaxPermSize
  • -XX:InitialCodeCacheSize and -XX:ReservedCodeCacheSize
  • -XX:+UseCodeCacheFlushing

  • -XX:NewSize and -XX:MaxNewSize
  • -XX:SurvivorRatio
  • -XX:+PrintTenuringDistribution
  • -XX:InitialTenuringThreshold, -XX:MaxTenuringThreshold and -XX:TargetSurvivorRatio
  • -XX:+NeverTenure and -XX:+AlwaysTenure

  • Throughput vs. pause times
  • Garbage collection on the HotSpot JVM
  • -XX:+UseSerialGC
  • -XX:+UseParallelGC
  • -XX:+UseParallelOldGC
  • -XX:ParallelGCThreads
  • -XX:-UseAdaptiveSizePolicy
  • -XX:GCTimeRatio
  • -XX:MaxGCPauseMillis

  • -XX:+UseConcMarkSweepGC
  • -XX:+UseParNewGC
  • -XX:+CMSConcurrentMTEnabled
  • -XX:ConcGCThreads
  • -XX:CMSInitiatingOccupancyFraction
  • -XX+UseCMSInitiatingOccupancyOnly
  • -XX:+CMSClassUnloadingEnabled
  • -XX:+CMSIncrementalMode
  • -XX:+ExplicitGCInvokesConcurrent and -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses
  • -XX:+DisableExplicitGC

  • -XX:+PrintGC
  • -XX:+PrintGCDetails
  • -XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps
  • -Xloggc

More References:


Virtual Machine Garbage Collection Tuning https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/

Friday, September 11, 2015


If you configure a Gridlink Datasource with the wrong ServiceName, you might get:

java.sql.SQLException: Pool connect failed : java.lang.ClassCastException: java.lang.Boolean cannot be cast to oracle.ucp.jdbc.oracle.OracleConnectionAffinityContext

This is possibly one of the weirdest exceptions I have ever seen...spooky and unreal...
It turns out that the root cause could also be that tablespace is full, or other DB-related issues.... but it's NOT a WLS or Application issue (apart from being a useless error message...)

If you wonder what the "ucp" means, it's a RAC-specific implementation of the more traditional connection pool : https://docs.oracle.com/cd/E11882_01/java.112/e12265/intro.htm#BABDIDGD

The way I understand it is that a UCP is a RAC-specific Connection Pool service running UNDER the WebLogic connection pool, and handling RAC-specific features like seamless recovery from node failures.

PS the whole stacktrace is:

Caused By: java.sql.SQLException: java.lang.ClassCastException: java.lang.Boolean cannot be cast to oracle.ucp.jdbc.oracle.OracleConnectionAffinityContext
        at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:278)
        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:503)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:594)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:545)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:538)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.ClassCastException: java.lang.Boolean cannot be cast to oracle.ucp.jdbc.oracle.OracleConnectionAffinityContext
        at oracle.ucp.jdbc.oracle.RACManagerImpl.selectConnectionPerRCLBAndAffinity(RACManagerImpl.java:2126)
        at weblogic.jdbc.common.rac.internal.UCPRACModuleImpl.getConnection(UCPRACModuleImpl.java:269)
        at weblogic.jdbc.common.internal.HAConnectionPool.reserve(HAConnectionPool.java:264)
        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:137)
        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:501)
        Truncated. see log file for complete stacktrace

Tuesday, September 8, 2015

The simplest possible Windows FTP server

http://www.sentex.net/~mwandel/ftpdmin/ ftpdmin .... if all you need is to have a colleague send you a big file which doesn't fit in an email, this utility is for you.... just remember to start it with a root dir, because you don't want to give access to your whole hard drive.... you can operate on it using the windows built-in fto client, and just hit ENTER when asked for username and password...

WebLogic CAT (classloader analysis tool) fails if the application under exam is FAILED

CAT encountered an error
 java.io.IOException: Could not locate: myapplication version: module: 
      at weblogic.tools.cat.internal.DefaultClassLoaderAnalyzerManager.getClassLoaderAnalyzer(DefaultClassLoaderAnalyzerManager.java:185)

I guess this is a CATCH 22 situation... you call the doctor because the patient is sick, but the doctor can't examine the patient because he is sick....

Try it again, Sam...

Friday, September 4, 2015

Toptal recruitment process

I have stumbled upon a couple of blog posts speaking very negatively about Toptal... I have (proudly) passed their first 2 stages of interviews - then I have decided to opt out because I prefer to employ my free time to study, do sport and read books, rather than trying to make extra money coding websites for which I have really no knack.

Here a few considerations

The Codility tests are difficult, yes. But they are not impossible and with some exercise (I have invested maybe 3 days) you can really improve and learn some common strategies. For me it was very rewarding to learn all those algorithms, and I found the tutorials published on Codility site tremendously interesting. Maybe in real life very rarely you have to implement such algorithms, just like in real life you don't have to use Maxwell equations, but still being able to code complex problems is part of your engineer background, so stop bitching.

Part 2 of the interview can be some time after the Codility tests, so I suggest you review your code before part 2, because they might ask you to justify your coding.

The "face to face" interview is also quite challenging, you must code real time while the interviewer watches you. In my case the interviewer was extremely kind and this helped me a lot not to go totally in panic. The problem was relatively simple, much simpler than the Codility problems, but honestly having to code while someone watches adds quite some hindrance to the task. On the whole it was stressful but rewarding. Hey I know nobody loves being interviewed, but unfortunately no pain no gain, it's part of life, so stop bitching.

I have no clue how much you get paid and which kind of projects you can work on.

And no, nobody in Toptal asked me to write this panegyric post.

Thursday, September 3, 2015

Vagrant up fails on Windows 10 with VBoxManage.exe creating the host-only adapter

Bringing machine 'admin' up with 'virtualbox' provider...
==> admin: Clearing any previously set network interfaces...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["hostonlyif", "create"]

Stderr: 0%...
Progress state: E_FAIL
VBoxManage.exe: error: Failed to create the host-only adapter
VBoxManage.exe: error: Code E_FAIL (0x80004005) - Unspecified error (extended info not available)
VBoxManage.exe: error: Context: "int __cdecl handleCreate(struct HandlerArg *,int,int *)" at line 66 of file VBoxManageHostonly.cpp

Just upgrade to latest vagrant an VirtualBox 5.0.2, it worked for me... even if they say VBOX 5.0.2 is not officially supported... I have no other choice...