Thursday, September 29, 2016

UnknownHostException returned by DNS, in reality due to not enough file descriptors available

interesting case, intermittently we get this error: somehostnamehere
                at Method)
                at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(
                at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(
                at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(
                at org.apache.http.impl.execchain.MainClientExec.establishRoute(
                at org.apache.http.impl.execchain.MainClientExec.execute(
                at org.apache.http.impl.execchain.ProtocolExec.execute(
                at org.apache.http.impl.execchain.RetryExec.execute(
                at org.apache.http.impl.execchain.RedirectExec.execute(
                at org.apache.http.impl.client.InternalHttpClient.doExecute(
                at org.apache.http.impl.client.CloseableHttpClient.execute(
                at org.apache.http.impl.client.CloseableHttpClient.execute(

"DNS resolver that uses the default OS implementation for resolving host names"

public InetAddress[] resolve(String host) throws UnknownHostException 
{ return InetAddress.getAllByName(host); }

InetAddress Caching 
 The InetAddress class has a cache to store successful as well as unsuccessful host name resolutions. 
 By default, when a security manager is installed, in order to protect against DNS spoofing attacks, the result of positive host name resolutions are cached forever. When a security manager is not installed, the default behavior is to cache entries for a finite (implementation dependent) period of time. The result of unsuccessful host name resolution is cached for a very short period of time (10 seconds) to improve performance. 

If the default behavior is not desired, then a Java security property can be set to a different Time-to-live (TTL) value for positive caching. Likewise, a system admin can configure a different negative caching TTL value when needed. 

Two Java security properties control the TTL values used for positive and negative host name resolution caching: 

networkaddress.cache.ttl Indicates the caching policy for successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the successful lookup. The default setting is to cache for an implementation specific period of time. 
 A value of -1 indicates "cache forever". 

networkaddress.cache.negative.ttl (default: 10)Indicates the caching policy for un-successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups. 
 A value of 0 indicates "never cache". A value of -1 indicates "cache forever". 

Unfortunately public native InetAddress[] lookupAllHostAddr(String hostname) throws UnknownHostException; is a NATIVE method, and it seems that the only exception he is capable of is UnknownHostException.. that is, even if the OS can't connect to DNS server, you get a UnknownHostException (which is totally incorrect)

Monday, September 26, 2016

List of videos to learn German - for beginners A1-B1 "Deutsch lernen Extra auf Deutsch Abschnitt 1 " there are some 20 episodes of this series, of growing difficulty. It's a funny sit-com a bit silly but enjoyable. Deutsch Plus - BBC , there are 20 episodes. VERY nice and realistic, the story of a Romanian immgrant in Germany. Deutschlandlabor by Goethe Institut - not much fun but decent. There are some 15 episodes (Folge) Typisch, also by Goethe Institut, again not too enjoyable but decent. Some 12 episodes (Folge) Mein Weg nach Deutschland , some 8 episodes, really really cool.

Wednesday, September 21, 2016

WebLogic fails with NullPointerException at weblogic.deploy.service.internal.adminserver.AdminDeploymentService$

<Sep 20, 2016 9:26:42 PM CEST> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed
        at weblogic.deploy.service.internal.adminserver.AdminDeploymentService$

We had similar cases ... always look in the weblogic_yyyy....log file for a root cause, it could be that the port used for administration is already in use, or that some security setting is messed up... you can also use "strace" to trace in detail the issue

Tuesday, September 6, 2016

WebLogic cluster: "Failed to deserialize statedump from server "

If you have an error "Failed to deserialize statedump from server " "java.lang.ClassNotFoundException"
you should read Oracle Document:

"How To Avoid De-Serialize Statedump Errors While Starting Manage Servers in Cluster Environment (java.lang.ClassNotFoundException) (Doc ID 796357.1)"

The solution is to start first the Admin, and only when this is running you should start the managed servers one by one, and not in parallel (the Oracle doc says the opposite). Don't ask me questions, I have no clue.

The other possible root cause could be an invalid EAR, try rebuilding it and redeploying it.

 Corrupted EJB files due to temporary communication failure between WebLogic admin and managed server instances

 1. Stop Weblogic Managed/Admin Server
 2.Delete the directory under $DOMAIN_HOME/servers//tmp/_WL_user/
 3.Restart Admin/Managed servers. The file will create automatically.

See also "When a server (re)joins the cluster, that server will ask another server in the cluster to provide the current view of its JNDI tree (known as a JNDI state dump) to initialize its view and then rely on JNDI replication messages to maintain it. This JNDI state dump does not use the cluster messaging protocol and relies on a point-to-point connection with the other server."

 <BEA-000140> <Failed to deserialize statedump from server 8572742343661541822S:[-1,-1,31623,31623,-1,-1,-1]:pipposerver_cluster2:pippovaimsi0_chlp2520199_server with java.lang.ClassNotFoundException: com.pippo.Pluto.
java.lang.ClassNotFoundException: com.pippo.Pluto
                at weblogic.application.internal.AppClassLoaderManagerImpl.loadApplicationClass(
                at weblogic.common.internal.ProxyClassResolver.resolveProxyClass(
                at weblogic.common.internal.WLObjectInputStream.resolveProxyClass(
                at weblogic.common.internal.WLObjectInputStream.readObjectWL(
                at weblogic.cluster.BasicServiceOffer.readExternal(
                at java.util.ArrayList.readObject(
                at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(
                at java.lang.reflect.Method.invoke(
                at weblogic.common.internal.WLObjectInputStream.readArrayList(
                at weblogic.cluster.StateDumpMessage.readExternal(

One should assign a warmup period and make sure you use default protocol t3s

Thursday, September 1, 2016

WebLogic deployment plan for a EAR with EJB deployed in a WAR and not in a separate JAR module

"To include enterprise bean class files in a WAR module, the class files should be in the WEB-INF/classes directory. To include a JAR file that contains enterprise beans in a WAR module, add the JAR to the WEB-INF/lib directory of the WAR module. WAR modules that contain enterprise beans do not require an ejb-jar.xml deployment descriptor. If the application uses ejb-jar.xml, it must be located in the WAR module’s WEB-INF directory. "

in this case you MUST put an empty weblogic-ejb-jar.xml in the WAR's file WEB-INF directory:

<weblogic-ejb-jar xmlns=""

then you can provide a deployment plan plan.xml :

<?xml version='1.0' encoding='UTF-8'?>
<deployment-plan xmlns="" xmlns:xsi="" xsi:schemaLocation="">
    <module-descriptor external="false">
    <module-descriptor external="false">

and in the config.xml you configure the deployment plan: