UnknownHostException returned by DNS, in reality due to not enough file descriptors available

interesting case, intermittently we get this error:

java.net.UnknownHostException: somehostnamehere
                at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
                at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:922)
                at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1316)
                at java.net.InetAddress.getAllByName0(InetAddress.java:1269)
                at java.net.InetAddress.getAllByName(InetAddress.java:1185)
                at java.net.InetAddress.getAllByName(InetAddress.java:1119)
                at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
                at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:102)
                at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
                at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
                

https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/SystemDefaultDnsResolver.html

"DNS resolver that uses the default OS implementation for resolving host names"

public InetAddress[] resolve(String host) throws UnknownHostException 
{ return InetAddress.getAllByName(host); } 

https://docs.oracle.com/javase/7/docs/api/java/net/Inet4Address.html

https://docs.oracle.com/javase/7/docs/api/java/net/InetAddress.html#getAllByName(java.lang.String)

InetAddress Caching 
 The InetAddress class has a cache to store successful as well as unsuccessful host name resolutions. 
 By default, when a security manager is installed, in order to protect against DNS spoofing attacks, the result of positive host name resolutions are cached forever. When a security manager is not installed, the default behavior is to cache entries for a finite (implementation dependent) period of time. The result of unsuccessful host name resolution is cached for a very short period of time (10 seconds) to improve performance. 

If the default behavior is not desired, then a Java security property can be set to a different Time-to-live (TTL) value for positive caching. Likewise, a system admin can configure a different negative caching TTL value when needed. 

Two Java security properties control the TTL values used for positive and negative host name resolution caching: 

networkaddress.cache.ttl Indicates the caching policy for successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the successful lookup. The default setting is to cache for an implementation specific period of time. 
 A value of -1 indicates "cache forever". 

networkaddress.cache.negative.ttl (default: 10)Indicates the caching policy for un-successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups. 
 A value of 0 indicates "never cache". A value of -1 indicates "cache forever". 

Unfortunately public native InetAddress[] lookupAllHostAddr(String hostname) throws UnknownHostException; is a NATIVE method, and it seems that the only exception he is capable of is UnknownHostException.. that is, even if the OS can't connect to DNS server, you get a UnknownHostException (which is totally incorrect)

Tools to simulate network disruptions

On Windows: Clumsy https://jagt.github.io/clumsy/
On Linux, netem https://wiki.linuxfoundation.org/networking/netem

BEA-000449 Closing the socket, as no data read from it on 1.2.3.4:5,6789

BEA-000449 Closing the socket, as no data read from it on 1.2.3.4:5,6789 during the configured idle timeout of 5 seconds

1.2.3.4:5,6789 represents an IP (1.2.3.4) and port number 5,6789 (I find irritating that then print the , as separator....)

This message can be ignored NORMALLY (use a log filter is you like), and the IP addresses most likely are Load Balancer IPs. It means PROBABLY that a user doesn't wait for a web page to be entirely loaded, and navigates away closing abruptly the current socket transfer. The "configured idle timeout" however identifies a special case of "login timeout" kicking in (see below)

Read "Error Logs Say "Warning Socket BEA-000449 Closing socket as no data read" (Doc ID 2051032.1)" oracle document, saying that

"WebLogic Server tries to reuse sockets to improve performance, but sockets which are idle for a specified period are closed. The length of this period is controlled by the weblogic.client.socket.ConnectTimeout parameter, which specifies the amount of time the server waits before closing an inactive HTTP connection. This is set in the WebLogic Server startup script as one of the JAVA_OPTIONS. For example:

-Dweblogic.client.socket.ConnectTimeout=XXX"

Surely, if 5 seconds is too little , you can change it! Probably 5 s comes from the configuration value of login timeout :

"config / turning / login time out : (default is 5000ms)" "The login timeout for this server's default regular (non-SSL) listen port. This is the maximum amount of time allowed for a new connection to establish." ServerMBean.LoginTimeoutMillis

WebLogic network-access-point

If you need to invoke operations (EJB, WS...) on a specific IP different from the main listen address / port of WLS, you can create inside config.xml a network-access-point and give it a mnemonic name like "INT-Channel" :

   <network-access-point>
      <name>INT-Channel</name>
      <protocol>t3s</protocol>
      <listen-address>1.2.3.4</listen-address>
      <enabled>true</enabled>
      <two-way-ssl-enabled>true</two-way-ssl-enabled>
      <client-certificate-enforced>true</client-certificate-enforced>
    </network-access-point> 

and configure your component in your weblogic-ejb-jar.xml with a clause:

<weblogic-enterprise-bean>
     <network-access-point>INT-Channel</network-access-point>
</weblogic-enterprise-bean>

see https://docs.oracle.com/cd/E11035_01/wls100/ejb/DDreference-ejb-jar.html#network-access-point

VM to VM communication in VirtualBox (Linux)

(amazing how many examples on youtube are done with Weirdos... few of them use Linux...)

The official documentation says: https://www.virtualbox.org/manual/ch06.html
"NAT: by default virtual machines cannot talk to each other"

Problem is that most of the time I start multiple VM on my host with the purpose of making them communicate with each other! So am I screwed?

An option which looks promising is Internal Networking : "Internal Networking is similar to bridged networking in that the VM can directly communicate with the outside world. However, the "outside world" is limited to other VMs on the same host which connect to the same internal network.". One could use also " bridged networking " but with Internal you don't need an adapter. One Internal Network is available: intnet. But there is a HUGE limitation: "As a security measure, the Linux implementation of internal networking only allows VMs running under the same user ID to establish an internal network." . Bummer.

Frankly "Host-only networking " seems the most promising. it was designed exactly to let multiple appliances cooperate.

I usually use Host-only networking and give "real" IPs to my machines and they see each other without problem.

With Vagrant you can just use something like this:

config.vm.network "private_network", ip: "192.168.58.101"

And you will be able to connect from host or from any other guest using 192.168.58.101 and the good thing is that IP will only be visible on your machine.



Read these great presentations:
https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1
https://technology.amis.nl/2014/01/27/a-short-guide-to-networking-in-virtual-box-with-oracle-linux-inside/

HTTP GET in Java, performance issues

My first version was:

import java.net.*;

URL url = new URL(theURL);
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String wholeInput = "";
String inputLine = null;
while ((inputLine = in.readLine()) != null) 

{ wholeInput += inputLine; }

The above code proved to be EXTREMELY slow. The below version is MUCH faster:

URL url = new URL(theURL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String wholeInput = "";
String inputLine = null;
while ((inputLine = in.readLine()) != null) 
 { wholeInput += inputLine; }
in.close(); 

Configuring NIC network interface on RHEL

Note: ipconfig is deprecated in favour of ip, however a lot of legacy code still uses it.
Very interesting tutorial here and here
This is a UI to edit all configuration without requiring root privileges, bur probably if you are not root you won'et even be able to see any configuration values - forget about changing them:

/usr/sbin/system-config-network-tui

This is what you get:

ââââââââââ⤠Network Configuration âââââââââââ
â                                           â
â                                           â
â Name                 eth0________________ â
â Device               eth0________________ â
â Use DHCP             [*]                  â
â Static IP            ____________________ â
â Netmask              ____________________ â
â Default gateway IP   ____________________ â
â Primary DNS Server   ____________________ â
â Secondary DNS Server ____________________ â
â                                           â
â       ââââââ            ââââââââââ        â
â       â Ok â            â Cancel â        â
â       ââââââ            ââââââââââ        â
â                                           â
â                                           â
âââââââââââââââââââââââââââââââââââââââââââââ

this is equivalent, but requires root privileges:

/usr/bin/system-config-network

They don't require a X-terminal session, they support a text-ui mode.

To manually hack the configuration:
cd /etc/sysconfig/network-scripts/
less ifcfg-eth0

DEVICE="eth0"
BOOTPROTO="dhcp"
HWADDR="08:00:27:29:06:6F"
IPV6INIT="yes"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="ca6ef823-ed13-46ad-8a83-9d688f6cf239"

Some other "global" info is here:
vi /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=osb-vagrant.acme.com

"/sbin/ip -o addr" displays all IPs
to add an extra IP, you must specify an interface eth0:N where N > 0:

/sbin/ifconfig eth0:1 10.0.2.16 netmask 255.255.255.0

(there is an implicit "up" at the end)
Entering "/sbin/ifconfig eth0 10.0.2.16 netmask 255.255.255.0" will crash the NIC, removing the current primary address to replace with the new one.

Consensus-based server migration: caveat

I used to have a Database-based lease mechanism for Server Migration, but occasionally it was failing ("unable to contact DB", no clue why...) and the server would restart itself.
We changed to Consensus, hoping the network would be more robust. However, due to a network reconfiguration, some IPs were left undefined and the cluster broke:

<[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <f56960fb001bca18:-33f939f3:1413c1039a1:-8000-0000000000065ea0> <1380619475997> <BEA-000802> <ExecuteRequest failed
java.lang.AssertionError: Invalid state transition from failed to stable_leader.
java.lang.AssertionError: Invalid state transition from failed to stable_leader
        at weblogic.cluster.leasing.databaseless.ClusterState.setState(ClusterState.java:100)
        at weblogic.cluster.leasing.databaseless.ClusterState.setState(ClusterState.java:59)
        at weblogic.cluster.leasing.databaseless.ClusterFormationServiceImpl.leaderInitialization(ClusterFormationServiceImpl.java:318)
        at weblogic.cluster.leasing.databaseless.ClusterFormationServiceImpl.formClusterInternal(ClusterFormationServiceImpl.java:148)
        at weblogic.cluster.leasing.databaseless.ClusterFormationServiceImpl.timerExpired(ClusterFormationServiceImpl.java:339)
        at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>

The issue is that once the network has been fixed, the cluster didn't recover and we had to restart the servers... however this could simply be because when we restart the server, the Virtual IP associated to each server is readded to the NIC (/sbin/ifconfig -addif). Instead of restarting the servers I should have tried to add the IP manually... one should really monitor continuously the availability of those IPs...

Poor man's firewall test

on the Destination host:

nc -l myhost.acme.com 3872

and make sure you are actually listening:

netstat -an | grep 3872
tcp        0      0 10.33.80.121:3872           0.0.0.0:*                   LISTEN

On the Source host:

echo ciao | nc myhost.acme.com 3872

and the "ciao" should appear on Destination and the nc should exit.

If you don't have nc installed, there are alternatives to nc:

wlst or python:

import socket
HOST = 'myhost.acme.com'
PORT = 3872
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.connect((HOST, PORT)) 

s.send('Hello, world')
data = s.recv(1024)
s.close()

(see http://docs.python.org/release/2.5.2/lib/socket-example.html)

or simply  run

telnet myhost.acme.com 3872

To receive data, run Java or python:

from java.net import ServerSocket

ss = ServerSocket(3872)

ss.accept()

(see http://docs.oracle.com/javase/6/docs/api/java/net/ServerSocket.html )

The great advantage of nc is that you can bind to any IP on the source host:

nc -s "your_ip_here"

To check if nc could actually connect, do:
echo ciao | nc....
echo $?

1 means "unable to connect", 0 means "connected"

echo a | nc -s "10.26.20.116" -w 1 10.51.87.24 1722 ; echo $?

A script to check firewall could very well be:

#!/bin/sh
#This script is to check that a firewall rule is operational
#Author name : Pierluigi Vernetto


function checkFirewall {
 sourceIPsArray=$(echo $sourceIPs | tr "," "\n")
 destinationIPsArray=$(echo $destinationIPs | tr "," "\n")
 for sourceIP in $sourceIPsArray 
 do
        for destinationIP in $destinationIPsArray
        do
            echo a | nc -s "$sourceIP" -w 2 $destinationIP $port
            if [[ $? -eq 0 ]] 
             then echo $sourceIP $destinationIP $port success
             else echo $sourceIP $destinationIP $port failure
            fi   
        done
    done
}

sourceIPs=10.56.218.91,10.56.218.93,10.56.218.90,10.56.218.94,10.56.218.92
destinationIPs=10.56.128.10,10.56.128.8,10.56.128.9
port=1522

checkFirewall

tracert tutorial

http://en.wikipedia.org/wiki/Traceroute

Wikipedia pages to read if you want to become a network expert

http://en.wikipedia.org/wiki/Internet_Control_Message_Protocol

http://en.wikipedia.org/wiki/Nmap

http://en.wikipedia.org/wiki/Media_Access_Control

http://en.wikipedia.org/wiki/Tracert

http://en.wikipedia.org/wiki/Ping_%28networking_utility%29

http://en.wikipedia.org/wiki/Ifconfig

http://en.wikipedia.org/wiki/Nslookup

http://en.wikipedia.org/wiki/Netstat

http://en.wikipedia.org/wiki/Lsof

http://en.wikipedia.org/wiki/Subnet_mask

http://en.wikipedia.org/wiki/Default_gateway

http://en.wikipedia.org/wiki/Routing_table

http://en.wikipedia.org/wiki/DHCP


interesting Linux Networking Manual http://www.faqs.org/docs/linux_network/index.html
Basic intro to several commands

ntop tutorial

http://en.wikipedia.org/wiki/Ntop

tcpdump tutorial

tcpdump -D
tcpdump -i eth0 v
tcpdump -i eth0 port http (to display only http traffic)
tcpdump -n (to display numeric IPs)
tcpdump -w session filename.log (captures data to a file)
tcpdump -r session filename.log (displays the content of the captured session)
This is a very concise excellent quick guide of the main commands:
http://openmaniak.com/tcpdump.php
To use wireshark to view packets captured with tcpdump:
tcpdump -i -s 65535 -w

Tcpdum for Windows available here

Wireshark tutorial

and also this one is very informative

Windows mapping network drives

Open Windows Explorer, Tools, Map Netword Drives,
enter FOLDER (eg \\acme1999\PIPPOPPRD) and click on "Connect Using a different user name", enter username and pw

If you get this message:

"the network folder specified is currently mapped using a different name and pw"


then open a command prompt, enter

net use

if you see \\acme1999\PIPPOPPRD in the list, issue a

net use \\acme1999\PIPPOPPRD /delete

and try again

Welcome to the wonderful world of Widows (TM).

Network performance tools

nmon http://en.wikipedia.org/wiki/Nmon

download nmon at http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon

movie on nmon http://download.boulder.ibm.com/ibmdl/pub/systems/power/community/aix/AIX_Movies/nmon_Intro.wmv

Satan seems obsolete, and it's mostly for network security than for performance benchmarking.

http://en.wikipedia.org/wiki/Measuring_network_throughput

Good tutorial on networks:

http://iptables-tutorial.frozentux.net/iptables-tutorial.html#TCPMSSTARGET