ulimit and lsof

PID of WLS is 29518
to count how many file descriptors are held by a process:
ls /proc/29518/fd | wc -l
666

funnily this doesn’t match the count reported by lsof ( one day I will discover why):

lsof -p 29518 | wc -l
1288

to know the limit for the specific process:

cat /proc/29518/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 32000 32000 processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 515304 515304 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

which is different for the limit set by default for a user:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515304
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

To set the limit for a process open by the current shell :

ulimit -n MAX_FD_PER_PROCESS


In Linux, this is a PER-PROCESS limit! You cannot directly limit the number of Files opened by a User! You can only set a max-processes-per-user with ulimit -u MAX .
to check the actual limits for a process:
cat /proc//limits
cat /etc/security/limits.conf to check the hard limits for all users. If you try to set a ulimit larger than that. you get a ulimit: open files: cannot modify limit: Operation not permitted

java.lang.Runtime.getRuntime().exec leaking pipes

I do this in a WebApp in Tomcat:

Process proc = java.lang.Runtime.getRuntime().exec("generateSQLDeveloperConnections.sh");
proc.waitFor();

but apparently according to this post: http://mark.koli.ch/2011/01/leaky-pipes-remember-to-close-your-streams-when-using-javas-runtimegetruntimeexec.html this can leak resources.

To investigate, I run the PRICELESS lsof on the Tomcat pid 12193:

/usr/sbin/lsof -a -p 12193

and I notice after each run of getRuntime().exec 3 extra pipes:

java    12193  soa   76w  FIFO      0,6      0t0 14925870 pipe
java    12193  soa   77r  FIFO      0,6      0t0 14925871 pipe
java    12193  soa   79r  FIFO      0,6      0t0 14925872 pipe

who are never closed.

I will definitely add the code

import static org.apache.commons.io.IOUtils.closeQuietly;

Process p = null;
try {
  p = Runtime.getRuntime().exec(...);
  // Do something with p.
} finally {
  if(p != null) {
    closeQuietly(p.getOutputStream());
    closeQuietly(p.getInputStream());
    closeQuietly(p.getErrorStream());
  }
}

and see if this takes care of the issue.

OSB, number of files open

We run an environment with Whole Server Migration.

An important point is to make sure that we have enough resources (file handles) on a machine to be able to run 2 Managed Server - in case we migrate one.

This is how you discover how many files are used by the user "soa":

/usr/sbin/lsof -u soa | wc -l

It turned out that a single server running the Admin and 4 Managed Servers required some 9000 files. An important load was represented by the MANY Datasources we need to deploy.

This command:
ulimit -a

will tell you the hard limit imposed by the Linux Kernel:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 132607
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 132607
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


apparently the limit is flexible, even with a limit of 8192 we were able to open 9050 files...

lsof to find which process is keeping a socket open

I know I must already have blogged this...

If the port is 10011, do this:

/usr/sbin/lsof -nl | egrep "TCP|UDP" | grep 10011

COMMAND     PID     USER   FD   TYPE             DEVICE   SIZE/OFF    NODE NAME

java      10747      574   13u  IPv4 0xe0000004a31e6680        0t0     TCP 127.0.0.1:55785->127.0.0.1:10011 (ESTABLISHED)
java      10747      574   15u  IPv4 0xe0000004ab25aa00   0t296774     TCP 10.3.255.175:53916->10.3.255.175:10011 (ESTABLISHED)
java      10748      574   13u  IPv4 0xe0000004638b6d00     0t9957     TCP 10.3.255.175:53927->10.3.255.175:10011 (ESTABLISHED)
java      10748      574   15u  IPv4 0xe000000462b22100        0t0     TCP 127.0.0.1:55786->127.0.0.1:10011 (ESTABLISHED)
java      10749      574   13u  IPv4 0xe000000460469100        0t0     TCP 127.0.0.1:55787->127.0.0.1:10011 (ESTABLISHED)
java      10749      574   15u  IPv4 0xe0000004c2dce380   0t298907     TCP 10.3.255.175:53921->10.3.255.175:10011 (ESTABLISHED)
java      10750      574   13u  IPv4 0xe000000460f47680   0t300255     TCP 10.3.255.175:53935->10.3.255.175:10011 (ESTABLISHED)
java      10750      574   15u  IPv4 0xe000000462b22400    0t34979     TCP 127.0.0.1:54761->127.0.0.1:10011 (ESTABLISHED)
java      21200      574  597u  IPv4 0xe000000497104380     0t1688     TCP 10.3.255.168:55375->10.3.255.168:10011 (ESTABLISHED)
java      27806      574  342u  IPv4 0xe000000483b10400     0t2961     TCP 10.3.255.168:10011->10.3.255.168:55375 (ESTABLISHED)
java      27806      574  346u  IPv4 0xe00000049fe42a00     0t2956     TCP 127.0.0.1:10011->127.0.0.1:54761 (ESTABLISHED)
java      27806      574  351u  IPv4 0xe000000482c52980      0t220     TCP 127.0.0.1:10011->127.0.0.1:55785 (ESTABLISHED)
java      27806      574  356u  IPv4 0xe000000449aebc80        0t0     TCP 10.3.67.168:10011 (LISTEN)
java      27806      574  363u  IPv4 0xe000000462cfc100        0t0     TCP 10.3.255.175:10011 (LISTEN)
java      27806      574  364u  IPv4 0xe0000004ac2f7c80        0t0     TCP 10.3.255.168:10011 (LISTEN)
java      27806      574  365u  IPv4 0xe0000004ad028400        0t0     TCP 127.0.0.1:10011 (LISTEN)
java      27806      574  366u  IPv4 0xe0000004bf415980        0t0     TCP 10.3.255.174:10011 (LISTEN)
java      27806      574  367u  IPv4 0xe0000004d24b3400      0t220     TCP 127.0.0.1:10011->127.0.0.1:55786 (ESTABLISHED)
java      27806      574  368u  IPv4 0xe0000004bf415680        0t0     TCP 10.3.255.179:10011 (LISTEN)
java      27806      574  369u  IPv4 0xe00000049e0a3680      0t220     TCP 127.0.0.1:10011->127.0.0.1:55787 (ESTABLISHED)
java      27806      574  378u  IPv4 0xe0000004638b6700   0t244556     TCP 10.3.255.175:10011->10.3.255.175:53916 (ESTABLISHED)
java      27806      574  379u  IPv4 0xe0000001ef703980   0t247992     TCP 10.3.255.175:10011->10.3.255.175:53921 (ESTABLISHED)
java      27806      574  382u  IPv4 0xe0000004bbae2c80    0t10277     TCP 10.3.255.175:10011->10.3.255.175:53927 (ESTABLISHED)
java      27806      574  397u  IPv4 0xe00000046080da00   0t246284     TCP 10.3.255.175:10011->10.3.255.175:53935 (ESTABLISHED)
java      27806      574  397u  IPv4 0xe00000046080da00   0t246284     TCP 10.3.255.175:10011->10.3.255.175:53935 (ESTABLISHED)

you can see that the same socket is reported twice, once on the receiving side and the other on the sending side

Here you can find a splendid tutorial on lsof.

too many open files

weblogic.socket.SocketOptionException: Too many open files
weblogic.socket.SocketOptionException: Too many open files
at weblogic.socket.SocketMuxer.initSocket(SocketMuxer.java:431)
at weblogic.socket.SocketMuxer.newSocket(SocketMuxer.java:310)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:82)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:58)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:355)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:434)
at weblogic.net.http.HttpClient.(HttpClient.java:159)
at weblogic.net.http.HttpClient.(HttpClient.java:149)
at weblogic.net.http.HttpURLConnection.getHttpClient(HttpURLConnection.java:190)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:414)
at weblogic.rjvm.http.HTTPClientJVMConnection.receiveAndDispatch(HTTPClientJVMConnection.java:404)
at weblogic.rjvm.http.HTTPClientJVMConnection.run(HTTPClientJVMConnection.java:295)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)



start by logging with the same user which runs weblogic, and do
ulimit -a

you should see for sockets a limit of 1024

you can change it by

ulimit -n 2048

if you do lsof you can get an idea of who is eating those resources.
bear in mind that file handles and sockets are considered as same.


You can also:
su - (you become root)
vi /etc/security/limits.conf

insert:
* soft nofile 2048
* hard nofile 2048


and do:
sysctl -p
to reload the configuration