The specified size exceeds the maximum representable size

If you run this:

java -Xms4096m Bla

and you get this:

Invalid initial heap size: -Xms4096m
The specified size exceeds the maximum representable size

it simply means that you are using a 32bit JVM instead of a 64bit JVM.

check "java -d64"

java -d64 -version Error: This Java instance does not support a 64-bit JVM. Please install the desired version.

Too small initial heap for new size specified

You must provide a max total heap > min new heap.
This is fine:

java -XX:NewSize=256m -Xmx257m -version 
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.8) (rhel-1.22.1.9.8.el5_6-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

This is not fine, as no space is left for Old generation

java -XX:NewSize=256m -Xmx256m -version 
Error occurred during initialization of VM
Too small initial heap for new size specified

See http://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html for more documentation

I know this is trivial, but many people don't know.

Could not create the Java Virtual Machine

cat Test.java 
public class Test {
 public static void main(String[] args) {
        System.out.println("ciao");
 }

}
javac Test.java
export PIPPO="ciao - bello"
java -Dpippo=$PIPPO Test
Unrecognized option: -
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

This is how a simple space in the value of a property can mess up your JVM.
if you run this
java -Dpippo="$PIPPO" Test
(with additional quotes) all works fine!

How to view your JVM default settings

cat Test.java 

public class Test {
   public static void main(String[] args) {
        for (;;) {
        }

   } 

}

javac Test.java
java -XX:+PrintCommandLineFlags Test
you should get something like this:
-XX:InitialHeapSize=67108864 -XX:MaxHeapSize=1073741824 -XX:+PrintCommandLineFlags -XX:+UseParallelGC

JVM Parameters made easy

I have dreaming for many years of setting up such a tool:
http://jvmmemory.com/
to provide a guided wizard to configure all aspects of JVM.... good to see that it's already there! Great job!

No shared spaces configured

Heap
garbage-first heap   total 6291456K, used 1368757K [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000)
  region size 2048K, 97 young (198656K), 9 survivors (18432K)
compacting perm gen  total 1048576K, used 230859K [0x00000007c0000000, 0x0000000800000000, 0x0000000800000000)
   the space 1048576K,  22% used [0x00000007c0000000, 0x00000007ce172e10, 0x00000007ce173000, 0x0000000800000000)
No shared spaces configured.

this was seen in a

java/jdk170_91-64b/bin/java -XX:-UseBiasedLocking -XX:SurvivorRatio=10 -verbose:gc -XX:+PrintGCTimeStamps -Xms6144m -Xmx6144m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=1024m -XX:MaxPermSize=1024m  -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=60 

I am not really sure what this "No shared spaces configured" would mean - I could not find any decent documentation on this message. And it doesn't seem to be really an error message, nor something related to G1GC...

BEA-2165021 ResourceManagement is not enabled in this JVM

Starting WLS 12.2.1 with a JDK 8 I get this INFO message:

<Info> <RCM> <BEA-2165021> <"ResourceManagement" is not enabled in this JVM. Enable "ResourceManagement" to use the WebLogic Server "Resource Consumption Management" feature. To enable "ResourceManagement", you must specify the following JVM options in the WebLogic Server instance in which the JVM runs: -XX:+UnlockCommercialFeatures -XX:+ResourceManagement.>

here https://docs.oracle.com/javase/8/docs/technotes/tools/enhancements-8.html they say "##The options -XX:+ResourceManagement and -XX:ResourceManagementSampleInterval=value have been added. Resource Management is a commercial feature."

I think this has nothing to do with the try-with-resources clause.

Euphemistically speaking, there doesn't seem to be lot of documentation on this -XX:+ResourceManagement flag...

Useful JVM flags, revisited

This guy Patrick Peschlow (possibly a superhuman, a hero.... most likely a German, only Germans can love so much classifications) did an amazing job at illustrating all (most of) the JVM flags.
He explained it all in such a clear and precise way that I was about to cry for the emotion.

https://blog.codecentric.de/en/2012/07/useful-jvm-flags-part-1-jvm-types-and-compiler-modes/

• server and -client
• version and -showversion
• -Xint, -Xcomp, and -Xmixed

https://blog.codecentric.de/en/2012/07/useful-jvm-flags-part-2-flag-categories-and-jit-compiler-diagnostics/

• JVM flag categories
• -XX:+PrintCompilation and -XX:+CITime
• -XX:+UnlockExperimentalVMOptions
• -XX:+LogCompilation and -XX:+PrintOptoAssembly

https://blog.codecentric.de/en/2012/07/useful-jvm-flags-part-3-printing-all-xx-flags-and-their-values/

• -XX:+PrintFlagsFinal and -XX:+PrintFlagsInitial
• -XX:+PrintCommandLineFlags

https://blog.codecentric.de/en/2012/07/useful-jvm-flags-part-4-heap-tuning/

• -Xms and -Xmx (or: -XX:InitialHeapSize and -XX:MaxHeapSize)
• -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath
• -XX:OnOutOfMemoryError
• -XX:PermSize and -XX:MaxPermSize
• -XX:InitialCodeCacheSize and -XX:ReservedCodeCacheSize
• -XX:+UseCodeCacheFlushing

https://blog.codecentric.de/en/2012/08/useful-jvm-flags-part-5-young-generation-garbage-collection/

• -XX:NewSize and -XX:MaxNewSize
• -XX:SurvivorRatio
• -XX:+PrintTenuringDistribution
• -XX:InitialTenuringThreshold, -XX:MaxTenuringThreshold and -XX:TargetSurvivorRatio
• -XX:+NeverTenure and -XX:+AlwaysTenure

https://blog.codecentric.de/en/2013/01/useful-jvm-flags-part-6-throughput-collector/

• Throughput vs. pause times
• Garbage collection on the HotSpot JVM
• -XX:+UseSerialGC
• -XX:+UseParallelGC
• -XX:+UseParallelOldGC
• -XX:ParallelGCThreads
• -XX:-UseAdaptiveSizePolicy
• -XX:GCTimeRatio
• -XX:MaxGCPauseMillis

https://blog.codecentric.de/en/2013/10/useful-jvm-flags-part-7-cms-collector/

• -XX:+UseConcMarkSweepGC
• -XX:+UseParNewGC
• -XX:+CMSConcurrentMTEnabled
• -XX:ConcGCThreads
• -XX:CMSInitiatingOccupancyFraction
• -XX+UseCMSInitiatingOccupancyOnly
• -XX:+CMSClassUnloadingEnabled
• -XX:+CMSIncrementalMode
• -XX:+ExplicitGCInvokesConcurrent and -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses
• -XX:+DisableExplicitGC

https://blog.codecentric.de/en/2014/01/useful-jvm-flags-part-8-gc-logging/

• -XX:+PrintGC
• -XX:+PrintGCDetails
• -XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps
• -Xloggc

More References:

http://stas-blogspot.blogspot.de/2011/07/most-complete-list-of-xx-options-for.html

Virtual Machine Garbage Collection Tuning https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/

JVM Mechanics by Douglas Hawkins: awesome presentation

JVM Mechanics by Douglas Hawkins as Presented at Silicon Valley Java User Group by Azul Systems from Azul Systems on Vimeo.

NUMA and JVM performance

Apparently there is a 40% performance gain to be made configuring the JVM to take advantage of the NUMA architecture. Just pass the -XX:+UseNUMA flag to your JVM.

To check if your hardware is NUMA enabled, run numactl:

numactl --hardware

For instance I get

available: 1 nodes (0)
node 0 size: 15897 MB
node 0 free: 344 MB
node distances:
node   0 
  0:  10 

which probably means I have no NUMA since I have a single node. You can also run numastat.

It's interesting also to run taskset:

taskset -c -p 5794

pid 5794's current affinity list: 0-3

to determine to which CPUs a given process is associated. Apparently this is quite important for memory allocation/deallocation performance - I still don't understand how :o(

See also this http://www.glennklockwood.com/comp/affinity.php

JVM Troubleshooting Guide

I am going through a couple of interesting documents:

This one contains many word of wisdom about strategic approach to parametrization of the JVM...

http://www.javacodegeeks.com/wp-content/uploads/2013/06/JVM_Troubleshooting_Guide.pdf

for instance it's valuable the advice to distinguish applications where mostly short-lived object are created (most Service Bus applications are like that) from Applications which keep long lived objects - for instance HTTP sessions. In the first case, you should adjust the Heap to give more space to the Young generation space. In the second case, give more space to the Tenured space.



This document also contains valuable tips and tools:

http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/toc.html

Java JVM Flag PrintConcurrentLocks

When all hell break loose and you have threads hanging waiting for a lock, it's PARAMOUNT to be able to determine WHO is holding that lock.


By default, the option -XX:+PrintConcurrentLocks is not enabled. if you enable it, and you do a "kill -3 PID", you should get for each thread the list of locks being held. This option "should be" safe in PROD, despite of the Oracle warnings.

However, you can get the same info using

jstack -l PID

"Reference Handler" daemon prio=10 tid=0x000000004e8b6000 nid=0x48c8 in Object.wait() [0x000000004104b000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0000000782f36550> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x0000000782f36550> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
        - None

The "Locked ownable synchronizers:" bit is the one you would get extra by using the PrintConcurrentLocks flag.

Anyway this is only a superficial analysis... luckily working with OSB I never had to deal with locking issues... so I cannot claim to be an expert.

Java, discover all available flags with PrintFlagsFinal

not many people know that you can get the list of all supported JVM flags with -XX:+PrintFlagsFinal
example:
java -XX:+PrintFlagsFinal -version | grep -i gc
will print:

    uintx AdaptiveSizeMajorGCDecayTimeScale         = 10              {product}
    uintx AutoGCSelectPauseMillis                   = 5000            {product}
     bool BindGCTaskThreadsToCPUs                   = false           {product}
    uintx CMSFullGCsBeforeCompaction                = 0               {product}
    uintx ConcGCThreads                             = 0               {product}
     bool DisableExplicitGC                         = false           {product}
     bool ExplicitGCInvokesConcurrent               = false           {product}
     bool ExplicitGCInvokesConcurrentAndUnloadsClasses  = false           {product}
    uintx GCDrainStackTargetSize                    = 64              {product}
    uintx GCHeapFreeLimit                           = 2               {product}
    uintx GCLockerEdenExpansionPercent              = 5               {product}
     bool GCLockerInvokesConcurrent                 = false           {product}
     bool GCOverheadReporting                       = false           {product}
     intx GCOverheadReportingPeriodMS               = 100             {product}
    uintx GCPauseIntervalMillis                     = 0               {product}
    uintx GCTaskTimeStampEntries                    = 200             {product}
    uintx GCTimeLimit                               = 98              {product}
    uintx GCTimeRatio                               = 99              {product}
     bool HeapDumpAfterFullGC                       = false           {manageable}
     bool HeapDumpBeforeFullGC                      = false           {manageable}
    uintx MaxGCMinorPauseMillis                     = 18446744073709551615{product}
    uintx MaxGCPauseMillis                          = 18446744073709551615{product}
     bool OptimizeStringConcat                      = false           {C2 product}
     intx ParGCArrayScanChunk                       = 50              {product}
    uintx ParGCDesiredObjsFromOverflowList          = 20              {product}
     bool ParGCTrimOverflow                         = true            {product}
     bool ParGCUseLocalOverflow                     = false           {product}
     intx ParallelGCBufferWastePct                  = 10              {product}
     bool ParallelGCRetainPLAB                      = true            {product}
    uintx ParallelGCThreads                        := 10              {product}
     bool ParallelGCVerbose                         = false           {product}
     intx PerfMaxStringConstLength                  = 1024            {product}
     bool PrintClassHistogramAfterFullGC            = false           {manageable}
     bool PrintClassHistogramBeforeFullGC           = false           {manageable}
     bool PrintGC                                   = false           {manageable}
     bool PrintGCApplicationConcurrentTime          = false           {product}
     bool PrintGCApplicationStoppedTime             = false           {product}
     bool PrintGCDateStamps                         = false           {manageable}
     bool PrintGCDetails                            = false           {manageable}
     bool PrintGCTaskTimeStamps                     = false           {product}
     bool PrintGCTimeStamps                         = false           {manageable}
     bool PrintHeapAtGC                             = false           {product rw}
     bool PrintHeapAtGCExtended                     = false           {product rw}
     bool PrintJNIGCStalls                          = false           {product}
     bool PrintParallelOldGCPhaseTimes              = false           {product}
     bool PrintReferenceGC                          = false           {product}
     intx QueuedAllocationWarningCount              = 0               {product}
     bool ScavengeBeforeFullGC                      = true            {product}
     bool SpecialStringCompareToCC                  = true            {product}
     bool SpecialStringCompress                     = true            {product}
     bool TraceParallelOldGCTasks                   = false           {product}
     bool UseAdaptiveGCBoundary                     = false           {product}
     bool UseAdaptiveSizeDecayMajorGCCost           = true            {product}
     bool UseAdaptiveSizePolicyWithSystemGC         = false           {product}
     bool UseAutoGCSelectPolicy                     = false           {product}
     bool UseConcMarkSweepGC                        = false           {product}
     bool UseG1GC                                   = false           {product}
     bool UseGCOverheadLimit                        = true            {product}
     bool UseGCTaskAffinity                         = false           {product}
     bool UseMaximumCompactionOnSystemGC            = true            {product}
     bool UseParNewGC                               = false           {product}
     bool UseParallelGC                            := true            {product}
     bool UseParallelOldGC                          = false           {product}
     bool UseParallelOldGCCompacting                = true            {product}
     bool UseParallelOldGCDensePrefix               = true            {product}
     bool UseSerialGC                               = false           {product}
     bool UseStringCache                            = false           {product}

Nice-to-activate Sun JVM flags for GC and ThreadDumps

-Xloggc:ms1gc.log

http://stackoverflow.com/questions/1161647/how-to-redirect-verbose-garbage-collection-output-to-a-file



-Xloggc is like -Xverbose:gc but it logs in a separate file ms1gc.log

It is safe to activate in production (unlike -Xprof which is quite taxing)


This is a list of options for the JVM:

http://download.oracle.com/javase/6/docs/technotes/tools/windows/java.html


Sample test of –Xloggc:



cat DoNothing.java

public class DoNothing {

   public static void main(String[] args) throws Exception {

       for (;;) {

           Thread.sleep(1000);
           int zillion = 1000000;
           // just wasting memory here
           Object[] waste = new Object[zillion];
           for (int i = 0; i < zillion; i++) {
              waste[i] = new Object();
           }

          // yield CPU, don’t be greedy
           System.out.println("I am just sleeping... how lovely....");
       }
   }
}

javac DoNothing.java java -Xloggc:mygc.log DoNothing cat mygc.log

5.629: [GC 47048K->120K(187712K), 0.0045328 secs]

9.769: [GC 47016K->128K(187712K), 0.0015663 secs]

13.889: [GC 47024K->136K(187712K), 0.0011926 secs]

18.009: [GC 47032K->136K(187712K), 0.0011626 secs]

22.129: [GC 47032K->136K(178560K), 0.0056787 secs]

26.259: [GC 47032K->136K(177408K), 0.0011921 secs]

29.371: [GC 46792K->11618K(187584K), 0.4040244 secs]

34.925: [GC 68450K->21588K(226688K), 0.3109225 secs]

43.478: [GC 111764K->22196K(229824K), 0.1426002 secs]

51.857: [GC 109907K->16082K(284544K), 0.0322680 secs]

64.259: [GC 156772K->16082K(284672K), 0.1478636 secs]

76.779: [GC 156772K->16082K(370560K), 0.0095127 secs]

96.393: [GC 246994K->24276K(370816K), 0.3453224 secs]

117.345: [GC 255188K->20740K(356672K), 0.0354717 secs]

136.948: [GC 238833K->16082K(343040K), 0.0122887 secs]

155.509: [GC 227101K->16082K(334464K), 0.0144088 secs]

Be aware that every time the JVM is started the previous mygc.log is overwritten. The very ridiculous thing is that there is no readable timestamp associated with the GC event….how silly! The times are reported as milliseconds from the JVM start time! So when we interpret these results we should first determine the start time of the JVM (this should not be an issue…) -XX:+PrintGCDetails -XX:+PrintGCDateStamps I have tried both, and PrintGCDetails is supported on Itanium:

5.665: [GC [PSYoungGen: 47048K->136K(57152K)] 47048K->136K(187712K), 0.0024533 secs]

9.795: [GC [PSYoungGen: 47032K->128K(57152K)] 47032K->128K(187712K), 0.0010407 secs]

13.926: [GC [PSYoungGen: 47024K->128K(57152K)] 47024K->128K(187712K), 0.0010135 secs]

18.044: [GC [PSYoungGen: 47024K->128K(57152K)] 47024K->128K(187712K), 0.0010428 secs]

22.165: [GC [PSYoungGen: 47024K->136K(48000K)] 47024K->136K(178560K), 0.0080463 secs]

Heap

 PSYoungGen      total 48000K, used 11948K [5dab0000, 61290000, 73000000)

  eden space 47808K, 24% used [5dab0000,5e6390e0,60960000)

  from space 192K, 70% used [60a90000,60ab2050,60ac0000)

  to   space 192K, 0% used [61260000,61260000,61290000)

 PSOldGen        total 130560K, used 0K [33000000, 3af80000, 5dab0000)

  object space 130560K, 0% used [33000000,33000000,3af80000)

 PSPermGen       total 16384K, used 1820K [2f000000, 30000000, 33000000)

  object space 16384K, 11% used [2f000000,2f1c7168,30000000)

but PrintGCDateStamps is NOT (:o( ) Anyway XX options are not recommended for production system

See here http://stackoverflow.com/questions/541832/know-of-any-java-garbage-collection-log-analysis-tools for a list of tools to analyze GC logs

Here http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html a list of XX options

I have also tested this flag: -XX:+PrintConcurrentLocks
and it prints in the threadDump very valuable information on the locks being held by each thread:

Example:
java -Xloggc:mygc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintConcurrentLocks -XX:+PrintGCTimeStamps DoNothing

I do a kill -3 on the PID, and I get this on the stdout:

Full thread dump Java HotSpot(TM) Server VM (1.6.0.01 jinteg:06.07.08-04:50 IA64 mixed mode):

"Low Memory Detector" daemon prio=5 tid=00459790 nid=12 lwp_id=644290 runnable [00000000..00000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None


"CompilerThread1" daemon prio=9 tid=005e22e0 nid=10 lwp_id=644288 waiting on condition [00000000..2d7ff688]
   java.lang.Thread.State: RUNNABLE
 

   Locked ownable synchronizers:
        - None

"CompilerThread0" daemon prio=9 tid=005db1a0 nid=9 lwp_id=644287 waiting on condition [00000000..2dfff608]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" daemon prio=9 tid=00459590 nid=8 lwp_id=644286 waiting on condition [00000000..00000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"Finalizer" daemon prio=8 tid=00459390 nid=7 lwp_id=644283 in Object.wait() [733c0000..733c0b90]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <612802c0> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)

        - locked <612802c0> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)

        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:197)

 

   Locked ownable synchronizers:

        - None

 

"Reference Handler" daemon prio=10 tid=00459190 nid=6 lwp_id=644282 in Object.wait() [77540000..77540d10]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <61280350> (a java.lang.ref.Reference$Lock)

        at java.lang.Object.wait(Object.java:485)

        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123)

        - locked <61280350> (a java.lang.ref.Reference$Lock)

 

   Locked ownable synchronizers:

        - None

 

"main" prio=5 tid=00458f90 nid=1 lwp_id=-1 waiting on condition [7fffe000..7fffe798]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

        at java.lang.Thread.sleep(Native Method)

        at DoNothing.main(DoNothing.java:4)

 

   Locked ownable synchronizers:

        - None

 

"VM Thread" prio=9 tid=005a7860 nid=5 lwp_id=644281 runnable

 

"GC task thread#0 (ParallelGC)" prio=5 tid=004661f0 nid=3 lwp_id=644279 runnable

 

"GC task thread#1 (ParallelGC)" prio=5 tid=00466410 nid=4 lwp_id=644280 runnable

 

"VM Periodic Task Thread" prio=10 tid=005a7950 nid=11 lwp_id=644289 waiting on condition

 

JNI global references: 585

 

Heap

 PSYoungGen      total 57152K, used 39278K [5dab0000, 61a70000, 73000000)

  eden space 49024K, 79% used [5dab0000,600f3b38,60a90000)

  from space 8128K, 1% used [61280000,612a0040,61a70000)

  to   space 8128K, 0% used [60a90000,60a90000,61280000)

 PSOldGen        total 130560K, used 0K [33000000, 3af80000, 5dab0000)

  object space 130560K, 0% used [33000000,33000000,3af80000)

 PSPermGen       total 16384K, used 1797K [2f000000, 30000000, 33000000)

  object space 16384K, 10% used [2f000000,2f1c17d8,30000000)

This option is the default on JRockit… priceless!

Problem is that the XX options are not really recommended for production systems, so…

“Options that are specified with -XX are not stable and are not recommended for casual use. These options are subject to change without notice. “