Saturday, November 2, 2013

JMXTrans and Graphite: getting started

You can achieve very decent monitoring of a Java application - including the JVM itself - with jmxtrans and graphite .

No more horrible hand-crafted tool to extract and chart JMS data.

A TRANS in Italian is a transexual, typically Brazilian - more knows as VIADOS - so hearing JMXTrans always puts me in a good mood because Italians like to make jokes about sex.

OK, let's be serious.
First read the excellent jmxtrans Wiki here.
As suggested by jmstrans site, I listened to the presentation by Coda Hale but frankly it didn't say much new, he speaks well but not very informative

Getting started:
download the tar.gz here

tar xvzf jmxtrans-242.tar.gz
cd jmxtrans-242/rpm
.... mmm when I run build.sh it asks me a release and version?????


I will download a rpm from here (instructions on the site are incomplete... not good...)

rpm -i jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm

I verify the installation parameters:
less /etc/sysconfig/jmxtrans

# configuration file for package jmxtrans
export JAR_FILE="/usr/share/jmxtrans/jmxtrans-all.jar"
export LOG_DIR="/var/log/jmxtrans"
export SECONDS_BETWEEN_RUNS=60
export JSON_DIR="/var/lib/jmxtrans"
export HEAP_SIZE=512
export NEW_SIZE=64
export CPU_CORES=2
export NEW_RATIO=8
export LOG_LEVEL=debug


cd /usr/share/jmxtrans
./jmxtrans.sh
Usage: ./jmxtrans.sh {start|stop|restart|status} [filename.json]


I look into an example.json
{
  "servers" : [ {
    "port" : "1099",
    "host" : "w2",
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
        "settings" : {
        }
      } ],
      "obj" : "java.lang:type=Memory",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
    }, {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
        "settings" : {
        }
      } ],
      "obj" : "java.lang:name=CMS Old Gen,type=MemoryPool",
      "attr" : [ "Usage" ]
    }, {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
        "settings" : {
        }
      } ],
      "obj" : "java.lang:name=ConcurrentMarkSweep,type=GarbageCollector",
      "attr" : [ "LastGcInfo" ]
    } ],
    "numQueryThreads" : 2
  } ]
}

(I still need to decide if I hate more JSON or XML... they both hurt the eye...)

I write a sample Java class to monitor:
vi Sample.java
public class Sample {
        public static void main(String[] args) {
            try {
                Thread.sleep(100000000);
            }
            catch (Exception e) {};
        }
}


javac Sample.java
java -Dcom.sun.management.jmxremote.port=1105 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false Sample

cp example.json sample.json
I edit port number and host to match 1105 and my localhost

./jmxtrans.sh start sample.json

this runs in background :
ps -ef | grep jmxtrans
root 2240 1 6 07:47 pts/1 00:00:15 /usr/bin/java -server -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djmxtrans.log.level=debug -Djmxtrans.log.dir=. -Xms512M -Xmx512M -XX:+UseConcMarkSweepGC -XX:NewRatio=8 -XX:NewSize=64m -XX:MaxNewSize=64m -XX:MaxTenuringThreshold=16 -XX:GCTimeRatio=9 -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseTLAB -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=1 -Dsun.rmi.dgc.server.gcInterval=28800000 -Dsun.rmi.dgc.client.gcInterval=28800000 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=2101 -jar jmxtrans-all.jar -e -f sample.json -s 60


and if I do
tail -f jmxtrans.log
I get a line per minute:
[02 Nov 2013 07:47:22] [main] 0 INFO (com.googlecode.jmxtrans.JmxTransformer:134) - Starting Jmxtrans on : sample.json
[02 Nov 2013 07:47:23] [main] 1125 DEBUG (com.googlecode.jmxtrans.JmxTransformer:354) - Loaded file: /usr/share/jmxtrans/sample.json
[02 Nov 2013 07:47:23] [main] 1154 DEBUG (com.googlecode.jmxtrans.JmxTransformer:429) - Scheduled job: osb-vagrant.acme.com:1105-1383374843215-0158449004 for server: Server [host=osb-vagrant.acme.com, port=1105, url=null, cronExpression=null, numQueryThreads=2]
[02 Nov 2013 07:47:23] [ServerScheduler_Worker-1] 1167 DEBUG (com.googlecode.jmxtrans.jobs.ServerJob:31) - +++++ Started server job: Server [host=osb-vagrant.acme.com, port=1105, url=null, cronExpression=null, numQueryThreads=2]
[02 Nov 2013 07:47:23] [ServerScheduler_Worker-1] 1690 DEBUG (com.googlecode.jmxtrans.util.JmxUtils:102) - ----- Creating 3 query threads
[02 Nov 2013 07:47:24] [pool-1-thread-1] 1940 DEBUG (com.googlecode.jmxtrans.util.JmxUtils:195) - Executing queryName: java.lang:type=Memory from query: Query [obj=java.lang:type=Memory, resultAlias=null, attr=[HeapMemoryUsage, NonHeapMemoryUsage]]
[02 Nov 2013 07:47:24] [pool-1-thread-1] 2002 DEBUG (com.googlecode.jmxtrans.util.JmxUtils:209) - Finished running outputWriters for query: Query [obj=java.lang:type=Memory, resultAlias=null, attr=[HeapMemoryUsage, NonHeapMemoryUsage]]
[02 Nov 2013 07:47:24] [ServerScheduler_Worker-1] 2006 DEBUG (com.googlecode.jmxtrans.jobs.ServerJob:50) - +++++ Finished server job: Server [host=osb-vagrant.acme.com, port=1105, url=service:jmx:rmi:///jndi/rmi://osb-vagrant.acme.com:1105/jmxrmi, cronExpression=null, numQueryThreads=2]

Next step: install grafite and plot these metrics....

2 comments:

Asaf Mesika said...

If you want to lose the hand-crafted json configuration file, take a loot at jmx2graphite at: https://github.com/logzio/jmx2graphite

Guillaume Lederrey said...

Finding the latest version of JmxTrans can be ... not as easy as it should be. I've been getting report from user who were testing JmxTrans with the fairly old version referenced here.

To make sure your have the latest release, have a look at Maven Central.