Hyperic, my first WEBLOGIC working plugin (Still not working...)

java -Dplugins.include=mysecondplugin -Djmx.url=service:jmx:iiop://localhost:8001/jndi/weblogic.management.mbeanservers.runtime -Djava.naming.factory.initial=weblogic.jndi.WLInitialContextFactory -jar C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5.jar -m discover -a metric -Dlog=debug

WebLogic Cluster Availability:
com.bea:type=ClusterRuntime,name=%name%:AliveServerCount:jmx.url=service%3Ajmx%3Aiiop%3A//localhost%3A8001/jndi/weblogic.management.mbeanservers.runtime,jmx.username=weblogic,jmx.password=
getValue failed for metric: WebLogic Cluster:com.bea:type=ClusterRuntime,name=%name%:AliveServerCount:jmx.url=service%3Ajmx%3Aiiop%3A//localhost%3A8001/jndi/weblogic.management.mbeanservers.runtime,jmx.username=weblogic,jmx.password=%jmx.password%
MetricUnreachableException: Can't connect to MBeanServer [{jmx.username=weblogic, jmx.url=service:jmx:iiop://localhost:8001/jndi/weblogic.management.mbeanservers.runtime}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://localhost:7001: Destination unreachable; nested exception is:
java.net.ConnectException: Connection refused: connect; No available router to destination]
org.hyperic.hq.product.MetricUnreachableException: Can't connect to MBeanServer [{jmx.username=weblogic, jmx.url=service:jmx:iiop://localhost:8001/jndi/weblogic.management.mbeanservers.runtime}]: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://localhost:7001: Destination unreachable; nested exception is:
java.net.ConnectException: Connection refused: connect; No available router to destination]
at org.hyperic.hq.product.jmx.MxUtil.unreachable(MxUtil.java:156)
at org.hyperic.hq.product.jmx.MxUtil.getValue(MxUtil.java:320)
at org.hyperic.hq.product.jmx.MxMeasurementPlugin.getValue(MxMeasurementPlugin.java:58)
at org.hyperic.hq.product.MeasurementPluginManager.getPluginValue(MeasurementPluginManager.java:171)
at org.hyperic.hq.product.MeasurementPluginManager.getValue(MeasurementPluginManager.java:269)
at org.hyperic.hq.product.MeasurementPluginManager.getValue(MeasurementPluginManager.java:130)
at org.hyperic.hq.product.MeasurementPluginManager.getValue(MeasurementPluginManager.java:122)
at org.hyperic.hq.product.util.PluginDumper.getValue(PluginDumper.java:644)
at org.hyperic.hq.product.util.PluginDumper.fetchMetrics(PluginDumper.java:790)
at org.hyperic.hq.product.util.PluginDiscoverer.dumpProperties(PluginDiscoverer.java:375)
at org.hyperic.hq.product.util.PluginDiscoverer.dumpProperties(PluginDiscoverer.java:424)
at org.hyperic.hq.product.util.PluginDiscoverer.dumpServer(PluginDiscoverer.java:488)
at org.hyperic.hq.product.util.PluginDiscoverer.runtimeScan(PluginDiscoverer.java:563)
at org.hyperic.hq.product.util.PluginDiscoverer.start(PluginDiscoverer.java:321)
at org.hyperic.hq.product.util.PluginDumper.testDiscovery(PluginDumper.java:1148)
at org.hyperic.hq.product.util.PluginDumper.invoke(PluginDumper.java:456)
at org.hyperic.hq.product.util.PluginMain.main(PluginMain.java:321)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://localhost:7001: Destination unreachable; nested exception is:
java.net.ConnectException: Connection refused: connect; No available router to destination]
at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
at org.hyperic.hq.product.jmx.MxUtil.getMBeanConnector(MxUtil.java:538)
at org.hyperic.hq.product.jmx.MxUtil.getCachedMBeanConnector(MxUtil.java:434)
at org.hyperic.hq.product.jmx.MxUtil.getValue(MxUtil.java:651)
at org.hyperic.hq.product.jmx.MxUtil.getValue(MxUtil.java:310)
... 15 more


class is org.hyperic.hq.product.servlet.client.JMXRemote



C:\Oracle\Middleware\user_projects\domains\prod_domain>java weblogic.Admin -url t3://localhost:8001 -username weblogic -password weblogic1 -pretty GET -type ClusterRuntime
---------------------------
MBeanName: "prod_domain:ServerRuntime=ms1,Name=mycluster,Type=ClusterRuntime,Location=ms1"
ActiveSingletonServices:
AliveServerCount: 1
CachingDisabled: true
CurrentMachine: mymachine
CurrentSecondaryServer:
DetailedSecondariesDistribution:
ForeignFragmentsDroppedCount: 0
FragmentsReceivedCount: 0
FragmentsSentCount: 109
HealthState: Component:null,State:HEALTH_OK,MBean:null,ReasonCode:[]
JobSchedulerRuntime:
MulticastMessagesLostCount: 0
Name: mycluster
Parent: ms1
PrimaryCount: 0
Registered: true
ResendRequestsCount: 0
SecondaryCount: 0
SecondaryDistributionNames:
SecondaryServerDetails:
ServerMigrationRuntime: ServerMigrationRuntime
ServerNames: ms1
Type: ClusterRuntime
UnicastMessaging: UnicastMessagingRuntime

Hyperic versus Nagios

8.5M vs 860K.... by a factor of 10, Nagios is definitely more popular than Hyperic.

Here they say that "Hyperic customers have successfully transitioned from Nagios to Hyperic's deeper management capabilities and easier deployment. Many have identified Nagios' complex and time consuming setup and limited features as a main reason to migrate their management solution to Hyperic."

I don't have experience (sob) with Nagios, all I can say is that Hyperic is VERY easy to install and use, but when it comes to customization and documentation, it's really really poorly supported.

Troubleshooting a "server down" or a "filesystem full"

Server myserver down:
ping myserver
ssh myuser@myserver

check if Hyperic monitoring agent is running:
ps -eaf|grep agent|grep java|grep -v grep

For a file system full or about to be full:

df –k
cd filesystemName
du 2> /dev/null | sort -nr | cut -f2- | xargs du -hs 2> /dev/null | more

(see http://serverfault.com/questions/62411/how-can-i-sort-du-h-output-by-size)

Hyperic and SNMP traps

http://support.hyperic.com/display/DOC/Configuring+HQ+as+an+SNMP+Trap+Receiver

I add
snmpTrapReceiver.listenAddress=udp:0.0.0.0/1620
to agent.properties


restart agent

netstat -an

UDP 0.0.0.0:1620 *:*


Click on Resources/New Platform/

name=TrapProxy
description=receiving SNMP traps
Platform type=Network device
agent connection=127.0.0.1:2144
fqdn=myhostSNMP.acme.com
ipaddress=10.56.64.190
mac=???
netmask=???


OK

now scroll down and click Edit in the "Configuration Properties"

interface.index=ifDescr
snmpIp=10.56.64.190
snmpCommunity=public
snmpPort=1620 (instead of the default 161)

OK

I get "Error contacting resource: Unable to connect to SNMP Agent (10.56.64.190:1620 v2c,public)"

http://communities.vmware.com/message/1939292


... to be continued....

Hyperic, browse PostgreSQL DB

look in C:\pierre\hypericserver\server-4.6.5-EE\conf\hq-server.conf

server.database=PostgreSQL
server.database-driver=org.postgresql.Driver
server.database-url=jdbc:postgresql://127.0.0.1:9432/hqdb?protocolVersion=2
server.database-user=hqadmin
server.database-password=ENC(Nadmq0eNldP2+Cew2mu+fw==)

To browse the DB:
start HQ Server


C:\pierre\hypericserver\server-4.6.5-EE\bin\db-psql.bat


Welcome to psql 8.2.5, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit

Warning: Console code page (437) differs from Windows code page (1252)
8-bit characters may not work correctly. See psql reference
page "Notes for Windows users" for details.


Useful commands

\h help with SQL commands
? help with psql commands
\du list roles/users
\l list databases
\c to choose a database
\d to list tables once in a database
\q quit


I do:

\c hqdb
\d
and I get plenty of eam_", hq_metric* and qrtz* tables. It looks quite complex.

select * from eam_platform;
select * from eam_resource;


here http://linux.die.net/man/1/psql the manual of psql


If you need a GUI, here http://wiki.postgresql.org/wiki/Community_Guide_to_PostgreSQL_GUI_Tools a lost of GUI who support PostgreSQL


I have installed Squirrel, and copied postgresql-7.4.3.jar from Hyperic to C:\pierre\squirrel\lib (requires restart). It connects without problem (use jdbc:postgresql://127.0.0.1:9432/hqdb?protocolVersion=2 hqadmin hqadmin), but then I am lost in this UI.

Hyperic: my first WORKING plugin

(after 1 day of pain and sorrow)

Run your test application:

java -Dproc.java.home=C:\Oracle\Middleware\jdk160_24 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=6969 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false com.acme.hqautomation.jmx.SimpleAgent

and then run the monitoring plugin:

java -Dplugins.include=myfirstplugin -jar C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5.jar -m discover -a metric -Dlog=debug

BEWARE! the xml filename must be myfirstplugin-plugin.xml (I think the idea of adding the -plugin is REALLY stupid anyway, and it's not explained in the doc)


Here is SimpleAgent

package com.acme.jmx;
import javax.management.*;
import java.lang.management.*;


public class SimpleAgent {
   private MBeanServer mbs = null;

   public SimpleAgent() {

      // Get the platform MBeanServer
       mbs = ManagementFactory.getPlatformMBeanServer();

      // Unique identification of MBeans
      Hello helloBean = new Hello();
      ObjectName helloName = null;

      try {
         // Uniquely identify the MBeans and register them with the platform MBeanServer 
         helloName = new ObjectName("SimpleAgent:name=hellothere");
         mbs.registerMBean(helloBean, helloName);
      } catch(Exception e) {
         e.printStackTrace();
      }
   }

   // Utility method: so that the application continues to run
   private static void waitForEnterPressed() {
      try {
         System.out.println("Press  to continue...");
         System.in.read();
      } catch (Exception e) {
         e.printStackTrace();
      }
    }

   public static void main(String argv[]) {
      SimpleAgent agent = new SimpleAgent();
      System.out.println("SimpleAgent is running...");
      SimpleAgent.waitForEnterPressed();
   }
}

Here is myfirstplugin-plugin.xml (the entity process-metrics.xml has been removed since I am not monitoring the process, only the jmx metrics)

<?xml version="1.0"?>
 
<!DOCTYPE plugin>
 
<plugin package="org.hyperic.hq.plugin.java">
 
<classpath>
<include name="pdk/lib/mx4j"/>
</classpath>
 
<filter name="template" value="${OBJECT_NAME}:${alias}"/>
 
<metrics name="Class Loading Metrics">
<metric name="Loaded Class Count" indicator="false" category="THROUGHPUT"/>
<metric name="Total Loaded Class Count" indicator="false" category="THROUGHPUT"/>
<metric name="Unloaded Class Count" indicator="false" category="THROUGHPUT"/>
</metrics>
 
<metrics name="Compilation">
<metric name="Total Compilation Time" indicator="false" category="THROUGHPUT" collectionType="trendsup" units="ms"/>
</metrics>
 
<metrics name="Garbage Collector">
<metric name="Collection Count" indicator="false" category="THROUGHPUT" collectionType="trendsup"/>
<metric name="Collection Time" indicator="false" category="THROUGHPUT" collectionType="trendsup"/>
</metrics>
 
<metrics name="Memory">
<metric name="Object Pending Finalization Count" category="THROUGHPUT" indicator="false"/>
</metrics>
 
<metrics name="Threading">
<metric name="Thread Count" category="UTILIZATION" indicator="false"/>
<metric name="Daemon Thread Count" category="UTILIZATION" indicator="false"/>
</metrics>
 
<server name="Java" version="1.5.x">
<property name="HAS_BUILTIN_SERVICES" value="true"/>
<property name="VERSION_FILE" value="jre/lib/fontconfig.Sun.2003.bfc"/>
<property name="DEFAULT_PROGRAM" value="bin/java"/>
<property name="domain" value="Java"/>
 
<config>
<option name="jmx.url" description="JMX URL to MBeanServer"  default="service:jmx:rmi:///jndi/rmi://localhost:6969/jmxrmi"/>
<option name="jmx.username" description="JMX username" optional="true" default=""/>
<option name="jmx.password" description="JMX password" optional="true" default="" type="secret"/>
<!--option name="process.query" description="PTQL for Java Process" default="State.Name.eq=java,Args.*.ct=proc.java.home"/-->
</config>
 
<!--metric name="Availability" template="sigar:Type=ProcState,Arg=%process.query%:State" indicator="true"/-->

 
<property name="OBJECT_NAME" value="java.lang:type=ClassLoading"/>
 
<metrics include="Class Loading Metrics"/>
<property name="OBJECT_NAME" value="java.lang:type=Compilation"/>
 
<metrics include="Compilation"/>
<property name="OBJECT_NAME" value="java.lang:type=Memory"/>
 
<plugin type="log_track" class="org.hyperic.hq.product.jmx.MxNotificationPlugin"/>
 
<property name="OBJECT_NAME" value="java.lang:type=Threading"/>
<metrics include="Threading"/>
 
<!-- derive installpath from JAVA_HOME env prop... -->
<property name="PROC_HOME_ENV" value="JAVA_HOME"/>
 
<!-- derive installpath from -Dproc.java.home=... -->
<property name="PROC_HOME_PROPERTY" value="proc.java.home"/>
<plugin type="autoinventory" class="org.hyperic.hq.product.jmx.MxServerDetector"/>
<plugin type="measurement" class="org.hyperic.hq.product.jmx.MxMeasurementPlugin"/>
 
<service name="Java GC">
<plugin type="autoinventory"/>
<property name="OBJECT_NAME" value="java.lang:type=GarbageCollector,name=*"/>
<metrics include="Garbage Collector"/>
</service>
</server>
 
<server name="Java" version="1.6.x" include="1.5.x">
<property name="VERSION_FILE" value="jre/lib/management-agent.jar"/>
</server>
 
<!--
 ==================== Plugin Help ===========================
-->
<help name="Java">
<![CDATA[
  <p>
  <h3>Configure HQ for monitoring Java</h3>
  </p>
  <p>
  1) Add this line to the java options when executing the binary.
  <br>
  "-Dcom.sun.management.jmxremote \
  <br>
  -Dcom.sun.management.jmxremote.port=6969 \
  <br>
  -Dcom.sun.management.jmxremote.ssl=false \
  <br>
  -Dcom.sun.management.jmxremote.authenticate=false"
  <br>
  </p>
]]>
</help>
<help name="Java 1.5.x" include="Java"/>
<help name="Java 1.6.x" include="Java"/>
</plugin>

In the autodiscovery, you should now find a "Java 1.6 C:\Oracle\Middleware\jdk160_24" resource name, add it, then in the Inventory you should find 2 metrics:

Java 1.6.x Copy Java GC
Java 1.6.x MarkSweepCompact Java GC



The result of running the plugin tester is:




C:\Oracle\Middleware\jdk160_24\bin>java -Dplugins.include=myfirstplugin -jar C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5.jar -m discover -a metric -Dlog=debug
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/pierre/hypericagent/agent-4.6.5-EE/bundles/agent-4.6.5/pdk/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
INFO [main] [ProductPluginManager] agent.pdkDir=C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk
INFO [main] [ProductPluginManager] agent.pdkPluginDir=C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk/plugins
INFO [main] [ProductPluginManager] agent.pdkWorkDir=C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk/work
DEBUG [main] [ProductPluginManager] Initializing in client mode (pdk=C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk)
INFO [main] [ConfigTrackPluginManager] Running on Server, FileMonitor not initiated
DEBUG [main] [MIBTree] C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\mibs MIB dir does not exist
INFO [main] [ProductPluginManager] Loading plugin: myfirstplugin-plugin.xml (C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins)
DEBUG [main] [ProductPluginManager] Manifest retrieval had an exception (continuing): error in opening zip file
DEBUG [main] [PluginLoader] myfirstplugin-plugin.xml expanding C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\mx4j...
DEBUG [main] [PluginLoader] myfirstplugin-plugin.xml += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\mx4j\mx4j-3.0.1.jar
DEBUG [main] [PluginLoader] myfirstplugin-plugin.xml += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\mx4j\mx4j-jmx-3.0.1.jar
DEBUG [main] [PluginLoader] myfirstplugin-plugin.xml += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\mx4j\mx4j-remote-3.0.1.jar
DEBUG [main] [PluginLoader] myfirstplugin-plugin.xml += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\mx4j\mx4j-rjmx-2.1.1.jar
DEBUG [main] [PluginLoader] myfirstplugin-plugin.xml += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\mx4j\mx4j-tools-3.0.1.jar
INFO [main] [ProductPluginManager] Loading plugin: netservices-plugin.jar (C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins)
DEBUG [main] [ProductPluginManager]
--- Manifest entries for: jar:file:C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins\netservices-plugin.jar!/ ---
Implementation-Title - Hyperic HQ Plugin - Net Services Plugin
Implementation-Version - 4.6.5
Implementation-Vendor-Id - org.hyperic.hq.plugin
Build-Jdk - 1.6.0_29
Built-By - bob
Manifest-Version - 1.0
Specification-Title - HQ Plugin
Created-By - Apache Maven
Specification-Version - 4.6.5
Build-Date - 20120301-1052
Archiver-Version - Plexus Archiver

DEBUG [main] [ClientPluginDeployer] Unchanged file: C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\work\lib\netservices\dnsjava-2.0.6.jar
DEBUG [main] [PluginLoader] netservices-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\work\lib\netservices\dnsjava-2.0.6.jar
INFO [main] [MeasurementPlugin] Register TCP Socket proxy for domain: socket
INFO [main] [MeasurementPlugin] Register HTTP proxy for domain: http
INFO [main] [MeasurementPlugin] Register HTTP proxy for domain: url.availability
INFO [main] [MeasurementPlugin] Register FTP proxy for domain: ftp
INFO [main] [ProductPluginManager] Loading plugin: sqlquery-plugin.jar (C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins)
DEBUG [main] [ProductPluginManager]
--- Manifest entries for: jar:file:C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins\sqlquery-plugin.jar!/ ---
Implementation-Title - Hyperic HQ Plugin - SQL Query Plugin
Implementation-Version - 4.6.5
Implementation-Vendor-Id - org.hyperic.hq.plugin
Build-Jdk - 1.6.0_29
Built-By - bob
Manifest-Version - 1.0
Specification-Title - HQ Plugin
Created-By - Apache Maven
Specification-Version - 4.6.5
Build-Date - 20120301-1052
Archiver-Version - Plexus Archiver

DEBUG [main] [PluginLoader] sqlquery-plugin.jar expanding C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc...
DEBUG [main] [PluginLoader] sqlquery-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc\msbase-1.2.2.jar
DEBUG [main] [PluginLoader] sqlquery-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc\mssqlserver-1.2.2.jar
DEBUG [main] [PluginLoader] sqlquery-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc\msutil-1.2.2.jar
DEBUG [main] [PluginLoader] sqlquery-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc\mysql-connector-java-commercial-5.1.10.jar
DEBUG [main] [PluginLoader] sqlquery-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc\ojdbc5-11.2.0.2.0.jar
DEBUG [main] [PluginLoader] sqlquery-plugin.jar += C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\jdbc\postgresql-7.4.3.jar
INFO [main] [SQLQueryMeasurementPlugin] Registered proxy for domain: sql
INFO [main] [ProductPluginManager] Loading plugin: system-plugin.jar (C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins)
DEBUG [main] [ProductPluginManager]
--- Manifest entries for: jar:file:C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins\system-plugin.jar!/ ---
Implementation-Title - Hyperic HQ Plugin - System Plugin
Implementation-Version - 4.6.5
Implementation-Vendor-Id - org.hyperic.hq.plugin
Build-Jdk - 1.6.0_29
Built-By - bob
Manifest-Version - 1.0
Specification-Title - HQ Plugin
Created-By - Apache Maven
Specification-Version - 4.6.5
Build-Date - 20120301-1052
Archiver-Version - Plexus Archiver

DEBUG [main] [SystemPlugin] sigar.mirror.procnet=null
DEBUG [main] [PluginDiscoverer] Adding Auto Scanners=[ServerSignature {stName=Java 1.5.x, match=[null], exclude=[null], regMatch=[null]}, ServerSignature {stName=Java 1.6.x, match=[null], exclude=[null], regMatch=[null]}]
DEBUG [main] [PluginDiscoverer] Adding Registry Scanners=[]
DEBUG [Thread-1] [NullScan] Running AutoServerDetector for: Java 1.5.x
DEBUG [Thread-1] [MxServerDetector] ptql=State.Name.sw=java,Args.*.sw=-Dproc.java.home= matched pids=[[J@12b6c89]
DEBUG [Thread-1] [MxServerDetector] C:\Oracle\Middleware\jdk160_24\jre\lib\fontconfig.Sun.2003.bfc does not exist, skipping
DEBUG [Thread-1] [NullScan] Running AutoServerDetector for: Java 1.6.x
DEBUG [Thread-1] [MxServerDetector] ptql=State.Name.sw=java,Args.*.sw=-Dproc.java.home= matched pids=[[J@e53220]
DEBUG [Thread-1] [MxServerDetector] classpath unchanged using: C:\Oracle\Middleware\jdk160_24
DEBUG [Thread-1] [MxUtil] ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24 resolved to vmid=7960, jmx.url=service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjYwLjE2MC40NgAA5W6mztbP3rJ3QmqDpA4AAAE3QdfsrYACAHg=
DEBUG [Thread-1] [MxServerDetector] Using the local pid to create jmx url: ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24
DEBUG [Thread-1] [WindowsRegistryScan] Windows Registry Scan starting...
DEBUG [Thread-1] [WindowsRegistryScan] Windows Registry Scan completed, took: 0
1 servers detected

Server: CH30-0FRNC59 Java 1.6.x [C:\Oracle\Middleware\jdk160_24]
AIID.....C:\Oracle\Middleware\jdk160_24
config...
product..jmx.url=ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24,
metric...
control..null
DEBUG [main] [MxUtil] ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24 resolved to vmid=7960, jmx.url=service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjYwLjE2MC40NgAA5W6mztbP3rJ3QmqDpA4AAAE3QdfsrYACAHg=
DEBUG [main] [MxUtil] created new JMXConnector url=service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjYwLjE2MC40NgAA5W6mztbP3rJ3QmqDpA4AAAE3QdfsrYACAHg=, classloader=sun.misc.Launcher$AppClassLoader@17182c1
Java 1.6.x Loaded Class Count:
java.lang:type=ClassLoading:LoadedClassCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(1086.0) -> 1,086.0
=>1,086.0<=
Java 1.6.x Total Loaded Class Count:
java.lang:type=ClassLoading:TotalLoadedClassCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(1086.0) -> 1,086.0
=>1,086.0<=
Java 1.6.x Unloaded Class Count:
java.lang:type=ClassLoading:UnloadedClassCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(0.0) -> 0.0
=>0.0<=
Java 1.6.x Total Compilation Time:
java.lang:type=Compilation:TotalCompilationTime:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(102.0) -> 102ms
=>102ms<=
Java 1.6.x Thread Count:
java.lang:type=Threading:ThreadCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(17.0) -> 17.0
=>17.0<=
Java 1.6.x Daemon Thread Count:
java.lang:type=Threading:DaemonThreadCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(16.0) -> 16.0
=>16.0<=
DEBUG [main] [PluginDiscoverer] Runtime discover for CH30-0FRNC59 Java 1.6.x, config=jmx.username=,jmx.password=******,jmx.url=ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24,log=debug,installpath=C:\Oracle\Middleware\jdk160_24,
DEBUG [main] [ProductPluginManager] No ProductPlugin found for: Java 1.6.x
DEBUG [main] [MxServerDetector] [discoverServices] serverConfig=platform.log_track.level=Warn,jmx.username=,platform.log_track.enable=true,jmx.password=******,platform.config_track.enable=true,log=debug,installpath=C:\Oracle\Middleware\jdk160_24,platform.fqdn=CH30-0FRNC59.euc.nestle.com,platform.type=Win32,platform.ip=127.0.0.1,platform.name=localhost,platform.log_track.eventlogs=System,jmx.url=ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24,
DEBUG [main] [MxUtil] ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24 resolved to vmid=7960, jmx.url=service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjYwLjE2MC40NgAA5W6mztbP3rJ3QmqDpA4AAAE3QdfsrYACAHg=
DEBUG [main] [MxUtil] created new JMXConnector url=service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc3AAtVbmljYXN0UmVmMgAADDEwLjYwLjE2MC40NgAA5W6mztbP3rJ3QmqDpA4AAAE3QdfsrYACAHg=, classloader=sun.misc.Launcher$AppClassLoader@17182c1
DEBUG [main] [MxServerDetector] [discoverMxServices] url=ptql:State.Name.sw=java,Args.*.eq=-Dproc.java.home=C:\Oracle\Middleware\jdk160_24
DEBUG [main] [MxServerDetector] [discoverMxServices] objName=java.lang:type=Threading
DEBUG [main] [MxServerDetector] [discoverMxServices] objName=java.lang:type=Threading
DEBUG [main] [MxServerQuery] [findServices] servicePlugins={Java 1.6.x Java GC=null}
DEBUG [main] [MxServerQuery] [findServices] name=java.lang:type=GarbageCollector,*
DEBUG [main] [MxServerQuery] [findServices] services=(2)[java.lang:type=GarbageCollector,name=Copy, java.lang:type=GarbageCollector,name=MarkSweepCompact]
DEBUG [main] [MxServerDetector] discovered 2 services
DEBUG [main] [ProductPluginManager] No ControlPlugin found for: Java 1.6.x Java GC
DEBUG [main] [ProductPluginManager] No ControlPlugin found for: Java 1.6.x Java GC
DEBUG [main] [MxServerDetector] discovered 2 services

Runtime Resource Report...
Platform=Win32, fqdn=CH30-0FRNC59.euc.nestle.com (Microsoft Windows 2008)
config...
product..null
metric...platform.log_track.level=Warn,platform.log_track.eventlogs=System,platform.log_track.enable=true,platform.config_track.enable=true,
control..null
cprops...cpuSpeed=4 @ 2394 MHz,secondaryDNS=160.213.132.241,defaultGateway=0.0.0.0,arch=x86,ram=2992 MB,vendor=Microsoft,vendorVersion=2008,primaryDNS=160.213.132.237,ip=10.60.160.46,version=6.0,
CH30-0FRNC59 Java 1.6.x cprops:
CH30-0FRNC59 Java 1.6.x services:
CH30-0FRNC59 Java 1.6.x Copy Java GC
config...
product..name=Copy,
metric...
control..null
rt.......null
Java 1.6.x Java GC Collection Count:
java.lang:type=GarbageCollector,name=Copy:CollectionCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(8.0) -> 8.0
=>8.0<=
Java 1.6.x Java GC Collection Time:
java.lang:type=GarbageCollector,name=Copy:CollectionTime:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(12.0) -> 12.0
=>12.0<=
CH30-0FRNC59 Java 1.6.x MarkSweepCompact Java GC
config...
product..name=MarkSweepCompact,
metric...
control..null
rt.......null
Java 1.6.x Java GC Collection Count:
java.lang:type=GarbageCollector,name=MarkSweepCompact:CollectionCount:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(0.0) -> 0.0
=>0.0<=
Java 1.6.x Java GC Collection Time:
java.lang:type=GarbageCollector,name=MarkSweepCompact:CollectionTime:jmx.url=ptql%3AState.Name.sw%3Djava%2CArgs.*.eq%3D-Dproc.java.home%3DC%3A\Oracle\Middleware\jdk160_24,jmx.username=%jmx.username%,jmx.password=
DEBUG [main] [UnitsFormat] format(0.0) -> 0.0
=>0.0<=

C:\Oracle\Middleware\jdk160_24\bin>

Hyperic plugin software factory

After many errors and botch-ups, I am persuaded that the best way to develop a plugin is:

a C:\pierre\workspace\HypericAutomation\plugins where to keep in SVN the plugin xml

a C:\pierre\workspace\HypericAutomation\scripts with:

copyAllPlugins.bat:

copy ..\plugins\*-plugin.xml C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins\
copy ..\plugins\*-plugin.xml C:\pierre\hypericserver\server-4.6.5-EE\hq-engine\hq-server\webapps\ROOT\WEB-INF\hq-plugins

stopall.bat

cd C:\pierre\hypericagent\agent-4.6.5-EE\bin
call C:\pierre\hypericagent\agent-4.6.5-EE\bin\hq-agent.bat stop
cd C:\pierre\hypericserver\server-4.6.5-EE\bin
call C:\pierre\hypericserver\server-4.6.5-EE\bin\hq-server.bat stop

startall.bat

cd C:\pierre\hypericagent\agent-4.6.5-EE\bin
call C:\pierre\hypericagent\agent-4.6.5-EE\bin\hq-agent.bat start
cd C:\pierre\hypericserver\server-4.6.5-EE\bin
call C:\pierre\hypericserver\server-4.6.5-EE\bin\hq-server.bat start

and the sequence is:

edit the plugin
stopapp
copyAllPlugins
startall


this because I have the impression that the agent plugin get overwritten when you restart the server


Also, make sure you shutdown and restart your monitored applications whenever your IP address is changed (if you are on DHCP) !

Hyperic Plugin HowTo

http://support.hyperic.com/display/hyperforge/Deploying+Plugins+-+Basic+How+To


you must deploy the plugin on both the server and agent side

server and agent:

Windows
C:\pierre\hypericserver\server-4.6.5-EE\hq-engine\hq-server\webapps\ROOT\WEB-INF\hq-plugins
c:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\plugins

Linux
/opt/hyperic/server/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/hq-plugins
/opt/hyperic/agent/agent-4.6.5-EE/bundles/agent-4.6.5/pdk/plugins



http://support.hyperic.com/display/DOCS46/JMX+Plugin+Tutorial

see immediately this for help http://communities.vmware.com/message/1939306
"I tried running the commands with the argument -Dproc.java.home= and it is able to match the processes running."

(of course Hyperic documentation make a lot of blablabla but they forget to tell you clearly how to set each property, and leave to you the reverse engineering of their product...)

this is a SERVER plugin ("Java"), exposing SERVICES ("Java GC")

Here how to get the HQ source code:
svn co http://svn.hyperic.org/projects/hq/trunk hyperic-hq


How to have SIGAR discover a WebLogic server:

ps State.Name.eq=java,CredName.User.eq=soa,Args.*.ct=-Dweblogic.Name=osbpp1ms1

where osbpp1ms1 is the server name

Hyperic on Oracle DB

http://support.hyperic.com/display/DOC/Set+Up+Oracle

(full DDL to create tablespaces at the bottom of the above document....)

create temporary tablespace TEMP_HQDB
tempfile 'TEMP_HQDB.DBF' size 2G extent management local;

this will create a file
C:\oraclexe\app\oracle\product\10.2.0\server\database\TEMP_HQDB.DBF


create tablespace TS_HQDB logging datafile 'TS_HQDB.dbf' size 3g extent management local;

this will create a file
C:\oraclexe\app\oracle\product\10.2.0\server\database\TS_HQDB.DBF

This could take several minutes.


CREATE USER HQUSER IDENTIFIED BY HQPASSWORD DEFAULT TABLESPACE TS_HQDB;
GRANT CONNECT, RESOURCE, CREATE VIEW TO HQUSER;


copy ojdbc6.jar to the installer (what? you need to reinstall Hyperic Server to switch to Oracle DB? Oh no :o(((( ....)

Hyperic JMX Plugin tutorial

this presentation is a bit fuzzy



Notable classes:

MxServerDetector

MxNotificationPlugin

MBeanServer

This one is a lot better (although video quality sucks, too many skipped frames) on how to query JMX Mbeans in the Hyperic console. I wish the sample code was attached.

Hyperic, GridLink and JDBCConnectionPoolRuntime

DataSource=SOADataSource

Host Server="Weblogic 10.3 osbpp1do osbpp1ms1" (osbpp1do=wldomain, osbpp1ms1=wlserver)

Resource Type="Weblogic 10.3 JDBC Connection Pool"

jdbc.conn=SOADataSource

JDBC URL=jdbc:oracle:thin:@mydbserver.acme.com:1551:osbdb1
JDBC Driver Name : oracle.jdbc.OracleDriver

MBean as constructed by Hyperic:

'osbpp1do:Location=osbpp1ms1,ServerRuntime=osbpp1ms1,Name=SOADataSource2,Type=JDBCConnectionPoolRuntime,ApplicationRuntime=SOADataSource1'

Name=SOADataSource2 is the content of jdbc.conn (JDBC Connection Pool name)
ApplicationRuntime=SOADataSource1 is the content of application (Application name)

application must be at least 1 characters long


java weblogic.Admin -url t3://myserver:8001 -username weblogic -password weblogic1 GET -pretty -type JDBCConnectionPoolRuntime

osbpp1do:ServerRuntime=osbpp1ms1,Name=ACMEDataSource,Type=JDBCConnectionPoolRuntime,Location=osbpp1ms1

SOADataSource has a JDBCOracleDataSourceRuntime
(wls:/osbpp1do/serverRuntime/JDBCServiceRuntime/osbpp1ms1/JDBCDataSourceRuntimeMBeans/SOADataSource)

GridLink stuff is a
JDBCOracleDataSourceRuntime extends JDBCDataSourceRuntime which extends a ComponentRuntime

while a Generic DataSource is a JDBCDataSourceRuntime extends ComponentRuntime

so... I am afraid you can't monitor GridLink Datasources with Hyperic OOTB, using a traditional prototype "Weblogic 10.3 JDBC Connection Pool": you need to write a custom JMX plugin for that.

Hyperic SIGAR PTQL (Process Table Query Language)

java -jar /opt/hyperic/agent/agent-4.6.5-EE/bundles/agent-4.6.5/pdk/lib/sigar-1.6.4.jar


to find a WebLogic process:

ps State.Name.eq=java,CredName.User.eq=soa,Args.*.ct=-Dweblogic.Name=osbdv2ms1



to find all processes:
ps


to find all Java processes:
ps State.Name.eq=java


sigar> ps State.Name.eq=java
6300 pvernet 09:08 1.2G 81M - R 0:15 java:weblogic.Server
9512 ??? ??? 340M 107M - R ??? java
7728 ??? ??? 311M 42M - R ??? java
4784 ??? ??? 908M 325M - R ??? java
4512 pvernet 10:30 262M 18M - R 0:1 java:org.hyperic.sigar.cmd.Runner




you can do ps 6300

sigar> ps 6300
6300 pvernet 09:08 1.2G 82M - R 0:15 java:weblogic.Server









Official doc of Process Table Query Language

SIGAR doc (SIGAR=System Information Gatherer)

You even have a SIGAR API

Hyperic, Anatomy of a JMX plugin

http://support.hyperic.com/display/DOC/JMX+Plugin

http://support.hyperic.com/display/DOC/JMX+Plugin+Tutorial

More modern (for 4.6.5):
http://support.hyperic.com/display/EVO/JMX+Plugin


java -Duser=weblogic -Dpass=weblogic1 -Dplugins.include=jmx -jar C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5.jar jmx MBeanDumper service:jmx:rmi:///jndi/rmi://myhost:8888/jmxrmi

(8888 is the remote jmx port)


java -Duser=weblogic -Dpass=weblogic1 -Dplugins.include=jmx -jar C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5.jar jmx MBeanDumper service:jmx:iiop://myhost:8001/jndi/weblogic.management.mbeanservers.runtime

(8001 is the listen port of the WLS)

you will get:

java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial


so you will add -Djava.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
and a we have this:

java -cp C:\Oracle\Middleware\wlserver_10.3\server\lib\wlfullclient.jar -Duser=weblogic -Dpass=weblogic1 -Djava.naming.factory.initial=weblogic.jndi.WLInitialContextFactory -Dplugins.include=jmx -jar C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5.jar jmx MBeanDumper service:jmx:iiop://myhost:8001/jndi/weblogic.management.mbeanservers.runtime

but adding the wlfullclient.jar is not enough, because the classpath is set in the MANIFEST.MF of the hq-pdk-4.6.5.jar:

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: bob
Build-Jdk: 1.6.0_29
Main-Class: org.hyperic.hq.product.util.PluginMain
Class-Path: hq-common-4.6.5.jar hq-util-4.6.5.jar commons-logging-1.0.
 4.jar httpclient-4.1.1.jar httpcore-4.1.jar commons-codec-1.4.jar com
 mons-validator-1.3.1.jar commons-beanutils-1.7.0.jar commons-digester
 -1.6.jar commons-collections-2.1.1.jar xml-apis-1.3.04.jar jsch-0.1.4
 2.jar log4j-1.2.14.jar velocity-1.6.3.jar commons-lang-2.4.jar oro-2.
 0.8.jar sigar-1.6.4.jar jdom-1.1.jar spring-beans-3.0.5.RELEASE.jar s
 pring-asm-3.0.5.RELEASE.jar spring-context-3.0.5.RELEASE.jar spring-a
 op-3.0.5.RELEASE.jar aopalliance-1.0.jar spring-expression-3.0.5.RELE
 ASE.jar org.springframework.roo.file.monitor-1.0.2.RELEASE.jar org.sp
 ringframework.roo.support-1.0.2.RELEASE.jar org.springframework.roo.f
 ile.monitor.polling-1.0.2.RELEASE.jar wrapper-3.5.6.jar xstream-1.3.1
 .jar xpp3_min-1.1.4c.jar hibernate-3.2.6.ga.jar asm-attrs-1.5.3.jar d
 om4j-1.6.1.jar antlr-2.7.6.jar cglib-2.1_3.jar asm-1.5.3.jar spring-t
 x-3.0.5.RELEASE.jar jboss-remoting-2.2.2.SP8.jar jboss-common-logging
 -spi-2.0.4.GA.jar hq-lather-4.6.5.jar backport-util-concurrent-3.1.ja
 r getopt-1.0.13.jar geronimo-j2ee-management_1.0_spec-1.1.jar snmp4j-
 1.11.jar xalan-2.7.1.jar serializer-2.7.1.jar cm-file-monitor-1.0.1.j
 ar cm-versioncontrol-1.0.1.jar slf4j-log4j12-1.6.1.jar slf4j-api-1.6.
 1.jar jcl-over-slf4j-1.6.1.jar org.eclipse.jgit-1.0.0.201106090707-r.
 jar org.eclipse.jgit.junit-1.0.0.201106090707-r.jar spring-oxm-3.0.5.
 RELEASE.jar jpathwatch-0-94.jar jaxb-api-2.1.jar stax-api-1.0-2.jar a
 ctivation-1.1.jar jaxb-impl-2.1.3.jar

all those classes are in
C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib

I copy there the wlfullclient.jar

and I run
java -cp "C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\*" -Duser=weblogic -Dpass=weblogic1 -Djava.naming.factory.initial=weblogic.jndi.WLInitialContextFactory -Dplugins.include=jmx org.hyperic.hq.product.util.PluginMain jmx MBeanDumper service:jmx:iiop://myhost:8001/jndi/weblogic.management.mbeanservers.runtime


and I get

java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root ex
ception is java.net.ConnectException: t3://localhost:7001: Destination unreachable; nested exception
 is:
        java.net.ConnectException: Connection refused: connect; No available router to destination]
        at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
        at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
        at org.hyperic.hq.product.jmx.MxUtil.getMBeanConnector(MxUtil.java:538)
        at org.hyperic.hq.product.jmx.MxUtil.connect(MxUtil.java:564)
        at org.hyperic.hq.product.jmx.MxUtil.getMBeanServer(MxUtil.java:621)
        at org.hyperic.hq.product.jmx.MBeanDumper.getMBeanServer(MBeanDumper.java:138)
        at org.hyperic.hq.product.jmx.MBeanDumper.dump(MBeanDumper.java:164)
        at org.hyperic.hq.product.jmx.MBeanDumper.dump(MBeanDumper.java:160)
        at org.hyperic.hq.product.jmx.MBeanDumper.main(MBeanDumper.java:235)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.hyperic.hq.product.util.PluginMain.runMain(PluginMain.java:218)
        at org.hyperic.hq.product.util.PluginMain.main(PluginMain.java:317)

Were on earth is this "t3://localhost:7001" ? crap!!!


First, enable your Java process to be monitored by remote jmx (see help name="Java" at the end of the myfirstplugin.xml plugin below)

take this file myfirstplugin.xml and put it into $HQ_AGENT_HOME/pdk/plugins
(like /opt/hyperic/agent/agent-4.6.5-EE/bundles/agent-4.6.5/pdk/plugins)

and adjust the localhost:6969 bit:

<?xml version="1.0"?>
 
<!DOCTYPE plugin [<!ENTITY process-metrics SYSTEM "/pdk/plugins/process-metrics.xml">]>
 
<plugin package="org.hyperic.hq.plugin.java">
 
<classpath>
<include name="pdk/lib/mx4j"/>
</classpath>
 
<filter name="template" value="${OBJECT_NAME}:${alias}"/>
 
<metrics name="Class Loading Metrics">
<metric name="Loaded Class Count" indicator="false" category="THROUGHPUT"/>
<metric name="Total Loaded Class Count" indicator="false" category="THROUGHPUT"/>
<metric name="Unloaded Class Count" indicator="false" category="THROUGHPUT"/>
</metrics>
 
<metrics name="Compilation">
<metric name="Total Compilation Time" indicator="false" category="THROUGHPUT" collectionType="trendsup" units="ms"/>
</metrics>
 
<metrics name="Garbage Collector">
<metric name="Collection Count" indicator="false" category="THROUGHPUT" collectionType="trendsup"/>
<metric name="Collection Time" indicator="false" category="THROUGHPUT" collectionType="trendsup"/>
</metrics>
 
<metrics name="Memory">
<metric name="Object Pending Finalization Count" category="THROUGHPUT" indicator="false"/>
</metrics>
 
<metrics name="Threading">
<metric name="Thread Count" category="UTILIZATION" indicator="false"/>
<metric name="Daemon Thread Count" category="UTILIZATION" indicator="false"/>
</metrics>
 
<server name="Java" version="1.5.x">
<property name="HAS_BUILTIN_SERVICES" value="true"/>
<property name="VERSION_FILE" value="jre/lib/fontconfig.Sun.2003.bfc"/>
<property name="DEFAULT_PROGRAM" value="bin/java"/>
<property name="domain" value="Java"/>
 
<config>
<option name="jmx.url" description="JMX URL to MBeanServer"  default="service:jmx:rmi:///jndi/rmi://localhost:6969/jmxrmi"/>
<option name="jmx.username" description="JMX username" optional="true" default=""/>
<option name="jmx.password" description="JMX password" optional="true" default="" type="secret"/>
<option name="process.query" description="PTQL for Java Process" default="State.Name.eq=java,Args.*.ct=proc.java.home"/>
</config>
 
<metric name="Availability" template="sigar:Type=ProcState,Arg=%process.query%:State" indicator="true"/>
&process-metrics;
 
<property name="OBJECT_NAME" value="java.lang:type=ClassLoading"/>
 
<metrics include="Class Loading Metrics"/>
<property name="OBJECT_NAME" value="java.lang:type=Compilation"/>
 
<metrics include="Compilation"/>
<property name="OBJECT_NAME" value="java.lang:type=Memory"/>
 
<plugin type="log_track" class="org.hyperic.hq.product.jmx.MxNotificationPlugin"/>
 
<property name="OBJECT_NAME" value="java.lang:type=Threading"/>
<metrics include="Threading"/>
 
<!-- derive installpath from JAVA_HOME env prop... -->
<property name="PROC_HOME_ENV" value="JAVA_HOME"/>
 
<!-- derive installpath from -Dproc.java.home=... -->
<property name="PROC_HOME_PROPERTY" value="proc.java.home"/>
<plugin type="autoinventory" class="org.hyperic.hq.product.jmx.MxServerDetector"/>
<plugin type="measurement" class="org.hyperic.hq.product.jmx.MxMeasurementPlugin"/>
 
<service name="Java GC">
<plugin type="autoinventory"/>
<property name="OBJECT_NAME" value="java.lang:type=GarbageCollector,name=*"/>
<metrics include="Garbage Collector"/>
</service>
</server>
 
<server name="Java" version="1.6.x" include="1.5.x">
<property name="VERSION_FILE" value="jre/lib/management-agent.jar"/>
</server>
 
<!--
 ==================== Plugin Help ===========================
-->
<help name="Java">
<![CDATA[
  <p>
  <h3>Configure HQ for monitoring Java</h3>
  </p>
  <p>
  1) Add this line to the java options when executing the binary.
  <br>
  "-Dcom.sun.management.jmxremote \
  <br>
  -Dcom.sun.management.jmxremote.port=6969 \
  <br>
  -Dcom.sun.management.jmxremote.ssl=false \
  <br>
  -Dcom.sun.management.jmxremote.authenticate=false"
  <br>
  </p>
]]>
</help>
<help name="Java 1.5.x" include="Java"/>
<help name="Java 1.6.x" include="Java"/>
</plugin>

restart the agent:

./hq-agent.sh restart

Notifications in Hyperic HQApi

Normally in the HQ GUI you can setup an email notification, in this example the user "pippo" would be notified:

AlertDefinition d = new AlertDefinition();
AlertAction emailAlertAction = new AlertAction();

emailAlertAction.setClassName("com.hyperic.hq.bizapp.server.action.email.EmailAction");

AlertActionConfig alertActionConfigNotifyTypes = new AlertActionConfig();
alertActionConfigNotifyTypes.setKey("notifyType");
alertActionConfigNotifyTypes.setValue("users");

AlertActionConfig alertActionConfigNames = new AlertActionConfig();
alertActionConfigNotifyTypes.setKey("names");
alertActionConfigNotifyTypes.setValue("pippo");

emailAlertAction.getAlertActionConfig().add(alertActionConfigNotifyTypes);
emailAlertAction.getAlertActionConfig().add(alertActionConfigNames);

d.getAlertAction().add(emailAlertAction);

Unfortunately in Hyperic 4.5.3 this doesn't seem to work (see also https://jira.hyperic.com/browse/HQAPI-147)

The only way which seems to work in HQApi is by escalation:

    <AlertDefinition mtime="1336305843209" ctime="1336305430467" id="10745" name="[Down]" description="[Down]" priority="2" enabled="true" active="true" frequency="0" count="0" range="0" willRecover="true" notifyFiltered="false" controlFiltered="false">
        <Resource id="12473" name="/tmp/hyperictestdir"/>
        <Escalation id="10001" name="ACME Project Team Escalation" description="" pauseAllowed="true" maxPauseTime="9223372036854775807" notifyAll="false" repeat="true"/>
        <AlertCondition required="true" type="1" thresholdValue="0.0" thresholdComparator="=" thresholdMetric="Availability"/>
    </AlertDefinition>

The escalation "ACME Project Team Escalation" must have been previously setup in Administration / Escalation Schemes Configuration
or with the org.hyperic.hq.hqapi1.EscalationApi.createEscalation() API

You can also retrieve existing escalations by name with EscalationApi.getEscalation(String name)

Creating Alert Recovery with Hyperic HQApi

Some doc on the Alert Condition topic

In general, you should setup two AlertDefinitions, a [Down] and a [Fixed - Down] wchich recovers [Down]:

[Down]
If Condition: JMS Destination Availability = 0.0%
Enable Action(s): Each time conditions are met.
Generate one alert and then disable alert definition until fixed

and

[Fixed - Down]
If Condition: JMS Destination Availability = 100.0%
Recovery Alert: for [Down]
Enable Action(s): Each time conditions are met.


The AlertDefinitions are:

<AlertDefinition mtime="1336300002609" ctime="1336300002609" id="10737" name="[Down]" description="" priority="2" enabled="true" active="true" frequency="0" count="0" range="0" willRecover="true" notifyFiltered="false" controlFiltered="false">
        <Resource id="12469" name="/tmp/hyperictestdir"/>
        <AlertCondition required="true" type="1" thresholdValue="0.0" thresholdComparator="=" thresholdMetric="Availability"/>
    </AlertDefinition>
    <AlertDefinition mtime="1336300075132" ctime="1336300075132" id="10738" name="[Fixed- Down]" description="" priority="2" enabled="true" active="true" frequency="0" count="0" range="0" willRecover="false" notifyFiltered="false" controlFiltered="false">
        <Resource id="12469" name="/tmp/hyperictestdir"/>
        <AlertCondition required="true" type="1" thresholdValue="1.0" thresholdComparator="=" thresholdMetric="Availability"/>
        <AlertCondition required="true" type="5" recover="[Down]" recoverId="10737"/>
    </AlertDefinition>

The bit "Generate one alert and then disable alert definition until fixed" correspond to the attribute "willRecover=true" in the [Down] alert.

If you create an alert whenever a FileServerDirectory (or any other resource) goes DOWN, you will get alerts all the time until you fix it. You might want to have the alert triggered only once, and then another UP alert when it's fixed.

In the first scenario, you do:

Resource res = .... (retrieve or create the resource, can be a platform, a server or a service)
String name = "my first alert";
// create the AlertDefinition
AlertDefinition alertDefinition = new AlertDefinition();
alertDefinition.setName(name);
alertDefinition.setDescription(name);
alertDefinition.setPriority(AlertPriority.MEDIUM.getPriority());
alertDefinition.setActive(true);
alertDefinition.setWillRecover(true);

// assign the AlertDefinition to the Resource
alertDefinition.setResource(res);
alertDefinition.setEnabled(true);

// create the AlertCondition
AlertCondition thresholdCondition0 = AlertDefinitionBuilder.createThresholdCondition(true, ALERT_METRIC_AVAILABILITY, AlertComparator.EQUALS, 0.0);

// assign the AlertCondition to the AlertDefinition 
alertDefinition.getAlertCondition().add(thresholdCondition0);

List definitions = new ArrayList();
definitions.add(alertDefinition);

AlertDefinitionsResponse response = api.syncAlertDefinitions(definitions);

So far it works, but you will get an alert a minute (depending on your aggregation time)

To enable recovery, you must also:


AlertDefinition recoveryDefinition = .... create another AlertDefinition
AlertCondition recoveryCondition = AlertDefinitionBuilder.createRecoveryCondition(true, alertDefinition);
AlertCondition thresholdCondition100 = AlertDefinitionBuilder.createThresholdCondition(true, ALERT_METRIC_AVAILABILITY, AlertComparator.EQUALS, 1.0);
recoveryDefinition.getAlertCondition().add(thresholdCondition100);
recoveryDefinition.getAlertCondition().add(recoveryCondition);


and add also this recoveryDefinition to the List of AlertDefinition to sync:


definitions.add(recoveryDefinition);
AlertDefinitionsResponse response = api.syncAlertDefinitions(definitions);


This way you have 2 Alerts for the same resource, one triggers on DOWN and the other on UP to automatically fix the DOWN alert.

Hyperic MBeanDumper

According to the Hyperic doc, one should run this:

java -Duser=weblogic -Dpass=weblogic1 -Dplugins.include=jmx -jar pdk/lib/hq-product.jar jmx MBeanDumper service:jmx:rmi:///jndi/rmi://myserver:1099/jmxrmi

this fails with

Failed to load Main-Class manifest attribute from
pdk/lib/hq-product.jar

I google and find that one is supposed to use a hq-pdk-shared-VERSION.jar .....to be found nowhere, even looking for hq-pdk-shared

At last I try this:

cd C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5

java -Duser=weblogic -Dpass=weblogic1 -Dplugins.include=jmx -cp C:\pierre\hypericagent\agent-4.6.5-EE\bundles\agent-4.6.5\pdk\lib\hq-pdk-4.6.5
.jar org.hyperic.hq.product.jmx.MBeanDumper service:jmx:rmi:///jndi/rmi://myserver:8888/jmxrmi


and it works... with some exceptions.... anyway it doesn't provide insight in the WebLogic MBean tree.... maybe the connection parameters should be different, and the jar file wlfullclient.jar should be added to the classpath

On the whole, I would think Hyperic is in urgent need of a good documentation technical writer.

Batch creation of Resources in Hyperic

Here http://svn.hyperic.org/projects/hqapi/trunk/src/org/hyperic/hq/hqapi1/test/ plenty of Java code examples on how to use HQApi.

I will drop Groovy and Python and use only Java to automate Hyperic configuration.

Playing with Resource and XmlStarlet in Hyperic

This command extract all resources from our HQ Server:

hqapi resource list -prototype Linux --children > allresources.xml

The XPath to extract all resources of type "FileServer Directory" is:

//Resource[./ResourcePrototype/@name="FileServer Directory"]


and the syntax to execute it in XmlStarlet is:


c:\apps\xmlstarlet-1.3.1\xml sel -t -c "//Resource[./ResourcePrototype/@name='FileServer Directory']" allresources.xml


This resource is a child element of another Resource:

<Resource id="10916" name="myserver.acme.com Linux FileServer" description="" location="" instanceId="10007" typeId="2">

which in turn is inside

<Resource id="10882" name="myserver.acme.com" description="Red Hat Enterprise Linux 5" location="" instanceId="10001" typeId="1">

To create a resource, the easy way is:
hqapi resource createService --prototype="FileServer Directory" --resourceId=10882 --name="/data/acme/" path="/data/acme/"


XmlStarlet doc is available here

Hyperic, exploring (and exporting) autodiscovered resources with HQApi

http://support.hyperic.com/display/DOC/HQApi+3.2+resource+command


List details of a platform (machine)

cd C:\pierre\workspace\Hyperic\lib\hqapi1-client-5.0.0\bin
hqapi resource list -platform myhost.acme.com

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ResourcesResponse>
    <Status>Success</Status>
    <Resource id="11328" name="myhost.acme.com" description="Red Hat Enterprise Linux 5"
location="" instanceId="10003" typeId="1">
        <ResourcePrototype resourceTypeId="1" instanceId="10001" id="10001" name="Linux"/>
        <Agent id="10003" address="10.58.25.132" port="2144" version="4.6.5" unidirectional="false"/>

        <Ip address="10.58.25.132" mac="34:8E:99:27:FD:80" netmask="255.255.255.0"/>
        <Ip address="127.0.0.1" mac="00:00:00:00:00:00" netmask="255.0.0.0"/>
        <ResourceInfo key="fqdn" value="myhost.acme.com"/>
    </Resource>
</ResourcesResponse>

List all details of a all Linux platforms (machine)

hqapi resource list -prototype Linux

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ResourcesResponse>
    <Status>Success</Status>
    <Resource id="10882" name="myhost1.acme.com" description="Red Hat Enterprise Linux 5"
location="" instanceId="10001" typeId="1">
        <ResourcePrototype resourceTypeId="1" instanceId="10001" id="10001" name="Linux"/>
        <Agent id="10001" address="10.56.5.191" port="2144" version="4.5.3" unidirectional="false"/>

        <Ip address="10.56.53.191" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="10.56.53.174" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="127.0.0.1" mac="00:00:00:00:00:00" netmask="255.0.0.0"/>
        <ResourceInfo key="fqdn" value="myhost1.acme.com"/>
    </Resource>

    <Resource id="10883" name="myhost2.acme.com" description="Red Hat Enterprise Linux 5"
location="" instanceId="10002" typeId="1">
        <ResourcePrototype resourceTypeId="1" instanceId="10001" id="10001" name="Linux"/>
        <Agent id="10001" address="10.56.5.192" port="2144" version="4.5.3" unidirectional="false"/>

        <Ip address="10.56.53.191" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="10.56.53.174" mac="99:8E:99:27:FD:98" netmask="255.255.255.0"/>
        <Ip address="127.0.0.1" mac="00:00:00:00:00:00" netmask="255.0.0.0"/>
        <ResourceInfo key="fqdn" value="myhost.acme.com"/>
    </Resource>
</ResourcesResponse>

Something really fantastic is when you do:

hqapi resource list -prototype Linux --children > myresources.xml

this will dump the ENTIRE database of resources!

And the real great thing is that you can
resource sync --file myresources.xml

to import into a HQ Server the resources exported by "resource list"

Hyperic HQApi and the Groovy Console

HQApi documentation here

"HQApi allows you to circumvent the Hyperic user interface and directly access Hyperic Server functionality. " ... great, I love it!

Let's get our hands dirty....

On HQ Console, Click on Administration, Groovy Console...

type:
print 'ciao'
return 'pippo'

and click "execute"... great, in the "result" tab I see "pippo".... where is "ciao" though? I can't find the stdout of "Server for Hyperic HQ" Windows service.

Scripts are stored in a temporary file C:\pierre\hypericserver\server-4.6.5-EE\hq-engine\hq-server\temp\gcon3792879954783360219.tmp

I click on "HQ Web Services Api" and download the hqapi1-client-5.0.0.tar.gz

The most common useful classes to run the examples in Hyperic site are in

<classpathentry kind="lib" path="C:/pierre/workspace/Hyperic/lib/hqapi1-client-5.0.0/hqapi1-5.0.0.jar"/>
<classpathentry kind="lib" path="C:/pierre/workspace/Hyperic/lib/hqapi1-client-5.0.0/hqapi1-tools-5.0.0.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/hq-rendit-4.6.5.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/hq-server-4.6.5.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/httpclient-4.1.1.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/httpcore-4.1.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericagent/agent-4.6.5-EE/bundles/agent-4.6.5/pdk/lib/httpmime-4.1.1.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/hq-engine/hq-server/webapps/ROOT/WEB-INF/lib/activemq-all-5.3.0.jar"/>
<classpathentry kind="lib" path="C:/pierre/workspace/Hyperic/lib/hqapi1-client-4.3.0/lib/spring-core-3.0.1.RELEASE.jar"/>
<classpathentry kind="lib" path="C:/pierre/hypericserver/server-4.6.5-EE/lib/commons-codec-1.1.jar"/>



Some Groovy Console samples are here:

http://support.hyperic.com/display/hyperforge/Groovy+Snippets