ELK docker

http://elk-docker.readthedocs.io/

you have to adjust permanently the max_map_count parameter:

sysctl -w vm.max_map_count=262144

(sysctl - configure kernel parameters at runtime)

ls /proc/sys/vm to get list of available parameters

sudo vi /etc/sysctl.conf
vm.max_map_count=262144

try starting the container like this:

sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

if it fails with this error:

waiting for Elasticsearch to be up (30/30)
Couln't start Elasticsearch. Exiting.

try allowing more time:

sudo docker run -e ES_CONNECT_RETRY=300 -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

(see https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file )

then

http://127.0.0.1:5601/app/kibana KIBANA (you have to wait 1 minute for Kibana to come up)

http://127.0.0.1:9200/ Elasticsearch JSON

to create dummy entry:

docker exec -it elk /bin/bash
/opt/logstash/bin/logstash --path.data /tmp/logstash/data -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'
this is a dummy entry
this is a dummy entry2
CTRL-C


Kibana logs : less /var/log/kibana/kibana5.log
Elasticsearch logs : less /var/log/elasticsearch/elasticsearch.log
Logstash logs: less /var/log/logstash/logstash-plain.log

tail -f /var/log/elasticsearch/elasticsearch.log /var/log/logstash/logstash-plain.log /var/log/kibana/kibana5.log



docker network create -d bridge elknet


good practical presentation of ELK:

osb, logback, logstash, grok and elasticsearch, putting it all together

the Java class that I invoke with a custom XPath to trace events :

• "interface" is like "service/operation"
• "eventtype" is like "FileConsumed", "JMSMessageConsumed", "WSInvoked"
• TechnicalMessageID and BusinessID are usinque identifiers for the request/payload
• ServerName is the WebLogic managed server
• Priority is P1... P5 in case of error
• Payload is the actual request... it's traced only if level is "DEBUG"
• Fault is the error description

package com.acme.osb.logging;

import org.apache.xmlbeans.XmlObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Used by OSB to report an Event
 * 
 * @author NNVernetPI
 * 
 */

public class MessageTrackerSLF4J {

 public static String logMessage(String technicalMessageid,
   String businessId, String eventType, String interfaceName,
   XmlObject payload) {

  Logger projectLogger = LoggerFactory.getLogger(interfaceName);

  if (projectLogger.isDebugEnabled()) {
   projectLogger.info(" ::InterfaceName:: {} ::EventType:: {} ::TechnicalMessageID:: {} ::BusinessID:: {} ::ServerName:: {} ::Priority:: {} ::Payload:: {} ::Fault:: {}", 
     interfaceName, eventType, technicalMessageid, businessId, System.getProperty("weblogic.Name"), "NONE", payload != null ? payload.xmlText().replaceAll("\\r\\n|\\r|\\n", " ") : "NONE", "NONE");

  } else {
   projectLogger.info(" ::InterfaceName:: {} ::EventType:: {} ::TechnicalMessageID:: {} ::BusinessID:: {} ::ServerName:: {} ::Priority:: {} ::Payload:: {} ::Fault:: {}", 
     interfaceName, eventType, technicalMessageid, businessId, System.getProperty("weblogic.Name"), "NONE", "NONE", "NONE");
  }

  String logmessage = "Info Message Logged for:: " + interfaceName;
  return logmessage;
 }

 public static String errorLogger(String technicalMessageid,
   String businessId, String interfaceName, XmlObject payload,
   String priority, XmlObject fault) {

  Logger projectLogger = LoggerFactory.getLogger(interfaceName);

  projectLogger.error(" ::InterfaceName:: {} ::EventType:: {} ::TechnicalMessageID:: {} ::BusinessID:: {} ::ServerName:: {} ::Priority:: {} ::Payload:: {} ::Fault:: {}", 
    interfaceName, "ERROR", technicalMessageid, businessId, System.getProperty("weblogic.Name"), priority, payload != null ? payload.xmlText().replaceAll("\\r\\n|\\r|\\n", " ") : "NONE", fault != null ? fault.xmlText().replaceAll("\\r\\n|\\r|\\n", " ") : "NONE");

  String responseMessage = "Error Message Logged for:: " + interfaceName;
  return responseMessage;
 }

}

To make my parsing easier, I have decided to remove all newlines from payload and fault, so as to have a single line per event in the logs.

You can also deal with multiline with this codec.

For a valid message, logback will log:
#### 2014-01-08 18:04:01,980 INFO [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] - ::InterfaceName:: ProdDeclAven ::EventType:: FileConsumed ::TechnicalMessageID:: ProdDeclAven^BAD_TST_PDA_REV.xml^AVE^1389200641480 ::BusinessID:: 000000000006741320_00376401092900001530 ::ServerName:: osbdev1ms1 ::Priority:: NONE ::Payload:: NONE ::Fault:: NONE ####

logback.xml file:

<?xml version="1.0" ?>
<configuration debug="true" scan="true" scanPeriod="30 seconds">
 <jmxConfigurator/>
 <property name="LOG_DIR" value="/opt/var/log/weblogic/server/"/>
 <appender class="ch.qos.logback.core.ConsoleAppender" name="STDOUT">
  <encoder>
   <pattern>
    #### %date{ISO8601} %level [%thread] - %msg ####%n            
   </pattern>
  </encoder>
 </appender>
 <appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="MyService">
  <file>
   ${LOG_DIR}acmeMyService.log
  </file>
  <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
   <fileNamePattern>
    acmeMyService.%i.log.zip
   </fileNamePattern>
   <minIndex>
    1
   </minIndex>
   <maxIndex>
    10
   </maxIndex>
  </rollingPolicy>
  <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
   <maxFileSize>
    50MB
   </maxFileSize>
  </triggeringPolicy>
  <encoder>
   <pattern>
    #### %date{ISO8601} %level [%thread] - %msg ####%n                
   </pattern>
  </encoder>
 </appender>

 <logger additivity="false" level="INFO" name="MyService_PSDB_RoutingService">
  <appender-ref ref="MyService"/>
 </logger>

 <root level="DEBUG">
  <appender-ref ref="ALL"/>
 </root>
</configuration>

esgrok.conf will be

input {
  file {
    path => "/opt/var/log/weblogic/server/nesoa2*.log"
  }

}
filter {
  grok {
    match => [ "message", "#### %{TIMESTAMP_ISO8601:timestamp} %{WORD:level} \[\[%{WORD:threadstatus}\] %{GREEDYDATA:threadname}\] -  ::InterfaceName:: %{WORD:interfacename} ::EventType:: %{WORD:eventtype} ::TechnicalMessageID:: %{GREEDYDATA:technicalmessageid} ::BusinessID:: %{GREEDYDATA:businessid} ::ServerName:: %{WORD:servername} ::Priority:: %{WORD:priority} ::Payload:: %{GREEDYDATA:payload} ::Fault:: %{GREEDYDATA:fault} ####" ]
  }
}

output {
  elasticsearch {
    embedded => true
  }
}

Preparing the grok match regexp was very time consuming. Using http://grokdebug.herokuapp.com/ the grok debugger was essential. It worked for me only on Chrome, not on Firefox.

Priceless also the list of grok patterns.

Run with nohup java -jar logstash-1.3.2-flatjar.jar agent -f esgrok.conf -- web > logstash.log 2>&1 &
The result is impressive.
Happy Kibana to you!

Elasticsearch

download the rmp: http://www.elasticsearch.org/download/
rpm -i elasticsearch-0.90.9.noarch.rpm
ps -ef | grep elasti

496       1734     1  1 10:32 ?        00:00:10 /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-0.90.9.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch

less /var/log/elasticsearch/elasticsearch.log

[2014-01-01 10:29:43,058][INFO ][node                     ] [Kala] version[0.90.9], pid[1734], build[a968646/2013-12-23T10:35:28Z]
[2014-01-01 10:29:43,058][INFO ][node                     ] [Kala] initializing ...
[2014-01-01 10:29:43,063][INFO ][plugins                  ] [Kala] loaded [], sites []
[2014-01-01 10:29:45,289][INFO ][node                     ] [Kala] initialized
[2014-01-01 10:29:45,289][INFO ][node                     ] [Kala] starting ...
[2014-01-01 10:29:45,377][INFO ][transport                ] [Kala] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.0.2.15:9300]}
[2014-01-01 10:29:48,424][INFO ][cluster.service          ] [Kala] new_master [Kala][Z9XUjvk0QxK6aXyCOhExqg][inet[/10.0.2.15:9300]], reason: zen-disco-join (elected_as_master)
[2014-01-01 10:29:48,498][INFO ][discovery                ] [Kala] elasticsearch/Z9XUjvk0QxK6aXyCOhExqg
[2014-01-01 10:29:48,526][INFO ][http                     ] [Kala] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.2.15:9200]}
[2014-01-01 10:29:48,527][INFO ][node                     ] [Kala] started
[2014-01-01 10:29:48,543][INFO ][gateway                  ] [Kala] recovered [0] indices into cluster_state
[2014-01-01 10:35:47,549][INFO ][cluster.service          ] [Kala] added {[Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]],}, reason: zen-disco-receive(join from node[[Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]]])
[2014-01-01 10:42:00,830][INFO ][cluster.service          ] [Kala] removed {[Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]],}, reason: zen-disco-node_left([Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]])

Watch the excellent tutorial

curl 10.0.2.15:9200

{
  "ok" : true,
  "status" : 200,
  "name" : "Kala",
  "version" : {
    "number" : "0.90.9",
    "build_hash" : "a968646da4b6a2d9d8bca9e51e92597fe64e8d1a",
    "build_timestamp" : "2013-12-23T10:35:28Z",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

Let's put some data:

curl -XPUT 10.0.2.15:9200/books/eco/one -d '
> {
>   "author" : "pierre",
>   "title" : "how to cook Pizza"
> }'

I get a response:

{"ok":true,"_index":"books","_type":"eco","_id":"one","_version":1}

I can examine the mapping:

curl 10.0.2.15:9200/books/_mapping

{"books":{"eco":{"properties":{"author":{"type":"string"},"title":{"type":"string"}}}}}

I can get my document back:

curl 10.0.2.15:9200/books/eco/one

{"_index":"books","_type":"eco","_id":"one","_version":1,"exists":true, "_source" :
{
  "author" : "pierre",
  "title" : "how to cook Pizza"
}}

I can search based on an attribute:
curl 10.0.2.15:9200/books/_search?q=_author=pierre

{"took":71,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.02250402,"hits":[{"_index":"books","_type":"eco","_id":"one","_score":0.02250402, "_source" :
{
  "author" : "pierre",
  "title" : "how to cook Pizza"
}}]}}

start and stop:
/etc/init.d/elasticsearch start
/etc/init.d/elasticsearch stop


edit configuration:
vi /etc/elasticsearch/elasticsearch.yml

To view configuration: http://youripaddress:9200/ and more http://youripaddress:9200/_status?pretty=true



If you get Content-Type header [application/x-www-form-urlencoded] is not supported", add this: -H 'Content-Type: application/json'

If you get "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes", put the json in a file pizza.json and invoke curl -H 'Content-Type: application/json' -XPUT 10.0.2.15:9200/books/eco/one -d@pizza.json


References: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html#getting-started