Book: The Logstash Book

http://www.logstashbook.com/ .
I was hoping this book would take me by the hand in small, progressive steps to harness to power of Logstash.
It proved so far to be a confusing book, piling up a lot of disparate information in a quite chaotic way.
It tries to make you set up a very complicated example juggling with a lot of products (syslog with redis and elastisearch, forwarder...) instead of going in a progressive, step by step, way.
Also, scripts are provided only for Ubuntu, so if you use RHEL you are on your own.
Which proves, once more, that one thing is being a great technician, one thing is being a great pedagogue.

However, it's still worth buying and reading; logs are of a capital importance and most of the time they are very poorly managed, so I need to educate myself on the state-of-the-art technology.

Redis

Training here http://try.redis.io/
To install:

yum install redis
vi /etc/redis.conf
comment out the "bind 127.0.0.1" line
/usr/sbin/redis-server /etc/redis.conf

ps -ef | grep redis
root      4092     1  0 21:27 ?        00:00:00 /usr/sbin/redis-server /etc/redis.conf

/usr/bin/redis-cli ping
PONG

netstat -an | grep 6379

tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN

if in /etc/redis.conf you specify a "bind IP" (IP=10.0.2.159 then you have to use the -h option:
/usr/bin/redis-cli -h 10.0.2.15 shutdown

you can issue several commands:
set key value
get key
incr key

Elasticsearch

download the rmp: http://www.elasticsearch.org/download/
rpm -i elasticsearch-0.90.9.noarch.rpm
ps -ef | grep elasti

496       1734     1  1 10:32 ?        00:00:10 /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-0.90.9.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch

less /var/log/elasticsearch/elasticsearch.log

[2014-01-01 10:29:43,058][INFO ][node                     ] [Kala] version[0.90.9], pid[1734], build[a968646/2013-12-23T10:35:28Z]
[2014-01-01 10:29:43,058][INFO ][node                     ] [Kala] initializing ...
[2014-01-01 10:29:43,063][INFO ][plugins                  ] [Kala] loaded [], sites []
[2014-01-01 10:29:45,289][INFO ][node                     ] [Kala] initialized
[2014-01-01 10:29:45,289][INFO ][node                     ] [Kala] starting ...
[2014-01-01 10:29:45,377][INFO ][transport                ] [Kala] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.0.2.15:9300]}
[2014-01-01 10:29:48,424][INFO ][cluster.service          ] [Kala] new_master [Kala][Z9XUjvk0QxK6aXyCOhExqg][inet[/10.0.2.15:9300]], reason: zen-disco-join (elected_as_master)
[2014-01-01 10:29:48,498][INFO ][discovery                ] [Kala] elasticsearch/Z9XUjvk0QxK6aXyCOhExqg
[2014-01-01 10:29:48,526][INFO ][http                     ] [Kala] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.2.15:9200]}
[2014-01-01 10:29:48,527][INFO ][node                     ] [Kala] started
[2014-01-01 10:29:48,543][INFO ][gateway                  ] [Kala] recovered [0] indices into cluster_state
[2014-01-01 10:35:47,549][INFO ][cluster.service          ] [Kala] added {[Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]],}, reason: zen-disco-receive(join from node[[Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]]])
[2014-01-01 10:42:00,830][INFO ][cluster.service          ] [Kala] removed {[Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]],}, reason: zen-disco-node_left([Grant, Greer][-TR-MXApSOqMcPWw7_qYMg][inet[/10.0.2.15:9301]])

Watch the excellent tutorial

curl 10.0.2.15:9200

{
  "ok" : true,
  "status" : 200,
  "name" : "Kala",
  "version" : {
    "number" : "0.90.9",
    "build_hash" : "a968646da4b6a2d9d8bca9e51e92597fe64e8d1a",
    "build_timestamp" : "2013-12-23T10:35:28Z",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

Let's put some data:

curl -XPUT 10.0.2.15:9200/books/eco/one -d '
> {
>   "author" : "pierre",
>   "title" : "how to cook Pizza"
> }'

I get a response:

{"ok":true,"_index":"books","_type":"eco","_id":"one","_version":1}

I can examine the mapping:

curl 10.0.2.15:9200/books/_mapping

{"books":{"eco":{"properties":{"author":{"type":"string"},"title":{"type":"string"}}}}}

I can get my document back:

curl 10.0.2.15:9200/books/eco/one

{"_index":"books","_type":"eco","_id":"one","_version":1,"exists":true, "_source" :
{
  "author" : "pierre",
  "title" : "how to cook Pizza"
}}

I can search based on an attribute:
curl 10.0.2.15:9200/books/_search?q=_author=pierre

{"took":71,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.02250402,"hits":[{"_index":"books","_type":"eco","_id":"one","_score":0.02250402, "_source" :
{
  "author" : "pierre",
  "title" : "how to cook Pizza"
}}]}}

start and stop:
/etc/init.d/elasticsearch start
/etc/init.d/elasticsearch stop


edit configuration:
vi /etc/elasticsearch/elasticsearch.yml

To view configuration: http://youripaddress:9200/ and more http://youripaddress:9200/_status?pretty=true



If you get Content-Type header [application/x-www-form-urlencoded] is not supported", add this: -H 'Content-Type: application/json'

If you get "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes", put the json in a file pizza.json and invoke curl -H 'Content-Type: application/json' -XPUT 10.0.2.15:9200/books/eco/one -d@pizza.json


References: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html#getting-started