LOGSTASH

 

INSTALLATION OF LOGSTASH SERVER.

Do not take this intallation documentation as a literal bible of how to install logstash. Systems will vary and needs will vary and I can almost guarantee that you will have to make some tweaks, changes and adjustments relative to your system and your particular needs in order to get a working logstash installation that makes you happy. I cannot guarantee that you will not completely crash your system while trying to install Logstash (though you’d have to try pretty darn hard). I have said it on other parts of this site and I will say it now: USE A TESTING SERVER THE FIRST TIME YOU INSTALL LOGSTASH!! ALWAYS DO A TEST INSTALL FIRST! You should never install Logstash, or any other program for that matter, on a live production server until you have tested it thoroughly and you are confident in your ability to handle the install and any problems that may arise as a result of the install.

Before You Start! – When installing logstash, I would STRONGLY reccomend that you first conduct a test installation on a test server if you have one available. I would recommend you to install Ubuntu 12.04 full installation for installation of a Logstash server so that all the packages needed will be installed. If you’re new to logstash, chances are you will probably screw the install up the first time through. Do a favor to first install on a test server that has no importance to you, so that in case you come across any issues it can be resolved.

What to expect from this installation guide

First and foremost, know this: The Source installation of Logstash DOES WORK. It’s been tested time and time again. But don’t get me wrong, the guide is not a work of perfection. I’m sure there are a few minor mistakes or typos here and there, but the install does work as a whole and any MAJOR problems you encounter are typically going to be due to something you’ve done wrong or something you’re system isn’t doing right. It is completely normal for most everyone to experience some problems during the install. That’s life. Get used to it. While I make all efforts to make this guide easy and error free, the ease of the installation process really depends on the skill of the individual person attempting the install. As I said, most everyone experiences a problem or 2 along the way. If you are a newbie, these problems can be very difficult to overcome without help. If you are a seasoned sysadmin, you should be able to use common sense to fix most problems as they come up. It is for this reason that experienced Linux/Unix people find this guide to be excellent while newbies have a more hit or miss experience. Some love it, some get frustrated. So goes the life of the newbie. But the frustration that newbies feel at time arises out of that person’s own inabilities and NOT out of some huge flaw in the guide. I’ve used this installtion guide myself countless times verbatim, and it DOES work. To ensure a positive experience, make sure your system is properly prepared and make sure you have the appropriate skill set.

What NOT to expect from this installation guide

While the Logstash guide does work, you SHOULD NOT expect it to be a seamless and flawless experience. System environments and server setups vary, so portions of the installation guide may be more difficult for some users.

This guide is also NOT absolutely perfect. As I said, it DOES work for most people, but I am always finding areas for improvement, clarification and correction. If you ever find an area of the install process that you feel needs improvement, clarification or correction, by all means let me know.

A successful Logstash installation requires certain packages be installed and certain configurations be present on your server. I’ve put together this page to provide a general checklist for visitors to use before they begin the installation. Keep in mind that, since setups will vary from server to server, you may find some requirements that are not listed here.

How much disk space should I have available on my server?

The following is the partition schema maintained by us for the server.

/ partition: about 20GB.

/home partition: About 150GB

Some information on logstash and why do we prefer it on our production environment .

If anything goes wrong in our system then we prefer log files to troubleshoot the problems and clues .
Logstash, with build in analysis tool is a log server , consolidates logs from many servers and evn makes the data searchabe.

Strictly speaking, Logstash alone does not ensure meaningful and centralized management of logfiles. To perform its tasks as promised, Logstash needs some assistance. Logstash itself is a Java application, and despite all the prejudices that administrators have against Java – justified or not – the Logstash developers’ decision to go with Java was well founded. Because Java is also installed on Windows as a matter of course, Logstash can include Windows logfiles in its collection; this would be difficult to achieve with other Rsyslog services in many cases.
A Logstash installation that extends over more than one server consists of at least five different services. The central role is played by Logstash’s own components: The shipper – basically a client running on each target system – collects log messages. In the next step, it sends them to theindexer, which interprets and processes the log messages as specified by the admin. The host on which the indexer is found generally also runs the Logstash web server, which offers admins a search box for logfiles. In the background, two other services that do not directly belong to Logstash but are important for its function go about their duties: the Redis message broker and the ElasticSearch storage and search environment.
Redis is the focal point for communication between the shipper and the indexer. The Logstash instances on each server deliver their messages to the Redis server, where the Logstash indexer retrieves them in the next step. ElasticSearch, also a Java application, builds the index in the background and provides the interface to which the Logstash web server forwards search requests from the web interface.

Requirements of Logstash Server

1. Redis server : It is focal point for communication between the shipper and the indexer.
2.Elasticsearch : Stores all of the logs
3. Java : To run logstash server and client new version of java is required
5. Apache : Kibana needs apache to run
6.Kibana Web Interface : Web interface for searching and visualizing logs
7.Logstash: This component of logstash server is used to process incoming server logs to elasticsearch and shipping logs form client to server.It plays the part of both server and client
based on the configuration.Both client and server need java for running this.

First lets prepare our system to able to compile source file.

logstash@hari:~$ sudo apt-get install build-essential
logstash@hari:~$ apt-get install tcl8.5

REDIS SERVER INSTALLATION

Please install newer version of redis .You can get the redis from the link
http://redis.io/download .Here I have used redis-2.6.14.tar.gz .

logstash@hari:~/downloads$ wget http://download.redis.io/releases/redis-2.6.14.tar.gz
logstash@hari:~/downloads$ tar -zxvf redis-2.6.14.tar.gz
logstash@hari:~/downloads$ mv redis-2.6.14 ~/
logstash@hari:~$ ln -s redis-2.6.14 redis
logstash@hari:~$ mkdir tmp/
logstash@w..eb02:~$ nohup /home/logstash/redis/src/redis-server ~/redis/redis.conf > ~/tmp/redis.out 2>&1 &
logstash@hari:~$ redis/src/redis-cli -h localhost
redis localhost:6379> ping
PONG

Redis is working fine.We can tune redis.conf file as we want.

ELASTICSEARCH INSTALLTION

logstash@hari:~/downloads$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz
logstash@hari:~/downloads$ tar -zxvf elasticsearch-1.3.2.tar.gz
logstash@hari:~/downloads$ mv elasticsearch-1.3.2 ~/
logstash@hari:~$ ln -s elasticsearch-1.3.2 elatsicsearch
logstash@hari:~$ nohup /home/logstash/elasticsearch/bin/elasticsearch -f > ~/tmp/elasticsearch.out 2>&1 &
logstash@hari:~$ ps -ef | grep elast
logstash 2273 1988 50 22:28 pts/0 00:00:05 /usr/local/java/jdk1.7.0_45/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/home/logstash/elasticsearch -cp :/home/logstash/elasticsearch/lib/elasticsearch-1.1.1.jar:/home/logstash/elasticsearch/lib/*:/home/logstash/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
logstash 2293 1988 0 22:28 pts/0 00:00:00 grep elast

It s good practice to tune elasticsearch.yml file .Its avoid unnecessary logs file coming to our logstash server. By deafult elastic search accept any logs if it found within the network matching its cluster name.So name clustername to your own is good practice.Do not keep it as default.

logstash@hari:~/elasticsearch/config$ vi elasticsearch.yml
cluster.name: elasticsearch_hari

JAVA INSTALLATION

root@hari:/usr/local/java# wget http://uni-smr.ac.ru/archive/dev/java/SDKs/sun/j2se/7/jdk-7u45-linux-i586.tar.gz
logstash@hari:~$ vi .bash_profile
export PATH=”/usr/local/java/jdk1.7.0_45/bin:$PATH”
logstash@hari:~$ java -version
java version “1.7.0_45”
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Client VM (build 24.45-b08, mixed mode)

APACHE INSTALLATION

logstash@hari:~/download$ wget http://www.apache.org/dist/httpd/httpd-2.2.27.tar.bz2
logstash@hari:~/download$ tar -jxvf httpd-2.2.27.tar.bz2
logstash@hari:~# sudo apt-get build-essential
logstash@hari:~# sudo apt-get build-dep apache2
logstash@hari:~# sudo apt-get install openssl libssl-dev
logstash@hari:~/download$ cd httpd-2.2.27
logstash@hari:~/download/httpd-2.2.27$ vi ins
“./configure” \
“–prefix=/home/logstash/apache” \
“–enable-so” \
“–enable-cgi” \
“–enable-info” \
“–enable-rewrite” \
“–enable-speling” \
“–enable-usertrack” \
“–enable-deflate” \
“–enable-ssl” \
“–enable-mime-magic” \
“–with-included-apr” \
“$@”
logstash@hari:~/download/httpd-2.2.27$ chmod +x ins
logstash@hari:~/download/httpd-2.2.27$ sh ins
logstash@hari:~/download/httpd-2.2.27$ make

Lets set port for the apache and check starting the apache server.

logstash@hari:~/apache/conf$ vi httpd.conf
Listen 8888
logstash@hari:~/apache/bin$ ./apachectl start
logstash@hari:~/apache/bin$ ps -ef | grep http
logstash 2555 1 0 22:56 ? 00:00:00 /home/logstash/apache/bin/httpd -k start
logstash 2556 2555 0 22:56 ? 00:00:00 /home/logstash/apache/bin/httpd -k start
logstash 2557 2555 0 22:56 ? 00:00:00 /home/logstash/apache/bin/httpd -k start
logstash 2558 2555 0 22:56 ? 00:00:00 /home/logstash/apache/bin/httpd -k start

KIBANA INSTALLATION

logstash@hari:~/downloads$ wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gz
logstash@hari:~/downloads$ tar -zxvf kibana-3.0.1.tar.gz
logstash@hari:~/downloads$ mv kibana-3.0.1 ~/apache/htdocs/kibana

Now lets open kibana UI

http://192.168.153.129:8888/kibana/

Kibana Normal UI looks this like

LOGSTASH INSTALLATION

logstash@hari:~/downloads$ wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
logstash@hari:~/downloads$ mv logstash-1.4.2 ~/
logstash@hari:~$ ln -s logstash-1.4.2 logstash
logstash@hari:~$ mkdir conf

Lets put all the server configuration file we make in logstash@hari:~/conf$

Lets ships logs of local machine /var/log to logsatsh .For that the configuration is below.
We can give the name of the file anything as we like .Here I have given indexer.conf

logstash@hari:~/conf$ vi indexer.conf
input {
file {
path => “/var/log/*”
type => “syslog” # a type to identify those logs
start_position => “end”
}
}
filter {
}
output {
stdout { }
elasticsearch {
cluster => “elasticsearch_hari”
}
}

Lets get some more examples of conf file at logstash server side.

Here input of logs are coming from client having different key name
to redis server and then output is going to elasticsearch.Kibana
is displaying the elasticsearch data on UI for searching and graphing.
File name of conf I have given is logstashweblogs.conf

input {
redis {
host => “192.168.153.129”
threads => 4
data_type => “list”
key => “haproxy”
}
redis {
host => “192.168.153.129”
data_type => “list”
key => “apache”
}
}
output {
stdout { codec => rubeydebug } # This line is needed to debug the logstash server.To know that logs are coming or not .We can know this by checking the .out file of logstash.
elasticsearch {
cluster => “elasticsearch_hari”
host => “127.0.0.1”
}
}

Now lets run the logstash server with indexer.conf configuration file which is given above

root@hari:~# chmod 644 /var/log/
logstash@hari:~/conf$ nohup ~/logstash/bin/logstash -f ~/conf/indexer.conf > ~/tmp/indexer.out 2>&1 &
logstash@hari:~/tmp$ tail -f indexer.out
2014-09-16T06:59:05.578+0000 0.0.0.0 Sep 15 23:59:05 hari su[3132]: Successful su for root by logstash
2014-09-16T06:59:05.583+0000 0.0.0.0 Sep 15 23:59:05 hari su[3132]: + /dev/pts/0 logstash:root
2014-09-16T06:59:05.585+0000 0.0.0.0 Sep 15 23:59:05 hari su[3132]: pam_unix(su:session): session opened for user root by root(uid=1040)

LOGSTASH AT CLIENT SIDE

REQUIREMENTS :
1.JAVA : Needed by the logstash client.
2.Logstash : Here logstash is used as client as we configure in conf file.

Refer java installation given above..

Logstash client installion.

logstash@hari:~/downloads$ wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
logstash@hari:~/downloads$ mv logstash-1.4.2 ~/
logstash@hari:~$ ln -s logstash-1.4.2 logstash
logstash@hari:~$ mkdir conf

Lets create conf file at client side , an example.
Name given to the file is shipper.conf

logstash@hari:~/conf$ vi shipper.conf

input {
file {
path => “/var/log/*”
type => “syslog” # a type to identify those logs
}
}

output {
stdout { codec => rubydebug }
redis { host => “192.168.153.129” data_type => “list” key => “logstash” }
}

REFRENCES:

http://logstash.net/docs/1.0.17/installation
https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04
https://blog.basefarm.com/blog/how-to-install-logstash-with-kibana-interface-on-rhel/
http://www.slashroot.in/logstash-tutorial-linux-central-logging-server

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s