Installing ElasticSearch Logstash & Kibana

Installing ElasticSearch Logstash & Kibana #EKL #Logstash-forwader #COMBINEDAPACHELOG #AmazonEC2

It’s been a year since I last updated the blog, laziness wins any day! :P. This blog entry will illustrate how to setup an out of the box installation for EKL.This setup was done on Amazon EC2 instances, this will cover the following topics:

     a) Setting up ElasticSearch 

     b) Setting up Logstash Server

     c) Setting up Logstash-Forwader

     d) Setting up Kibana
        [Logstash 1.4.2 and Kibana 3 ElasticSearch 1.3]

Below is the pictorial setup which I have up and running.The Final aim would be to send Apache access logs from Server [EC2_A] to Server [EC2_B] and create a Elastic cluster named (elasticsearch) and show the graphical representation in Kibana

Here the EC2_A Server is our Logstash_forwader/Shipper. On our EC2_B we have the Elastic cluster and the logstash Server which is running, and the UI is shown in Kibana.

There are many online resources, which we can refer to get the above setup. However, they are not at one single place, I had to search at multiple places to get the above setup running.

One of the best resource I came across is the Logstash Cook Book  and the EKL installation guide by Digital Ocean.

Certainly, there were lot of initial breakers I faced, but with this blog entry I suppose one should not face any problems while installing.

For starters, who are not familiar with EKL can read about these components here. ElasticSearch(http://en.wikipedia.org/wiki/Elasticsearch), Logstash(http://logstash.net/), Kibana(http://rashidkpc.github.io/Kibana/).

Let’s Start!

We will try setting up the EC2_B box first.

EC2_B Config: (Micro instance) Ubuntu Server 14.04_32bit: Linux ip-192.168.2.2 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:02:19 UTC 2014 i686 i686 i686 GNU/Linux

Installing Dependencies:

(The only prerequisite required by Logstash is Java runtime)

$ sudo add-apt-repository -y ppa:webupd8team/java

$ sudo apt-get update

$ sudo apt-get -y install oracle-java7-installer

Now try:
$ java -version

java version "1.7.0_45"

Java(TM) SE Runtime Environment (build 1.7.0_45-b18)

Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

Install Elastic Search:

$ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.tar.gz

$ tar zxvf elasticsearch-1.1.1.tar.gz

$ cd elasticsearch-1.1.1/

$ ./bin/elasticsearch &    --This will start Elastic Search

Install Kibana:

$ cd ~; wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gz

$ tar xvf kibana-3.0.1.tar.gz

$ sudo vi ~/kibana-3.0.1/config.js --Now change the text from 9200 to 80

  elasticsearch: "http://"+window.location.hostname+":80”,

$ sudo mkdir -p /var/www/kibana3

$ sudo cp -R ~/kibana-3.0.1/* /var/www/kibana3/

Install nginx to host Kibana:

$ sudo apt-get install nginx

$ cd ~; wget https://github.com/elasticsearch/kibana/raw/master/sample/nginx.conf

$ vi nginx.conf   --Now change the value of root as below

  root to /var/www/kibana3;

$ sudo service nginx restart

Now go to http://[IP]/kibana3 to check if Kibana UI is visible.

Install Logstash:

$ curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

$ tar zxvf logstash-1.4.2.tar.gz

Now Generate the SSL Certificate:

$ sudo mkdir -p /etc/pki/tls/certs

$ sudo mkdir /etc/pki/tls/private

Now we will edit the openssl.cnf file so that later on we won’t face any issues when we compile our logstash-forwader using go1.3 linux/amd64 on EC2_A (More details here)

$ Vi /etc/ssl/openssl.cnf

In the [v3_ca] section add the following entry

subjectAltName = IP:192.168.2.2

Note: Here the IP address has to be of the EC2_B. machine.

Now lets create a index on our Elastic cluster:

Lets first install a plugin named "head"

$ cd ~/elasticsearch-1.1.1/
$ bin/plugin --install mobz/elasticsearch-head

Now go to http://IP(EC2_B):9200/_plugin/head/

Go to indices tab and create a new index called "apache"

Now Generate the Self signed certs:

$ cd /etc/pki/tls; sudo openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

The same certificate "logstash-forwarder.crt” has to be imported to logstash_forwader server (EC2_A). Please do this using appropriate “scp” commands.

Configure Logstash:
$ nano ~/logstash-1.4.2/logstash.conf

input {

  lumberjack {

    port => 5000

    type => "apache-access"

    ssl_certificate => "/etc/pki/tls/certs/logstash- forwarder.crt"

    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" 

}

}

filter {

  grok {

    type => "apache-access"

    pattern => "%{COMBINEDAPACHELOG}"

  }

}

output { 

  elasticsearch {

 host => localhost

 protocol => http

 index => “apache"

 cluster => "elasticsearch"

 index_type => "apache"

     } }

This creates a configuration file which will make the log stash listen on port 5000 (lumberjack) and accept incoming logs from the logstash forwarder. Also, the grok filter here I have specified as %{COMBINEDAPACHELOG} since we will be sending the apache access logs from the EC2_A server.

Now setting up our EC2_A server (Logstash_Forwader/Shipper):

EC2_A: (Micro instance) Ubuntu Server 14.04_64bit: Linux ip-192.168.2.1 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Make sure your Apace Server in running on this machine and java is also installed, or please refer to the first step.This machine will be used as a shipper to send apache logs to EC2_B.

$ wget https://github.com/elasticsearch/logstash-forwarder/archive/master.zip

$ unzip logstash-forwarder-master.zip

$ cd logstash-forwarder-master

Installing the developer tools:

$ sudo apt-get install build-essential

Installing Go:

$ sudo apt-get install python-software-properties

$ sudo apt-add-repository ppa:duh/golang

$ sudo apt-get update

$ sudo apt-get install golang

$ sudo apt-get install ruby rubygems ruby-dev

$ sudo gem install fpm

Creating the forwarder deb package.

$ umask 022

$ make deb

You'll see a long sequence of compilation and then some final execution as the fpm command runs and creates the DEB package.

Listing 1.34: Forwarder make output

Installing the forwarder:

$ sudo dpkg -i logstash-forwarder_0.2.0_i386.deb

Now create a folder to place the "logstash-forwarder.crt" certificate. Before that we need to import the "logstash-forwarder.crt" cert file that we created on the EC2_B server. Please do it via necessary scp commands.

$ mkdir /etc/certs

Place the "logstash-forwarder.crt" file in the /certs folder.Also, create the logstash-conf file:

$ nano /etc/logstash-forwarder/logstash-forwarder.conf

change Below IP to the IP of your Logstash Server

{

"network":

 {

"servers": [ “IP[EC2_B]:5000" ],

"ssl ca": "/etc/certs/logstash-forwarder.crt",

"timeout": 15

 },

"files": [

    {

"paths": ["/var/log/apache2/access.log"],

"fields": { "type": "apache-access" }

     }

        ]

}

Now Start the forwarder:

$ cd /opt/logstash-forwarder
$ bin/logstash-forwarder -config="/etc/logstash-forwarder/logstash-forwarder.conf" &

Finally Starting the Logstash Server on (EC2_B):
$ cd ~/logstash-1.4.2/
$ bin/logstash -f logstash.conf & --This will start the logstash server 

Any further changes to the access logs will be now visible in your kibana dashboard. To check the above setup, hit the default apache page @ (http://IP[EC2_A]/) and check the changes recorded by your elastic cluster on the kibana dashboard.

The dashboard I use is my personal favourite, which can be found here.
I hope this blog entry will be useful for a successful EKL installation. Do write a comment below if you get stuck anywhere.

Hacking Tutorials

Learning to become hacker is not as easy as learning to become a software developer. I realized this when I started looking for learning resources for simple hacking people do. Even to start doing the simplest hack on own, a hacker requires to have in depth knowledge of multiple topics. Some people recommend minimum knowledge of few programming languages like C, Python, HTML with Unix operating system concepts and networking knowledge is required to start learning hacking techniques.
Though knowing a lot of things is required, it is not really enough for you to be a competent and successful hacker. You must have a passion and positive attitude towards problem solving. The security softwares are constantly evolving and therefore you must keep learning new things with a really fast pace.

If you are thinking about ethical hacking as a career option, you may need to be prepared for a lot of hard/smart work. I hope these free resources will help you speed up on your learning. If you decide you pursue ethical hacking as a career option, you may also want to read some in depth ethical hacking books.
A lot of people (including me before doing research for this article) think that they can become a hacker using some free hacking tools available on web. Its true that some common types of hacking can be easily done with help of tools, however doing it does not really make you a hacker. A true hacker is the one who can find a vulnerability and develop a tool to exploit and/or demonstrate it.
Hacking is not only about knowing “how things work”, but its about knowing “why things work that way” and “how can we challenge it”.
Below are some really useful hacking tutorials and resources you may want to explore in your journey of learning to hack.

Learning to become hacker is not as easy as learning to become a software developer. I realized this when I started looking for learning resources for simple hacking people do. Even to start doing the simplest hack on own, a hacker requires to have in depth knowledge of multiple topics. Some people recommend minimum knowledge of few programming languages like C, Python, HTML with Unix operating system concepts and networking knowledge is required to start learning hacking techniques.
Though knowing a lot of things is required, it is not really enough for you to be a competent and successful hacker. You must have a passion and positive attitude towards problem solving. The security softwares are constantly evolving and therefore you must keep learning new things with a really fast pace.
If you are thinking about ethical hacking as a career option, you may need to be prepared for a lot of hard/smart work. I hope these free resources will help you speed up on your learning. If you decide you pursue ethical hacking as a career option, you may also want to read some in depth ethical hacking books.
A lot of people (including me before doing research for this article) think that they can become a hacker using some free hacking tools available on web. Its true that some common types of hacking can be easily done with help of tools, however doing it does not really make you a hacker. A true hacker is the one who can find a vulnerability and develop a tool to exploit and/or demonstrate it.
Hacking is not only about knowing “how things work”, but its about knowing “why things work that way” and “how can we challenge it”.
Below are some really useful hacking tutorials and resources you may want to explore in your journey of learning to hack

Hacking For Dummies – Beginners Tutorials

These tutorials are not really simple for anyone who is just starting to learn hacking techniques. However, these should be simple starting point for you. I am sure you have different opinion about complexity of each tutorial however advanced hacker are going to be calling this a job of script kiddie (beginner hacker). Even to acquire the skills of a script kiddie you need to have good understanding of computer fundamentals and programming.

1. Cybrary – For those looking to learn ethical hacking skills online, Cybrary provides the perfect platform to do so. Cybrary is a free online IT and cyber security training network that provides instruction in the form of self-paced, easy-to-follow videos. Featuring courses on topics such as Penetration Testing and Ethical Hacking, Advanced Penetration Testing, Post Exploitation Hacking and Computer and Hacking Forensics, Cybrary provides instruction from the beginner to the highly-advanced level of hacking. Additionally, Cybrary offers supplemental study material along with their courses free of charge. With their in-depth training videos and study guides, Cybrary ensures that users develop the best hacking skills.
2. Hacking Tutorials for Beginners – By BreakTheSecurity.com
3. How to learn Ethical hacking – By Astalavista.com
4. Penetration Testing Tutorial – By Guru99.com
5. Backtrack Penetration Testing Tutorial
6. Introduction to Penetration Testing
7. Information Gathering with Nmap
8. Simple How To Articles By Open Web Application Security
9. The Six Dumbest Ideas in Computer Security
10. Secure Design Principles
11. 10 steps to secure software

Cryptography Related Tutorials

Cryptography is must know topic for any aspiring security professional or a ethical hacker. You must understand how encryption and decryption is done. You must understand why some of the old encryption techniques do not work in modern computing world.
This is a important area and a lot of software programmers and professional do not understand it very well. Learning cryptography involves a lot of good understanding of mathematics, this means you also need to have good fundamentals on discrete mathematics.

1. Introduction to Public Key Cryptography
2. Crypto Tutorial
3. Introduction to Cryptography
4. An Overview of Cryptography
5. Cryptography Tutorials – Herong’s Tutorial Examples
6. The Crypto Tutorial – Learn How to Keep Secret Secret
7. Introduction to cryptology, Part 1: Basic Cryptology Concepts

Websites For Security Related Articles And News

These are some websites, that you may find useful to find hacking related resources and articles. A lot of simple tricks and tips are available for experimenting through these sites for improving yourself to become advanced hacker.
In recent years, many people are aspiring to learn how to hack. With growing interest in this area, a lot of different types of hacking practices are evolving. With popularity of social networks many people have inclined towards vulnerability in various social networks like facebook, twitter, and myspace etc.
Continuous learning about latest security issues, news and vulnerability reports are really important for any hacker or a security professional. Some of the sites that keep publishing informative articles and news are listed here.

1. http://www.astalavista.com/
2. http://packetstormsecurity.com/
3. http://www.blackhat.com/
4. http://www.metasploit.com/
5. http://sectools.org/
6. http://www.2600.com/
7. DEF CON – Hacking conference
8. http://www.breakthesecurity.com/
9. http://www.hacking-tutorial.com/
10. http://www.evilzone.org/
11. http://hackaday.com/
12. http://www.hitb.org/
13. http://www.hackthissite.org/
14. http://pentestmag.com
15. http://www.securitytube.net/
16. https://www.ssllabs.com/

EBooks And Whitepapers

Some of the research papers by security experts and gurus can provide you a lot of information and inspiration. White papers can be really difficult to read and understand therefore you may need to read them multiple times. Once you understand the topic well, reading will become much faster and you will be able to skim through a lot content in less time.

1. Handbook of Applied Cryptography – This ebook contains some free chapter from one of the popular cryptography books. The full book is also available on amazon at Cryptography Book.
2. Network Penetration testing Guide
3. How to hack anything in Java
4. Mcafee on iPhone and iPad Security
5. A Good Collection of White papers on security and vulnerabilities – This site contains collection of white papers from different sources and some of these white papers are really worth referring.
6. Engineering Principles for Information Technology Security
7. Basic Principles Of Information Protection
8. Open Web Application Security Project – OWASP is one of the most popular sites that contains web application security related information .

Videos & Play Lists

Those who like to watch video tutorials, here are few I liked. However there are many small video available on youtube. Feel free to explore more and share with us if you like something.

1. Cryptography Course By Dan Boneh Stanford University
2. Open Security Training– Youtube Playlist of More than 90 hours. I have found this to be the biggest free training available for security related topic.
3. OWASP AppSec USA 2011: Youtube Playlist containing compilation of OWASP conference highlight in 2011.
4. Defcon: How I Met your Girlfriend – Defcon is one of the most popular hacker conference. The presenters in this conference are well know inside the hacking industry.
5. Defcon: What happens when you steal a hackers computer
6. Defcon: Nmap: Scanning the Internet
7. Public Key Cryptography: Diffie-Hellman Key Exchange
8. Web application Pen testing
9. Intro to Scanning Nmap, Hping, Amap, TCPDump, Metasploit

Forums For Hackers And Security Professionals

Just like any other area, forums are really great help for learning from other experts. Hundreds of security experts and ethical/non-ethical hackers are willing to share their knowledge on forums for some reason. Please keep in mind to do enough research before post a question and be polite to people who take time to answer your question.

1. Stackoverflow for security professionals
2. http://darksat.x47.net/
3. http://forums.securityinfowatch.com/
4. http://forums.cnet.com/spyware-viruses-security-forum/
5. http://www.hackforums.net/forumdisplay.php?fid=47

Vulnerability Databases And Resources

Vulnerability Databases are the first place to start your day as a security professional. Any new vulnerability detection is generally available through the public vulnerability databases. These databases are a big source of information for hackers to be able to understand and exploit/avoid/fix the vulnerability.

1. http://www.exploit-db.com/
2. http://1337day.com/
3. http://securityvulns.com/
4. http://www.securityfocus.com/
5. http://www.osvdb.org/
6. http://www.securiteam.com/
7. http://secunia.com/advisories/
8. http://insecure.org/sploits_all.html
9. http://zerodayinitiative.com/advisories/published/
10. http://nmrc.org/pub/index.html
11. http://web.nvd.nist.gov
12. http://www.vupen.com/english/security-advisories/
13. http://www.vupen.com/blog/
14. http://cvedetails.com/
15. http://www.rapid7.com/vulndb/index.jsp
16. http://oval.mitre.org/

Product Specific Vulnerability Information

Some of the very popular products in the world require a special attention and therefore you may want to look at the specific security websites directly from vendors. I have kept Linux. Microsoft and apache in this list, however it may apply to any product you may be heavily using.

1. Red Hat Security and other updates Site
2. Microsoft Products Security Bulletin
3. Apache Foundation Products Security Repository
4. Ubuntu Software Security Center
5. Linux Security Repository

Tools And Programs For Hacking / Security

There are dozens of tools available for doing different types of hacking and tests. Tools are really important to become more productive at your work. Some of the very common tools that are used by hackers are listed here. You may have different choice of tools based on your own comfort.

1. nmap
2. NSS
3. Hping
4. TCPDump
5. Metasploit
6. Wireshark
7. Network Stuff
8. Nikto

Summary

I have tried to compile some of these resources for my own reference for the journey of learning I am going to start. I am not even at a beginner level of becoming hacker but the knowledge of this field really fascinates me and keeps me motivated for learning more and more. I hope will be able to become successful in this.
A lot of people use their knowledge skills for breaking stuff and stealing. I personally think that doing harm to someone is a weak choice and will not have a good ending. I would recommend not to use your skills for any un-ethical endeavor. A single misuse of your skill may jeopardize your career since most companies do a strict third party background check before they hire a ethical hacker or a security personal.
There are dozens of companies looking for ethical hackers and security professionals. There are really good number of opportunities in this area and its really niche compensation segment. You will be easily able to get a decent job without even acquiring all the expert level skills to become a pro hacker.

network security website

http://networkcomputing.com/net-security/information-security-training-more-critical-ever-2016/1912142620


http://www.informationweek.com/


http://www.darkreading.com/

 

How to Create Virtual Machines in Linux Using KVM (Kernel-based Virtual Machine)

http://www.tecmint.com/install-and-configure-kvm-in-linux/

geekboy blog for security

http://www.geekboy.co/

CEH V9 PDF- Certified Ethical Hacker v9 Full Course With Tools

http://www.geekboy.co/hacking/ceh-v9-pdf-certified-ethical-hacker-v9-full-course-with-tools/

https://www.dropbox.com/s/kwzesacuje5nu0s/Certified%20Ethical%20Hacker%20%28CEH%29%20v.9%20Courseware%20by%20Gkillah.zip?dl=0


https://drive.google.com/folderview?id=0B5RHHIY8-fgGYVNObU0wdl9sS0k&usp=drive_web

Offensive Security: OSCP – Penetration Testing With Kali – A Review

https://ctf.rip/offensive-security-oscp-penetration-testing-with-kali-a-review/