ELK in the Wild - Empower your Data through Analytics

Blog Post created by whibr08 Employee on Apr 25, 2016

When first asked what I thought of ELK my initial response as an Idaho boy was, I love Elk. They are a beautiful wild animal, fun to observe, hunt, and provide a very tasty source of protein. But in the context of where this question was asked this was not the “ELK” I was being asked about.


This “ELK” is the name given to the suite of Elastic products which provides  real-time search capabilities that power many of today’s analytics. Companies collect a massive amount of “data” that can be very valuable and can be their competitive edge as long as they have an effective way to leverage this data. The three key steps that enable data to be as effective as possible are the ability to “Collect” the data, “Normalize” the data, and “Analyze” the data. The ELK product stack provides the necessary tools to support all three steps. Additionally, it’s open-source (free) and supported by a very diverse community, helping to reduce support costs.




Well that all sounds great but I can find all the above information details on Wikipedia. Before I’m convinced of the effectiveness of a technology I have to get my hands on it. Below you’ll find my experience with the ELK stack.


Step One - Birth

I started with a simple “Elasticsearch Download” web search. Elasticsearch (ES) is the core product of the ELK suite - - so it only made sense to begin my journey there. As a non-Windows user, I was pleased to find  a variety of supported platforms. I downloaded the appropriate installer which turned out to be a zip file. Once downloaded and extracted, I opened the readme which instructed me to make sure I had a supported version of java installed and then to move to the extracted /bin directory and run “elasticsearch”. I did this and the first thing that came to mind was “it couldn’t have been that simple”, but it was!


Step Two - Feed the ELK (Collect)

Traditionally my test data comes from my Computer Science Thesis work where I created a computer program for my father-in-law’s fish farm. This data includes everything that goes into raising a pond of fish (e.g. pounds of feed, type of feed, growth, mortality, etc.). I might be dating myself but it is a DOS-based Paradox 4.5 application (just after punch cards).


So how do you get Paradox data into Elasticsearch? There’s an ELK product for that, Logstash. Logstash was as simple to install as Elasticsearch and took my Paradox CSV output and fed it into Elasticsearch. I figured I would spend some time getting all my data properly into the NoSQL ES datastore, which it did. I became very familiar with Logstash config files, ES indexing, ES templates, analyzed and non-analyzed field mapping. I leveraged the community on this step and was finally able to get my data into ES with the correct, searchable, format. Let’s just say I became very familiar with the following curl commands:


curl -XDELETE 'http://localhost:9200/_all' #delete all indexes(data) in Elasticsearch

curl -XDELETE localhost:9200/_template/logstash #delete Elasticsearch template


Step Three - Find the ELK (Normalize)

I was now time for me to find my way through all the data I had just fed into Elasticsearch; enter Kibana. Again, the install was a simple download and run. I watched a short getting started with Kibana video created by a community member to educate myself on how to organize the data into meaningful Kibana visualizations.


Step Four - Observe the ELK (Analyze)

This is where my smile gets bigger as I see all this data coming together to paint a picture. Leveraging Kibana, I started creating dashboards that bring together various visualizations and allow me to play with (Analyze) the data.




Information like, which ponds/facilities produced the most fish gain, which feed type performed best, etc. All data that can then be leveraged by my father-in-law to invest where best for his business.


In conclusion, companies that can take advantage of the ELK stack to help them collect, normalize, and analyze their data, using it to drive business decisions, will have a competitive edge. Let’s just say I’m now a fan of both types of ELK.


Thanks for reading.


Bryan Whitmarsh