API Gateway + ELK

Blog Post created by Doyle_Reece Employee on Sep 21, 2016

In the Monitoring world, especially log based monitoring, there are a few common Products that I keep seeing in our Customer's Environments ( Splunk and ELK ). This particular Entry will focus on ELK, I'll touch on Splunk later.


ELK is a stack made up of ( ElasticSearch, Logstash, and Kibana ). 


ElasticSearch serves as the search engine in this case. It's a very quick/powerful tool for inserting,indexing,querying data from. Log stash is a popular tool used for manipulating log messages. It has grown very popular for it's ElasticSearch integration which allow it to convert Log Messages into a Document that can be inserted into ElasticSearch, which can be queried later. Last, we have Kibana, which is the frontend ( fancy graphs ), that sit on top of Elastic Search to populate graphs, based on on the data in ES ( ElasticSearch ).


In the diagram below, you'll see that Kibana sits on top of ElasticSearch to do all it's magic. You then see that the Gateway leverages two agents ( referred to as 'beats' in the Elastic world ) to ship data from the Gateway into either Logstash (which then sends to ES ) or ElasticSearch Directly. 

This beat concept is actually an open sourced platform that enables developers to get data into ElasticSearch or Logstash easily. 


<Borrowed terminology>

Beats is the platform for building lightweight, open source data shippers for many types of data you want to enrich with Logstash, search and analyze in Elasticsearch, and visualize in Kibana. Whether you’re interested in log files, infrastructure metrics, or network packets, Beats serves as the foundation for keeping a beat on your data.

</Borrowed terminology>


These guys are super simple to configure as well, in my case, i just pulled down the .rpm, installed it, modified the default config file to point to our Logstash/ES endpoint, and fired it up. It had a nice elegant feel to it ( once i figured out my typo after about an hour of troubleshooting   - ** FYI - YAML doesn't like tabs )


In our situation, we are looking at using 'Filebeat' and 'Metricbeat'.


Filebeat is a log forwarder and is replacing the previous 'Logstash-Forwarder' that was out in previous versions. This beat allows us to point to a file and say, forward these to this location or grab these files and inject them directly into ElasticSearch. There are several other features, that it has as well, but for simplicity purposes, i'll let you check those out, but at a high level, it's the agent used to get this log event from point A to point B.


Now, you may be asking, "Why use Logstash at all, if we can just forward our logs onto ElasticSearch Directly?". 


Well, 'EK' doesn't sound as cool as 'ELK', or in this case, 'MFEK'.


Outside of that, there is a real reason why we need Logstash in this specific scenario. When forwarding log files directly into ES, we don't have the ability to index our custom fields that exist within our log messages( using Filebeat ), so we lose a lot of valuable insight into those messages ( or in our case, Transactions ). What Logstash enables us to do, is when it receives a message(log), it parses that log and can index our custom fields before injecting it into ES, which allows us to query based on those custom fields.... It's a little complex, but makes sense if you think about it.


"Now how can Metricbeat talk directly to ES, but Filebeat can't?"


MetricBeat is a really cool agent(beat), that scrapes data from the OS and ships them to ES, to be queried later. It gives us system load, filesystem utilization, memory usage, Process information, and other goodies on the server. Its really cool how much info you get from this little guy. From personal experience, writing the script that scrapes this info can be kind of tough, so finding this guy was a real treat . Now since this data is static ( the fields and field types are static ), then the beat knows how to index this data for ES, so we could skip Logstash in this scenario, but if you wish to do more magic, then feel free to ship this data to LS as well, but in our case, we plan to ship this directly to ES.


So now, we have our mf-ELK stack.


Kibana is the GUI i referenced earlier. It can create nice looking dashboards like so. It uses a combination of the 'Lucene Query Language' and some cool 'Visualization Building' features to build out nice looking dashboards. This guy allows us to drag our 'panels/visualizations' around to really customize it exactly how we want. It's pretty nice and here recently, they 'FINALLY' introduced a 'Dark Theme', which is much nicer to the eyes in my opinion.