{"id":61,"date":"2016-02-19T15:25:29","date_gmt":"2016-02-19T15:25:29","guid":{"rendered":"http:\/\/redmonk.com\/fryan\/?p=61"},"modified":"2016-02-19T15:25:29","modified_gmt":"2016-02-19T15:25:29","slug":"devops-and-opensource-log-aggregation-tools-logstash-has-competition","status":"publish","type":"post","link":"https:\/\/redmonk.com\/fryan\/2016\/02\/19\/devops-and-opensource-log-aggregation-tools-logstash-has-competition\/","title":{"rendered":"DevOps and OpenSource Log Aggregation Tools \u2013 LogStash has Competition."},"content":{"rendered":"

 <\/p>\n

In the world of DevOps you are nothing without logs. As we move to more complex architectures developers can, and do, add in various types of advance telemetry and instrumentation \u2013 which of course opens up the debate as to what to log v\u2019s what to instrument, as noted by developers<\/a> such as Peter Bourgon.\u00a0 Here at RedMonk we are all fascinated by all things logging and telemetry related, recognising the value held in such data.<\/p>\n

For many organisations though, particularly those that are just starting on their DevOps journeys (which is far more companies than one might think), just getting the aggregation and analysis of logs in place is a great starting place. As the quote which is often attributed to statistician Karl Pearson says:<\/p>\n

\u201cThat which is measured improves. That which is measured and reported improves exponentially.\u201d<\/p><\/blockquote>\n

The starting point with logs in any reasonable scale system is aggregation. Among purely opensource solutions LogStash<\/a> has the undoubted lead, propelled by its close integration with Elastic<\/a> and Kibana<\/a> – the ubiquitous ELK stack.<\/p>\n

In our discussions around log aggregation we frequently\u00a0hear about two other solutions – fluentd<\/a>, which is used with Kubernetes, and Graylog2<\/a> \u2013 both of which garner far less attention than logstash on forums such as stackoverflow.<\/p>\n

\"allthelogs-stackoverflow\"<\/a><\/p>\n

However, when we look on github we can see a growth in interest In all three, with logstash and fluentd following reasonably similar trajectories.<\/p>\n

\"LogStash<\/a><\/p>\n

 <\/p>\n

Issues and Commits<\/h2>\n

Now github stars are an interesting proxy for interest, but when it comes to actual usage looking at the issues logged, and more importantly if they are coming from the community can be far more interesting data point. We can see that almost 70% of the issues reported against both LogStash and fluentd, while GrayLog2 is far closer to a 50\/50 split.<\/p>\n

\"logstash-graylog-fluentd-issues-barchart\"<\/a><\/p>\n

The trends are pretty consistent over the life of the projects as well.<\/p>\n

\"issues-logstash-c-v-c\"<\/a><\/p>\n

\"issues-fluentd-c-v-c\"<\/a>\"issues-graylog-c-v-c\"<\/a><\/p>\n

Similarly for commits, with both LogStash and fluentd the community accounts for approximately 70% of the activity, although for graylog direct company contributions account for almost all commits.<\/p>\n

\"logstash-graylog-fluentd-commits-barchart\"<\/a><\/p>\n

The trends over the lifetime of the project are also similar for commits<\/p>\n

\"commits-logstash-c-v-c\"<\/a><\/p>\n

\"commits-fluentd-c-v-c\"<\/a><\/p>\n

\"commits-graylog-c-v-c\"<\/a><\/p>\n

 <\/p>\n

By far the most interesting part of all this though is the rise in overall interest since early 2014. As people have come to realise just how much value is in in their log data we see more and more log aggregation solutions being deployed. The first part of a journey, but a very important part none the less.<\/p>\n

There are some caveats with this research to note:<\/p>\n

    \n
  1. We have endeavoured to match major committers to their companies, but that is not always possible.<\/li>\n
  2. Plugins are dealt with differently in each product, so we have limited the scope of our analysis to the main repo.<\/li>\n
  3. Issues can, and do, come from commercial support contracts. The actual issue logged will generally come from the main company contributing to the project.<\/li>\n<\/ol>\n

    Disclaimer<\/strong>: Treasure Data (major contributors to fluentd) are a current RedMonk client.<\/p>\n

    Found this interesting? Sign up for my newsletter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

      In the world of DevOps you are nothing without logs. As we move to more complex architectures developers can, and do, add in various types of advance telemetry and instrumentation \u2013 which of course opens up the debate as to what to log v\u2019s what to instrument, as noted by developers such as Peter<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,9,17],"tags":[],"class_list":["post-61","post","type-post","status-publish","format-standard","hentry","category-business","category-devops","category-infrastructure"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/posts\/61","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/comments?post=61"}],"version-history":[{"count":0,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/posts\/61\/revisions"}],"wp:attachment":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/media?parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/categories?post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/tags?post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}