{"id":52,"date":"2016-02-04T19:26:59","date_gmt":"2016-02-04T19:26:59","guid":{"rendered":"http:\/\/redmonk.com\/fryan\/?p=52"},"modified":"2016-02-04T19:26:59","modified_gmt":"2016-02-04T19:26:59","slug":"the-rise-and-rise-of-apache-kafka","status":"publish","type":"post","link":"https:\/\/redmonk.com\/fryan\/2016\/02\/04\/the-rise-and-rise-of-apache-kafka\/","title":{"rendered":"The Rise and Rise of Apache Kafka"},"content":{"rendered":"<p>One of the key technologies in the new data stack is <a href=\"http:\/\/kafka.apache.org\">Apache Kafka<\/a>, and over the last eighteen months we have been tracking a huge uptick in developer interest in, chatter around, and usage of, Kafka. If you have not heard of Kafka it is a highly scalable distributed publish-subscribe messaging system, which happens to be very well suited to use cases such as streaming event data.<\/p>\n<p>With new workloads in areas such as IoT, mobile and gaming generating massive, and ever increasing, streams of data, developers have been looking for a mechanism to easily consume the data in a consistent and coherent manner. Which is exactly where Kafka fits in. \u00a0This has lead to a number of commercial offerings and product combinations appearing over the last year from vendors such as <a href=\"http:\/\/confluent.io\">Confluent<\/a>, <a href=\"https:\/\/developer.ibm.com\/messaging\/message-hub\/\">IBM<\/a> and <a href=\"http:\/\/www.cloudera.com\/documentation\/kafka\/latest\/topics\/kafka.html\">Cloudera<\/a> among others.<\/p>\n<p>As my colleague Stephen O\u2019Grady has <a href=\"http:\/\/www.computerworld.com\/article\/2999864\/big-data\/how-apache-kafka-is-greasing-the-wheels-for-big-data.html\">stated recently<\/a><\/p>\n<blockquote><p>It&#8217;s [kafka] becoming more visible because it&#8217;s a high-quality open-source project, but also because its ability to handle high-velocity streams of information is increasingly in demand for usage in servicing workloads like IoT, among others&#8221;<\/p><\/blockquote>\n<p>I mentioned earlier we have seen a big uptick in developer interest over the last eighteen months. Looking at multiple data sources we can see some clear trends around this uptick.<\/p>\n<p>On Stack Overflow we saw a marked increase in questions beginning in mid 2014:<\/p>\n<p><a href=\"http:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-questions.jpg\" rel=\"attachment wp-att-54\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-54\" src=\"http:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-questions-300x198.jpg\" alt=\"apache-kafka-questions\" width=\"300\" height=\"198\" srcset=\"https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-questions-300x198.jpg 300w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-questions-768x508.jpg 768w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-questions-1024x677.jpg 1024w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-questions.jpg 1134w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>This trend on Stack Overflow\u00a0matches into the growth in stars for the Kafka project on GitHub.<\/p>\n<p><a href=\"http:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-github-stars.jpg\" rel=\"attachment wp-att-53\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-53\" src=\"http:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-github-stars-300x198.jpg\" alt=\"apache-kafka-github-stars\" width=\"300\" height=\"198\" srcset=\"https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-github-stars-300x198.jpg 300w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-github-stars-768x508.jpg 768w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-github-stars-1024x677.jpg 1024w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-github-stars.jpg 1134w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Google trends demonstrates the same growth, with an uptick starting in mind 2014.<\/p>\n<div align=\"center\"><script src=\"\/\/www.google.com\/trends\/embed.js?hl=en-US&amp;q=Apache+Kafka&amp;tz=Etc\/GMT&amp;content=1&amp;cid=TIMESERIES_GRAPH_0&amp;export=5&amp;w=500&amp;h=330\" type=\"text\/javascript\"><\/script><\/div>\n<p>&nbsp;<\/p>\n<p>The most interesting data point, however, has been the consistent growth of active users on the kafka users mailing list, which is just over 260% since July 2014.<\/p>\n<p><a href=\"http:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-maillist-activity.jpg\" rel=\"attachment wp-att-55\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-55\" src=\"http:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-maillist-activity-300x198.jpg\" alt=\"apache-kafka-maillist-activity\" width=\"400\" height=\"265\" srcset=\"https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-maillist-activity-300x198.jpg 300w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-maillist-activity-768x508.jpg 768w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-maillist-activity-1024x677.jpg 1024w, https:\/\/redmonk.com\/fryan\/files\/2016\/02\/apache-kafka-maillist-activity.jpg 1134w\" sizes=\"auto, (max-width: 400px) 100vw, 400px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>We define an active user as someone who has sent an e-mail to the kafka users list. The majority of these questions tend to come from a new users of Kafka. While the volumes of mails remain relatively constant this continuous growth in active, rather than passive, users, is indicative of a strong and vibrant community.<\/p>\n<p>Disclosure: IBM and Cloudera are current RedMonk clients.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the key technologies in the new data stack is Apache Kafka, and over the last eighteen months we have been tracking a huge uptick in developer interest in, chatter around, and usage of, Kafka. If you have not heard of Kafka it is a highly scalable distributed publish-subscribe messaging system, which happens to<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,7,5],"tags":[],"class_list":["post-52","post","type-post","status-publish","format-standard","hentry","category-apache","category-data","category-internet-of-things"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/posts\/52","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/comments?post=52"}],"version-history":[{"count":0,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/posts\/52\/revisions"}],"wp:attachment":[{"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/media?parent=52"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/categories?post=52"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redmonk.com\/fryan\/wp-json\/wp\/v2\/tags?post=52"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}