Charting Stacks

The Rise and Rise of Apache Kafka

Share via Twitter Share via Facebook Share via Linkedin Share via Reddit

One of the key technologies in the new data stack is Apache Kafka, and over the last eighteen months we have been tracking a huge uptick in developer interest in, chatter around, and usage of, Kafka. If you have not heard of Kafka it is a highly scalable distributed publish-subscribe messaging system, which happens to be very well suited to use cases such as streaming event data.

With new workloads in areas such as IoT, mobile and gaming generating massive, and ever increasing, streams of data, developers have been looking for a mechanism to easily consume the data in a consistent and coherent manner. Which is exactly where Kafka fits in.  This has lead to a number of commercial offerings and product combinations appearing over the last year from vendors such as Confluent, IBM and Cloudera among others.

As my colleague Stephen O’Grady has stated recently

It’s [kafka] becoming more visible because it’s a high-quality open-source project, but also because its ability to handle high-velocity streams of information is increasingly in demand for usage in servicing workloads like IoT, among others”

I mentioned earlier we have seen a big uptick in developer interest over the last eighteen months. Looking at multiple data sources we can see some clear trends around this uptick.

On Stack Overflow we saw a marked increase in questions beginning in mid 2014:

apache-kafka-questions

This trend on Stack Overflow matches into the growth in stars for the Kafka project on GitHub.

apache-kafka-github-stars

Google trends demonstrates the same growth, with an uptick starting in mind 2014.

 

The most interesting data point, however, has been the consistent growth of active users on the kafka users mailing list, which is just over 260% since July 2014.

apache-kafka-maillist-activity

 

We define an active user as someone who has sent an e-mail to the kafka users list. The majority of these questions tend to come from a new users of Kafka. While the volumes of mails remain relatively constant this continuous growth in active, rather than passive, users, is indicative of a strong and vibrant community.

Disclosure: IBM and Cloudera are current RedMonk clients.

6 comments

  1. […] on a roll. Last year it registered a 260 percent jump in developer popularity, as Redmonk’s Fintan Ryan highlights, a number that has only ballooned since then as IoT and other enterprise demands for real-time, […]

  2. […] pasado se registró un 260 por ciento de incremento en desarrollador de popularidad, como Redmonk Fintan Ryan destacados, un número que sólo se ha disparado desde entonces como IoT y otras demandas de la empresa en […]

  3. […] is on a roll. Last year it registered a 260 percent jump in developer popularity, as Redmonk’s Fintan Ryan highlights, a number that has only ballooned since then as IoT and other enterprise demands for real-time, […]

  4. […] is on a roll. Last year it registered a 260 percent jump in developer popularity, as Redmonk’s Fintan Ryan highlights , a number that has only ballooned since then as IoT and other enterprise demands for real-time, […]

  5. […] in demand for Apache Kafka courses on Udemy last year, making it the 4th hot tech skill for 2018. Apache Kafka-related questions on Google Search and tech forums like StackOverflow and Github have also skyrocketed in recent […]

  6. […] prioritizing enabling more real-time architecture and reducing the wait time on data availability. Apache Kafka-related questions on Google Search and tech forums like StackOverflow and Github have also skyrocketed in recent […]

Leave a Reply

Your email address will not be published. Required fields are marked *