One of the key technologies in the new data stack is Apache Kafka, and over the last eighteen months we have been tracking a huge uptick in developer interest in, chatter around, and usage of, Kafka. If you have not heard of Kafka it is a highly scalable distributed publish-subscribe messaging system, which happens to be very well suited to use cases such as streaming event data.
With new workloads in areas such as IoT, mobile and gaming generating massive, and ever increasing, streams of data, developers have been looking for a mechanism to easily consume the data in a consistent and coherent manner. Which is exactly where Kafka fits in. This has lead to a number of commercial offerings and product combinations appearing over the last year from vendors such as Confluent, IBM and Cloudera among others.
As my colleague Stephen O’Grady has stated recently
It’s [kafka] becoming more visible because it’s a high-quality open-source project, but also because its ability to handle high-velocity streams of information is increasingly in demand for usage in servicing workloads like IoT, among others”
I mentioned earlier we have seen a big uptick in developer interest over the last eighteen months. Looking at multiple data sources we can see some clear trends around this uptick.
On Stack Overflow we saw a marked increase in questions beginning in mid 2014:
This trend on Stack Overflow matches into the growth in stars for the Kafka project on GitHub.
Google trends demonstrates the same growth, with an uptick starting in mind 2014.
The most interesting data point, however, has been the consistent growth of active users on the kafka users mailing list, which is just over 260% since July 2014.
We define an active user as someone who has sent an e-mail to the kafka users list. The majority of these questions tend to come from a new users of Kafka. While the volumes of mails remain relatively constant this continuous growth in active, rather than passive, users, is indicative of a strong and vibrant community.
Disclosure: IBM and Cloudera are current RedMonk clients.
All your streaming data are belong to Kafka – hozt.me Tech News says:
July 31, 2017 at 10:09 am
[…] on a roll. Last year it registered a 260 percent jump in developer popularity, as Redmonk’s Fintan Ryan highlights, a number that has only ballooned since then as IoT and other enterprise demands for real-time, […]
Todos sus datos de transmisión son pertenecen a Kafka – High Tech Newz says:
July 31, 2017 at 11:23 am
[…] pasado se registró un 260 por ciento de incremento en desarrollador de popularidad, como Redmonk Fintan Ryan destacados, un número que sólo se ha disparado desde entonces como IoT y otras demandas de la empresa en […]
All Your Streaming Data Are Belong to Kafka - Nerd Junkie says:
August 1, 2017 at 12:54 am
[…] is on a roll. Last year it registered a 260 percent jump in developer popularity, as Redmonk’s Fintan Ryan highlights, a number that has only ballooned since then as IoT and other enterprise demands for real-time, […]
Apache Kafka continues its ascent as attention shifts from lumbering Hadoop and data lakes to real-time streams - Lagmen says:
August 1, 2017 at 12:23 pm
[…] is on a roll. Last year it registered a 260 percent jump in developer popularity, as Redmonk’s Fintan Ryan highlights , a number that has only ballooned since then as IoT and other enterprise demands for real-time, […]
9 Hot Tech Skills for 2018 says:
January 10, 2018 at 9:28 pm
[…] in demand for Apache Kafka courses on Udemy last year, making it the 4th hot tech skill for 2018. Apache Kafka-related questions on Google Search and tech forums like StackOverflow and Github have also skyrocketed in recent […]
4 Key Benefits of Apache Kafka for Real-Time Data says:
March 8, 2018 at 7:50 pm
[…] prioritizing enabling more real-time architecture and reducing the wait time on data availability. Apache Kafka-related questions on Google Search and tech forums like StackOverflow and Github have also skyrocketed in recent […]