What’s new in message queues? Message queue-based architectures, a dominant paradigm in distributed systems, are essential, well understood, and therefore increasingly ho-hum. Multiple open source and commercial messaging technologies have positioned themselves as necessary components for building scalable, distributed systems, particularly in the context of microservices. To name just a few popular options, the likes of LavinMQ, NATS, RabbitMQ, Redis, SwiftMQ and Apache’s ActiveMQ, Kafka, and Pulsar are open source tools that implement a message broker or support a messaging use case; and among commercial products we find enterprise and managed services such as Confluent, Oracle SOA, TIBCO, Red Hat AMQ, RedPanda, Google Cloud Pub/Sub, Amazon SQS, IBM MQ, HiveMQ, Synadia, and StreamNative…. (phew!).
These systems, which are sometimes called message brokers or queue managers, are a component of message-oriented middleware (MOM) that enable communication via two main patterns: message transmission and reception (1-1) and publish-subscribe (1-many). Communication is usually (not always) via a server, which provides value added services such as queueing, stream management, persistence, and high availability (HA). MOM is a tremendous improvement on the terminal server model inherited from the bygone mainframe era. Message queue-based architectures have become invaluable in distributed systems because they store, order, and deliver data packets for processing from one application to another. These standard, reliable tools are no longer touted as cutting-edge or exciting, but rather categorized as fundamental infrastructure beside the likes of TerraForm, Linux, and MySQL, with integration being a significant use case outlined in Gregor Hohpe’s book Enterprise Integration Patterns. Alex Hutcheson, staff software engineer at Google, encapsulates this sentiment in his recent Hacker News post:
In the late 2000s and early 2010s, I remember seeing lots of hype around building distributed systems using message queues (e.g. Amazon SQS, RabbitMQ, ZeroMQ, etc.) A lot of companies had blog posts highlighting their use of message queues for asynchronous communication between nodes, and IIRC the official AWS design recommendations at the time pushed SQS pretty heavily.
Now, I almost never see engineering blog posts or HN posts highlighting use of message queues. I see occasional content related to Kafka, but nothing like the hype that message queues used to have.
Factors such as technological maturity, changes in the developer mindset, and the rise of alternative architectures, go far toward explaining why developers like Hutcheson note the waning buzz for message queue systems among developers, despite their continuing wide usage. In fact, Datadog’s 2022 Container Report found that:
In Kubernetes StatefulSets, … Redis, Postgres, Elasticsearch, RabbitMQ, and Kafka were the most commonly deployed images.
Image credit: Datadog’s “9 Insights on Real-World Container Use”
Messaging services are well-represented in this graph (Kafka, RabbitMQ), with Redis being an especially interesting data point owing to its use as a message broker in distributed systems. Just because the sparkle of message-queues has dimmed doesn’t signal a technology’s demise. In fact, multiple analyst firms note the MOM market’s continuing growth. Following Confluent’s acquisition of rival WarpStream Labs last month for an undisclosed amount, the messaging space appears anything but stagnant.
So what’s going on? Opinions differ but that doesn’t make it any less worthwhile to take stock of messaging in 2024 in terms of where we’ve been and where we’re headed. As this is a tremendously large and unwieldy topic, I eventually despaired of trying to fit it all into one post and have instead opted to divide my research into a series—so buckle-up, friends!
In the coming months I will be discussing what we at RedMonk are seeing around messaging from vendors, developers, and insiders. I am leaning heavily on RedMonk’s rolodex by conducting interviews with several lights from the industry in order to uncover the less visible currents steering this ship (keep an eye on the MonkCast for many of these). In addition to a history of messaging, including transaction processing systems for airlines and banks, the appearance of IBM MQ and Tibco RV, and unification with Java and JMS, this series of blog posts, videos, and podcast episodes pay special attention to shifts in the messaging space for what they suggest about the future of distributed systems and microservices.
Acknowledgements: I want to thank James Governor, Alexis Richardson, and Clemens Vasters for reviewing and providing feedback on earlier drafts of this, and all the posts in this series.
Disclaimer: IBM, Google, Redis, Synadia, VMware, Oracle, Red Hat, and Amazon are all RedMonk clients.
No Comments