In this RedMonk conversation, Andy Stanford-Clark, Distinguished Engineer and Innovation Leader in IBM Research, and Clemens Vasters, Principal Architect at Microsoft, chat with Kate Holterhoff, senior analyst at RedMonk, about the history and evolution of messaging technology. They trace messaging to its roots from early telco technologies to modern events and streaming systems. The conversation highlights the significance of message queues, the role of IBM and Microsoft in advancing message-oriented middleware, and the evolution of open protocols like MQTT, AMQP, and CloudEvents.

Microsoft and IBM are RedMonk clients, but this is an independent and unsponsored conversation.

Links

Transcript

Kate Holterhoff (00:13)
Hello and welcome to this Redmonk conversation. My name is Kate Holterhoff, Senior Analyst at Redmonk. And with me today, I have two giants in the messaging space and longtime friends of Redmonk, Andy Stanford-Clark, distinguished engineer and innovation leader in IBM research, and Clemens Vasters, Principal Architect focused on messaging and real-time intelligence services at Microsoft. Clemens and Andy, thank you so much for joining me on the MonkCast.

Andy Stanford-Clark (00:36)
Great to be here, Kate. Thank you.

Clemens Vasters (00:38)
Yeah, thank you for having me.

Kate Holterhoff (00:40)
Yeah, this is going to be a really great conversation. know it. So I’m hoping we can weave together a discussion of messaging that synthesizes where we’ve been, where we’re headed. Because although messaging is a well-established and foundational computing technology, I have struggled in my own research to really pin down this story. So let’s begin at what you would consider to be the beginning.

you know, where did this need for messaging come from? What do you consider to be significant events in this timeline? And, you know, how does it stretch to where we are today? So Andy, would you go first and try to break apart this, I know, very challenging question.

Andy Stanford-Clark (01:13)
Sure, yeah, where do we start? So think I’m going to start. So one thing that happened yesterday, I met my dad for the first time in a while, and we’re reminiscing about the fact that It’s MQTT’s 25th birthday tomorrow. And I thanked him for being so tolerant of me using dial-up modem on our home phone line when I was a teenager to contact all the bulletin boards that I used to dial up with my TRS-80 Model 3 computer.

Kate Holterhoff (01:26)
Congratulations, yes.

Andy Stanford-Clark (01:44)
spending hours on making phone calls and he said he had the only house in the UK where the phone bill was more than the heating bill which was quite significant but that really started my career in computing my interest in getting computers to talk to each other and another interesting moment was a neighbour of ours when I was a teenager was asking me, he ran a holiday let company and each site had its own computer and you wanted to dial, you said is there any way I can connect to all of them and bring all the accounts data centrally so that I can then do the centralized accounts because at the moment you had to phone people up and you to like post it to him or read it to him over the phone. I yeah you need a modem and he came back to me and said this mode M thing that you’ve been talking about so it like a special mode of computers.

Kate Holterhoff (02:39)
Ha!

Andy Stanford-Clark (02:41)
So he didn’t know what a modem was, but to me that was the beginning of my interest in messaging. this was before the internet, so it was when this was point to point dial up connections to things like bulletin boards and CIX, a CompuLink information exchange we all used to dial into in London to download our messages and send what we now call emails to each other. So it was all starting to shape up and it was all proprietary protocols in those days and I think…

I’m sure we’ll get on to talk about SCADA stuff and how proprietary protocols pervaded that industry and how the need for standardization and things that anybody can use, whether they’re RFCs for the internet or whether they’re standards done through ISO or whatever, are essential for what later become competitors to collaborate, to create, but to level the playing field. So right now we’ve got this protocol we all agreed to speak over.

let’s go off and do what we do to help our customers do it. And that was the beginning of that. I will mention just one more thing. And that was when we did the Atlanta Olympics in 1996, we did a messaging system which sent live, there’s a little Java app that you could have, Java application in fact, you have on your screen and you could subscribe to the…

events and the sports people and the countries you’re interested in and when something happened in the Olympics for one of those sports or people or countries it would go and lights would come on and it would kind of flicker and all the the results will come up for just the things you’re interested in and we implemented that over IRC, Internet Relay Chat because it was a one-to-many distribution pattern and we invented a little protocol, little ASCII based protocol to send over that and but I

kind of looking back, and this was only two years before the start of the MQTT work, that was the precursor to one too many PubSub, publish and subscribe messaging. So it was all starting to plant seeds in my mind for what came later.

Kate Holterhoff (04:49)
That’s helpful. So Clemens, would you talk a little bit about what you consider to be the story of origins of messaging and talk about your own entry into that space?

Clemens Vasters (04:57)
Let me talk about my entry. The parallels are going to be quite stunning, I would say. So 0216657775 was the phone number of my BBS that I ran in 1988, 89, 90. It was called Programmer’s Paradise and it was a FidoNet node.

Andy Stanford-Clark (05:15)
I think I over called that one.

Clemens Vasters (05:25)
Back in the day in Germany, running high-speed modems was punishable by law with five years in prison for interfering with telecommunication systems. So quite serious business. But we ran the US Robotics HST modems anyways.

Andy Stanford-Clark (05:40)
Wow.

Clemens Vasters (05:46)
with special setups and ripcords. So if the postal man came, then we had to go and rip this out. Everybody in the family was instructed. So my distributed systems origins also come from speaking with other people that email and dealing with all the protocols, so doing some low-level work there.

And then also in parallel, I was, you know, with all the modems. I worked at funny. The Greeks wanted to have the 1996 Olympic Games, of course, the centennial. They wouldn’t, they didn’t get this because Coca-Cola is in Atlanta.

and was a big sponsor. So the Greeks were very, very unhappy that they didn’t get the games in 1990. So in 1991, they set up the Mediterranean games and said, well, we have everything ready. We’re going to go and make a big thing out of this. And I went with a German production company.

to help with those Mediterranean games. And my job was to go to the various sites and connect the modems and make sure that the transfer of the sports results could actually happen consistently. So, there, Andy. Interesting. So, I do a talk in…

Currently that’s called, what is the message queue and why would you use one? And as the, the introduction to that talk is I speak of messengers in general, right? And I have a lady with a carrier pigeon in her hands giving her a love letter. I have a couriers who take telegrams, you know, back to the people who should receive them. I have couriers with.

carrier pigeons who were in the field in the first World War, ensuring communication between the soldiers in the trenches on all sides. I think ultimately what we’re doing here in electronics is kind of in thousands of years of the Roman postal service tradition of getting data from A to B.

That’s, and we’re kind of building the modern kind of form of infrastructure of getting that data from A to B, but not in days or weeks, but in milliseconds. And we have, and the first formal infrastructure that I know of that had the name of a queue on it, and that he could go and run as a piece of software and send something into and get something out of was an IBM 360.

So system 360 built for in large part funded by the Apollo program in the United States. And they came up with in 1964, 65, they came up with that first version of a complete application platform that then also included formally in the telecommunication subsystem, a queue system. And I consider that kind of the first really queuing infrastructure that existed.

Andy Stanford-Clark (08:57)
Cool. I’m glad we were there at the beginning.

Clemens Vasters (09:01)
Yes, you were. Well, nobody else was there at the beginning, so you were there.

Andy Stanford-Clark (09:05)
if by default here.

Kate Holterhoff (09:08)
had no idea the Olympics were going to factor into this conversation. This is so exciting. And also, I’m based in Atlanta, so I feel like I’m at the heart of this. You can either blame me or thank me.

Clemens Vasters (09:11)
Well, there you go.

Andy Stanford-Clark (09:11)
Well, there you go.

Aha, very good.

Kate Holterhoff (09:20)
it’s no accident that in my efforts to understand messaging, I invited guests from IBM and Microsoft. These companies have been instrumental to advancing message-oriented middleware from the beginning. And as insiders, I’m extremely interested to hear about the sort of internal levers that were pulled behind the scenes. How did you identify messaging as a sort of worthwhile area of investment? what sort of incidents got the ball rolling? Would you want to begin again, Andy?

Andy Stanford-Clark (09:44)
Yeah, so think the key thing to focus on is asynchronous messaging. So as Clemens has just said, it’s a queue-based messaging. we, obviously people have been wiring computers together and wiring applications together for a long time. But it was always in a synchronous way. So in other words, one application had to be ready to receive the message before it could start sending it. They had to receive the whole message and acknowledge it before either could carry on. And we used to compare that to the wings of a

used to say a 747 back in the day, because it shows how long ago this was. Wings of a jumbo jet when it was going down the runway to take off, the wings actually flap slightly, I can’t see them on camera there, they go up and down slightly, because if they were rigid they’d snap off, so they have to have that flex in them, and by putting a cue or a springy connection in between two applications you stop them being rigid and then one of them snapping off when the whole thing breaking.

Clemens Vasters (10:17)
Yeah.

Andy Stanford-Clark (10:39)
you allow them to a little bit of flex. this one’s not quite finished the previous job yet. Message goes into the queue. This one can carry on with what it was doing. This one will pick it up when it’s ready. And it just, that give and take in the system allows the whole thing to carry on without it all having to be in strict synchronous lockstep. And it turns out that was the only real way to proceed with the complex situations, applications, people, sensors, all the things that were in an enterprise. So this idea of queue-based.

messaging came to pass and IBM had MQ series which later became WebSphere MQ which was literally point-to-point queue based messaging so you put a message into a queue and at that point application A could consider it actually delivered to application B because technically it was because we had this assured once and once only delivery so once you put it into the queue that was it you could you could safely forget about it there’d be no retries there’d be no comeback

it’s gone, it would at some point be delivered to application B. And that was a very powerful concept. It means you didn’t have to store it in a database locally, do, you know, get in case there any retries or anything. Once it had been accepted by the messaging system and we used two-phase commit to do proper transactions. So you could take it out of a database, put it into a queue, delete from the database, commit to the queue, and it will definitely happen. And banks were starting to, financial institutions were beginning to rely on this.

to transfer money, know, trillions of dollars in some cases between the bank clearing systems because once it had been entrusted to MQ, it was definitely going to be there. One of my customers said that, it was a European clearing bank, and they said, if we stopped all the MQQQs at this point, there’d be $7 trillion in flight at any given time. I went a bit white and thought, don’t lose any guys. So I went back to Hursley where,

the MQ development work was happening and I said just been told people are entrusting seven trillion dollars at a time to our queue so make sure you don’t lose any. So that was a real founding basis of it and I guess from that we then saw this additional thing that often application A and application B didn’t talk the same language so it’s a bit like giving a French person, a German person a telephone one could speak French and one could speak German they could hear each other but not understand each other.

Clemens Vasters (12:48)
Hahaha

Andy Stanford-Clark (13:06)
So we had this idea of putting a broker in the middle where message would be delivered from the application A to the broker where it can be transformed, like a dictionary lookup or some transformation from JSON to XML or whatever, and then sent from the broker to application B. A and B wouldn’t know that they weren’t both talking the same language, but they’d each receive the message in their appropriate language. having that centralized broker then became a very powerful way of essentially putting in.

additional code that would help glue these applications together. And then we thought of the idea of doing one-to-many distribution. This idea of publish and subscribe messaging pattern came into into Vogue. And so we all started doing PubSub with a centralized broker. And that was really the fact you could add an additional application, B, C, D, E, whereas application A was just publishing one message into the broker applications B, C, D, and E.

could be consuming that data, and A, wouldn’t know or care how many people were consuming that data. And that hugely increased the flexibility and the fact you didn’t have to keep going back and rewriting application A every time you thought of a new subscriber application as your business grew and your supply chains became more complex and so on. So that really was, I think, probably the blueprint for where we find ourselves today with message-based queuing.

Kate Holterhoff (14:27)
All right, and Clemens, let’s talk about Microsoft. What is the role of Microsoft in moving this space forward?

Clemens Vasters (14:32)
And so I think the story of messaging at Microsoft starts with IBM, indeed, and with the role of Windows relative to IBM systems. And in the earliest days, before I would say the mid 1990s, Windows was a client, Windows NT shipped as an operating system kind of in 1992, 93.

And really became, you there were always two versions before this, and then we had even a Unix, Scenics before this. But really like a serious…

game in terms of servers only became realized in the mid 1990s and going forward. Initially, just as file and print servers rivaling Novell NetWare, which was the dominant operating system at the time, but then more and more as application servers. And during those days, all the mission critical systems were all built on mostly on IBM mid-range and mainframe systems. And of course, they’re competitors, but IBM being the the biggest business system provider overall.

The first wave of kind of application services that came out was in 1997, MTS, the Microsoft Transaction Server, which contained several transaction oriented services, including a queue, the Microsoft Message queue, and the goal of the Microsoft Message queue and the systems around it, including SQL Server, et cetera, to provide a…

Small server based alternative to all the IBM to all the IBM capabilities. And that was kind of the start of client server and client server kind of with PC based architectures. And MSMQ was an important part of this. That went then on through become COM+.

inside of Windows. There were then new services that were called queued components where you could go and create a component, create a function, and then you could have a function automatically driven by a message showing up on the queue. That’s what you do today when you look at serverless architectures with Lambda or with Azure functions and you hook them up to a queue. It’s effectively the same functionality that then existed. 2000 is when they all came out officially with Windows 2000.

as a capability. And then as that happened, we were all kind of entangled, know, Andy’s people and my people in a love-hate relationship around XML web services with enormously formal meetings.

that were worse than meetings of two opposing law firms occasionally. But what came out of it was the WS-* set of specifications, the web services, SOAP, et cetera, specs. they kind of drove, they drove a, successful they were and they are, a lot of that XML stuff is still being used and a lot of the SOAP stuff is still being used.

Andy Stanford-Clark (17:55)
Yeah, For better or for worse.

Clemens Vasters (17:58)
Yes, for better or for worse, that’s right. But it has some echoes that I’m gonna get to in a moment. And then what then happened is that out of the team that owned web services in Microsoft, there was an incubation started by John Shewchuk which then consisted of two people who then explored simply…

Now we have, now when we are equipping all of these companies with the ability to go and run web services, how do they talk to each other? And reminder, this was before the day of public cloud, before the idea that there would be neutral spaces you could go and run anything under, but there was the assumption that everybody would live kind of behind their own firewalls and kind of how would that even work? I mean, we’re now building interoperability, but if they can’t talk to each other, how does that work? And they then pioneered the idea of the relay.

which is a web service bridge service that lives somewhere in neutral place, still exists as a service actually in Azure. It’s the oldest service that exists and that was the first beta came out in May 31st, 2006. And what we today have in Azure as the fleet of Azure messaging services which process over 11 trillion transactions per day,

is the straight development kind of from that nucleus of kind of building this bridge in the cloud that then actually became part of the cloud services. And since then we’ve added a queue broker.

that is speaking AMQP. We also have been driving this AMQP standards with industry partners that is providing queue based messaging and is one of the probably the largest JMS 2.0 compliant broker in the world, certainly in terms of absolute scale that it runs. We have event stream and event hubs, which is a Kafka compatible, but also AMQP compatible event stream broker and several other services including event grid speaks MQTT 3.1.1 and 5.0 and Andy can probably talk about me and MQTT at some point.

and so that’s, but that’s the, like, like we have, we’re working in the Azure cloud mostly to, you know, build this kind of services that customers want. And that’s how we come up with this large bouquet of services, because it’s all, it’s always as Andy says, always about getting data from A to B. But there’s so many facets, especially on the consumption side that, you know, we come, keep coming up with new services and customers keep using that entire spectrum of capabilities. But that’s kind of how this short story of how we got there.

Andy Stanford-Clark (20:55)
how some of those standards have really stood the test of time. Although more in concept rather than reality, in actuality. I was talking to some people the other day who were trying to invent a service discovery system for the internet and I said, don’t do that, it’s all been done. I know it’s all in XML and SOAP and horrible stuff like that but go and look at UDDI.

Clemens Vasters (21:19)
Yes.

Andy Stanford-Clark (21:20)
Please don’t reinvent that wheel because people die trying to get that thing to work. Absolutely, yeah. The other thing that’s probably worth mentioning is you mentioned Kafka. As things have come up to date and the web era, if we can still call it that, is still well underway. People observe that when you, this is to do with transactions, when you buy something on Amazon, you click to purchase.

And there isn’t a two-phase commit then between, we always say you can’t do two-phase commit to the eyeball, it’s the visual feedback. send and acknowledge, which I would refer to as QoS 1 in MQTT speak, simple acknowledgement is what we use for trying to, buying things on the internet. So surely it’s good enough for enterprise messaging, who needs transactions? what people, the youngsters have implemented web.

Clemens Vasters (21:55)
Yeah.

Andy Stanford-Clark (22:16)
interfaces did, where they said, we don’t need transactions, all that MQ stuff, it’s really, really heavyweight, we’ll just do a simple acknowledgement, and then if it doesn’t work, we’ll retry it, or we’ll hold on to the data until we’re sure it’s there. So they’ve invented all this stuff at both ends, which is what MQ and other transactional queue based messaging services, which do exist, implement, somewhat seem to have kind of fallen from grace in the

Clemens Vasters (22:23)
Yes.

Andy Stanford-Clark (22:45)
the front end of things, so between web clients and servers. Obviously, it’s still enormously important for enterprise and B2B, but the kind of B2C applications tend not to be using transactional stuff. So that’s ushered in this whole new era of lighter weight, less reliable messaging protocols where either the human has to intervene to confirm that it got there, or you have to retry after a while or make sure that the…

you don’t have idempotency so if the same message gets transmitted twice you go, we’ve already seen that one and throw away the second one. Stuff like that which you didn’t have to do when we just had once and once only sure delivery, bump, it’s gone, done.

Clemens Vasters (23:30)
Yeah, we fight that fight every day. And so since we have two indigenous services that kind of implement those patterns, Event Hubs and Service Bus, we’re confronted with this where customers always come and say, I want to use Event Hubs for everything. We’re like, no, no, no, don’t do that. So all of the 500 largest Microsoft customers use the messaging services that we have. And all of them,

because they actually get advice. All of them use all of the message services that we have. And they do so because AQ has one important feature. And that’s not the order or the buffering, it is exclusive assignment. What it does is it gives a processor a job for the processor to own.

and then to do the work on. And Kafka and all the event stream processing platforms cannot do this because they don’t have any individual tracking of the life cycle of a message. people build completely absurd contraptions around Kafka to kind of sort of get this. And the easiest way to get it is to just use a queue.

Andy Stanford-Clark (24:41)
Exactly, to recreate that functionality.

Clemens Vasters (24:48)
And there’s nothing wrong with using multiple of these messaging infrastructures in your architecture, right? If you can stand up a web server in the Kafka, well, then you can also set up a web server in the Kafka and the broker. So, and then use the right communication path for the right things. We’re doing that. I’m dealing with this every day. And I’m actually dealing with this internally. I just had an email from an internal team that says, well, we have decided to standardize on everything on Event Hubs. Where is an article that I

can read that tells me how to solve this particular problem, exclusive assignment. And I wrote, if we had an article about this, it would say, don’t do this. So I said, don’t. And I said, this is an authoritative answer. Do not do this, please. so yes, Andy, I feel you. There is a lot of propaganda that’s around from the younger companies.

And I call this one pony architecture, because if you only have one pony to sell, then you’re going to try to do everything with that one pony. but that pony doesn’t know all the tricks. So, you use something where you have an appropriate infrastructure, appropriate patterns, basically for the problem you’re trying to solve.

Andy Stanford-Clark (25:48)
Hmm.

Check.

Kate Holterhoff (26:03)
I’m glad we brought in Kafka at this stage because it is certainly the flashiest paradigm in messaging, especially at the edge that we talk about here at RedMonk. And I actually spoke with Alexis Richardson, the founder and CEO of Rabbit Technologies. later at VMware. But he talked about the fact that there is a lot of contention about whether streaming is a separate use case. And today it’s becoming sort of table stakes with RabbitMQ and NATS supporting streaming. So I guess I’m certainly interested in hearing more about your sort of sentiments about.

about streaming as this paradigm. But would you agree with that characterization that it is sort of becoming table stakes? It’s just everyone is supporting this now?

Clemens Vasters (26:33)
Mm-hmm.

why all the platforms are somewhat alike, if you look at AWS and Google and Microsoft and also with IBM’s offering, is that every year we go through the dance with analysts.

where we get a questionnaire with 500 different questions and that’s not an exaggeration. And they basically kind of nudge you into what the table stakes ought to be and what you ought to support. And you don’t want to come up blank in any of these categories. So it ends up being that the consensus amongst analysts kind of drives lots of the feature work really, which is kind of funny because the customer should drive it, but there’s lots of checkbox, make the analysts happy.

Andy Stanford-Clark (27:07)
Yeah

Clemens Vasters (27:23)
that is happening. meanwhile, yes they do. so, yes, streaming is table stakes and building better and better streaming platforms is a huge area of in the competition between us and AWS and the likes of Confluent, et cetera, to make streaming better.

Andy Stanford-Clark (27:24)
becomes a self-fulfilling prophecy. Then we come to the self-fulfilling prophecy exactly yeah.

Clemens Vasters (27:52)
But it is a different category from transactional job processing. It is about if you look at what were our

Like if you look at how Microsoft presents itself right now in the marketplace around streaming, you hear a lot about Microsoft Fabric. Microsoft Fabric being our big data platform, which has all the various data engines integrated and Spark and all these things kind of in one box that is speaking to the analysts, the data analysts, and some of the data engineers more than to application developers, because that’s where lots of the usage of streaming is. And jobs processing

queue based processing is more on the application side. So it’s split itself into kind of a data story, like of course usage in application, but very much a data story around streaming and an application integration story around queueing. So I think these are things that everybody needs to have in their toolbox.

to build apps, but we’ll see more and more kind of data-focused, data-centric development around streaming specifically.

Andy Stanford-Clark (29:02)
Although I think it’s fair to say that one person’s streaming is another person’s discrete messaging. And obviously a stream is just a series of discrete messaging. And if you look at the notion of a session across a number of messages and reduce the overhead of each message, known as the header, then you’ve effectively turned a series of discrete messages into a stream. So it’s a very, very gray line between, you know, is that a stream or is that a set of continuous stream of discrete messages? So it’s always a…

Clemens Vasters (29:07)
Yes.

That’s correct.

Yes, that’s correct.

Andy Stanford-Clark (29:32)
a boundary that you can step either side of depending on what you want to achieve.

Kate Holterhoff (29:35)
I want to return to something that we had talked about before, which is the idea of sort of open source and open protocols in messaging, because I think that’s really important part of this story. And you sort of mentioned the going into rooms and arguing about specifications. I feel like this is all part of that. So the complexity of this space seems to call for this. So talk to me about the history of open protocols. What does that look like? What do you think are some of the more important parts of that story.

Clemens Vasters (29:49)
Hehehehehe

Andy Stanford-Clark (30:01)
Well let me talk about MQTT. I’ve got the t-shirt. And MQTT is 25 years old on the 22nd of October. Essentially the own, so this was invented as a SCADA protocol for remote telemetry for oil and gas pipeline systems. And back then there were, going back to our conversation about dial-up modems, there was no real packet switching.

Kate Holterhoff (30:03)
Please? Yes.

Clemens Vasters (30:04)
Yeah

Andy Stanford-Clark (30:25)
no real internet being used on these links. a central computer would dial up a remote sensor and say, what’s your reading? And then it would go on to the next one, the next one is called poll response. there were hundreds and hundreds of proprietary protocols that were being taught across those dedicated links, almost from the same manufacturer, you’d get variants of Modbus and Allen-Bradley and Honeywell, all these different protocols, and literally hundreds of them.

It was becoming impossible to support all of those at the enterprise end when all those messages were coming back. You’d have to decode each one from each different protocol. And every time you bought a new sensor or replaced one or upgraded it, you have to implement another protocol across all your applications. It was becoming seriously unworkable. So we said, this is me and Arlen Nipper, who was working then in the SCADA industry, supplying the little boxes that go out at the far end of these links.

We said, if there was just one more protocol? It was very simple to implement, very easy to understand, very small footprint on the device, because those devices were only sort of really small and crummy, sort Z80, 16K memory type things in those days. Not very chatty on the wire because connections were either slow modems or really expensive satellite links. And imagine it wasn’t proprietary and you didn’t have to pay for the spec and it wasn’t 300 pages long, just really small, simple protocol.

And it was public domain, open source. Didn’t reuse the term open source in those days. was put into the public domain from day one. So anyone, there’s just one more protocol you’ve got to implement. And that’s the kind of one ring to bind them all. And that’s what became MQTT, which has been open and free since its inception. The protocol was standardized as an OASIS standard and an ISO IEC standard.

And the people who collaborated to create those standards then competed by implementing it the best, the server end, more scalable, more robust, more sort of CAP semantic. Consistency, availability and performance. I always forget what the C stands for. Those three things can be traded off. people made their livings. There are many companies that have sprung up because of MQTT.

Kate Holterhoff (32:36)
Okay.

Andy Stanford-Clark (32:47)
I was talking last week to HiveMQ, they’ve developed a broker and they run a very successful business just because of MQTT, which I’m very proud to know. And that really, when someone’s talking the same protocol, the things you can then implement, it gives good reason for device manufacturers to burn it into their silicon. It gives application vendors good reason to support MQTT in their applications because…

it’s always the same whether it’s 3.1, 3.1.1 or version 5 in a sense doesn’t matter as long as it’s the standard then you can guarantee to interoperate and that’s the wonderful thing about standards.

Kate Holterhoff (33:26)
All right, and let’s pause on this before we move on to AMQP, which I’m hoping we can also dig into. So with MQTT, talk to me about where this space is going, because 25 years is not a small amount of time in terms of computing. Do you think that it still retains the robustness because there continues to be this need in the IoT space for a service like that? And when it comes to open source and open protocols, having agreement in the community that this is the way we’re going to go.

can sometimes have give and take. I I talked to lot of folks who are excited about Rust right now, but a lot of older services aren’t written on Rust, and so maybe they’re a little bit slower, but they are gonna be interoperable, right? They’re gonna work for everybody. So how do you perceive the future of MQTT, and why are you still excited about it a quarter of a century later?

Andy Stanford-Clark (34:14)
Yeah, well, as we were saying on a podcast just a few days ago, I mean, we’re raising a glass to the next 25 years of MQTT because its popularity shows no sign of slowing down. In fact, in many ways, nature of an exponential curve, it took 10 years for anything to happen, pretty much. I could have enumerated all the projects in the world that were using MQTT because I was hands on involved in all of them. But then after this kind of tipping point around 10 years in, a lot of what was fuelled by…

Roger Light creating Eclipse Mosquitto, which is the open source MQTT broker and other things as well. This exponential curve took off. Now we’re not just seeing it in the IoT world. We’re seeing it in many application areas. We’re seeing people using it for, I heard there’s an antivirus company that’s using it for talking to all its endpoints, updating all the antivirus things. Some major car manufacturers are rolling out millions and millions of cars which use

always on MQTT connections for their calling back home. We’re seeing it all over the place and it’s becoming not a replacement for enterprise messaging, because obviously there are lots of features, transactionality being a key one, heavy duty as I might call it, enterprise messaging that needs a fiber or LAN to talk over makes perfect sense for, but for anything else, the low bandwidth end of things, or where you’ve just got to link two things, applications together and

A lot of this standardisation is that you don’t need to care about what’s below you. The reason we use TCP/IP is that you don’t care what’s down in the gubbins of that. With MQTT, you don’t need to worry about how it does it or how it works. Just decide what will you send and what will you do with it when it gets there. And being freed up from having to worry about anything else is so liberating because you spend all your time then worrying on the application domain, helping your customers with what they’re worried about.

And that’s what makes it, because it’s so easy to use, so easy to download a client, so easy to get working. I was talking to a colleague last week and he said, we used MQTT for this project. I said, great. How did you find it? we just downloaded the library and started using it. And that was it. OK. Boom, done. There was no further discussion needed. Great. That’s what it’s for. That’s what we hoped when we invented it, that people would find it easy to use.

Kate Holterhoff (36:34)
All right, here’s to the next 25 years then. Well, Clemens, would you mind talking a little bit about your sense around open source and open protocols? And I know that Microsoft uses AMQP for their messaging service, so maybe that would be a good place to start. But I also understand you have some experience with MQTT, so maybe you’d want to talk about that.

Clemens Vasters (36:47)
Yeah. Yeah, let me connect to the MQTT piece first and then I’ll talk about AMQP. We had looked at MQTT, started to look at MQTT when this IoT…

in the beginning of the IoT wave, when IoT became a term, and we were already building cloud services at the time. So now I’m talking 2012, 2013, just about that time. And then MQTT standardization was already going on. So IBM had donated their 3.0 version kind of into OASIS for standardization. And the charter was such that it was mostly like edit what we have and then turn it into a standard.

And the charter of the working group basically said, we can’t make any incompatible changes on this. Okay. So we did an analysis on this or I did that analysis on it. And we found that for implementing this in the cloud service, that would be difficult in some places because of, you know, error handling issues and a bunch of other things. made a list. Then the IBM folks were done with that work.

I made a list public and then we ended up having a formal discussion internally between IBM and Microsoft and said, hey, we’d like to participate in the next round of this. And then we ended up having a few folks involved in the next round, which then turned out MQTT 5, which is a super robust protocol with proper error handling, extensibility, et cetera, which is, I think a lot better than 3.1.1. it’s a, you know, everything around MQTT is open industry collaboration.

It’s a fantastic protocol specifically for connecting out to the edge. And we’re using that for automotive fleets. We’re also helping the OPC foundation. That’s another big story around standardization. OPC foundation, industrial manufacturing, who are adapting their PubSub standard kind of to MQTT or running that over MQTT. There’s also other things where we are, Sparkplug B is a

as a standard industry, it kind of sits on top of MQTT There’s a lot of good stuff going on even in the verticals. so AMQP AMQP since we’re a history podcast, AMQP was a reaction of Wall Street or the banking industry to IBM’s dominance in the space.

and to Tibco’s dominance in this space. so IBM was not supportive or happy about AMQP when that all started. I have for quite a while, I’ve actually, I thought that it was truly evil malicious intent as you are if you’re young person. But what I found later,

Andy Stanford-Clark (39:34)
True.

Clemens Vasters (39:51)
is through speaking to people, is that a lot of IBM’s hesitance to document their protocol properly, because that’s ultimately what was that one of the complaints was of JP Morgan, who’s kind of started this, but also the other banks that worked on it, is that they didn’t have any documentation for the IBM MQ protocols. And so therefore could not build services that were not depending on IBM.

But a lot of the IBM protocol was really just, there was code on one side and there was code on the other side and it kind of made it happen. But there wasn’t any formal like protocol spec that says.

Like an RFC that says, this is what the IBM MQ protocol was. It was basically, and there was three versions, I think, of that code where there were three protocols that were the IBM MQ protocols, effectively, but there were effectively just the implementations of what happened on either side. So.

In terms of technical debt and whatever we talk about, right? You can almost understand that, you know, that’s how it happened. So ultimately what was needed was an open, know, spec first protocol that everybody can then go and implement. And JP Morgan started this and then they got.

a bunch of people to consult on this. Amongst them I should mention my friend Pieter Hintjens, who unfortunately passed away and left us. He was a staunch opponent of big corporate entities, I would say, including IBM and Microsoft. Certainly we didn’t see eye eye in the beginning, but then towards later towards the…

His unfortunate passing, we really became friends and I got to see him. So big props to Pieter Hintjens for doing lots of that work. So AMQP ended up being standardized, not without hiccups. There was a 0.9 version that’s…

defined very clearly inside of the protocol, the role of a queue and the role of an exchange and the role of subscription, which was very rigid as a built in topology. We were against that. Other banking customers were also against that. The rabbit people were for it. so there was the AMQP schism that then happened where the RabbitMQ people stayed on 0.9 and evolved that a little bit, but on their own. And then the rest then moved on to AMQP 1.0

And Apache Artemis and the Apache Cupid projects, are kind of open protocol AMQP brokers. We implemented AMQP in our broker. The Solace folks implemented AMQP. There’s an AMQP bridge that sits in IBM MQ. So there’s numerous brokers that implement AMQP kind of as their primary protocol. And I’m super, super, super happy that the RabbitMQ folks finally came around. They just shipped last month.

4.0 version of RabbitMQ and that now has AMQP 1.0 as the primary protocol. And if you read what they’re saying in their blog release, they’re also saying, yeah, 0.9, we’ve been sitting on this for a long time. Here’s the kind of backstory, but sorry. And now we’re go into this interoperably and they’re actually using the stacks that we built for .NET as going forward as their client, as do the Red Hat folks who are kind of…

maintaining mostly the Apache brokers. One of the great stories, I think, around open source in that context and harmonization is the second phase of AMQP standardization happened like four years ago, maybe, where we sat down mostly with the Red Hat people to work on a set of companion specs with the goal of creating one GMS provider that would work across most AMQP brokers. So we brought a bunch of

standard specifications to enable this connection management, et cetera. And then we work together on the Apache Cupid JMS project to build this one client that supports everything, all the features of JMS 2, including transactions on top of that common base. And so we’re now at the place where Azure Service Bus, Apache…

ActiveMQ, Artemis, and Apache Cupid, these three brokers share literally the same code, the same client provider for JMS, which is fantastic from an interoperability perspective because JMS is terrifyingly bad in terms of an API feature set.

because it has a lot of weird things in the corners. Like the overall model is kind of okay, but once you kind of go and niche around the corners, then it’s awful. But so we did all of that work once and not everybody had to do it again. And so that’s a great achievement. And of course it sets us up, including now RabbitMQ implementing the APP window completely, that we now get to a place where this entire enterprise messaging space can basically move on and we can go and create a single set of clients.

for each language and then just compete on quality and scale rather than competing on API this and that and this and that because we all are now in the place where we have interoperability. So big story. And if I may then add another standard, another thing we did and that’s something we did in the CNCF with 40 companies including IBM.

is many more, is CloudEvents. And CloudEvents as a way to standardize what an event is, independent of what protocol it travels over. Because what we see more and more is that an event is being emitted from a device over MQTT.

It just lands in the cloud and then it’s being forwarded into a Kafka broker. It then is being picked up and is being forwarded into a queue for transactional processing. So it kind of makes all these hops. Of course, if you are protocol dependent.

That’s a problem because you’re kind of losing information. And if you don’t have a well understood way of mapping these events, then, you know, this implementation is going to be this. This is the implementation that’s going to be this. And so what we did is build a metadata model and tying back to what we said initially, what Andy said is like these web services standards have some good effects. We have been referencing SOAP.

many times in those discussions about CloudEvents as an example for things that we should not do. And amongst them was we didn’t want to invent a new protocol. We didn’t want to go and abstract the protocols by what we said. We’re going to be citizens in your house. You’re going to do Kafka.

Andy Stanford-Clark (46:31)
Yep, good. We learned something there.

Clemens Vasters (46:49)
we’re gonna do Kafka, which is simply gonna go and map our properties onto whatever the Kafka properties are. And you want to use XML, go use XML. You wanna use JSON, you’re gonna use JSON. We’re gonna map our models onto whatever the encodings are that you have. And it’s making CloudEvents more and more popular. So it’s trying to foster a proper ability by simply creating a convention. one of the instrumental people there has been Doug Davis who started at IBM then was Microsoft.

as the chair of that group. And we’ve been doing, and with lots and lots of contributors, we’ve doing great work. And a follow-on to this is the work that we’re currently doing in xRegistry around metadata, holistic metadata model about messaging and kind of formatting streams, et cetera. So we do as a principle, all of the wire protocols that we do in the Microsoft messaging services are all based on open standards. And what I’m super happy about is that in the messaging space,

we have been really disciplined, I have to say, around how many protocols there are. Because we could be having 20 standards, but we really only have like five that are complementing each other pretty well. we have MQTT is the choice for attaching devices and low bandwidth devices and for kind of this, this, you know, open namespace pops up

model. We have AMQP as the number one choice for transactional messaging. then we have Kafka has become kind of a go-to, I mean not a formal standard, but has become the go-to API where we now have lots of competing implementations on top of the same API standard that also kind of creates a level of inertia in the Kafka project.

for better or worse, that is now effectively becoming something that you can go and lean on also with many competing app implementations that use that as the standard. And AMQP can also do that streaming work. So we’re in a good place, I think, in terms of open standards and open source.

Andy Stanford-Clark (49:04)
And you should probably just mention xkcd927 as well, is the one that we, if you’ve got enough standards and one of them doesn’t quite do what you want, then just create new standards.

Clemens Vasters (49:09)
Yes! Yes. And, and I think we’re pretty good about, yeah, we’re pretty good about not having that. the there’s, there are a few newcomers, you know, which are, but they’re doing kind of like NATS for instance, right? NATS is text only protocol.

Andy Stanford-Clark (49:17)
frequently reference that.

Clemens Vasters (49:33)
that is complementing all of that bigger set nicely because it’s very low, it’s very, very simple, right? If you need the simple, simple, simple, simple, simple thing, you can use NATS. But they’re all very nicely differentiated and there’s very little true overlap in the use cases. And I am very happy that we’re, we, and possibly also,

with some help from maybe from us, that we’re not looking at a space that is as fractured as maybe other places like the database space.

Andy Stanford-Clark (50:11)
Absolutely.

Kate Holterhoff (50:12)
All right, well, I’ve only touched on about a third of the questions that I’d hope we’d get to, and that doesn’t surprise me in the least, but we are about out of time. So before we go, I know, I know, yeah, this is a to be continued conversation for sure. Before we wrap up, though, are there any ways that folks can follow you on social media? Where do you typically post your musings? Do you have a blog? How can folks keep up with you both?

Andy Stanford-Clark (50:18)
kept me and Clemens together in a room.

I used to be on Twitter as @andysc but now I’m not because of what Elon does and I’m on Mastodon instead as @andysc

Clemens Vasters (50:50)
Yeah. I’m @clemensv on various forums. can probably most technical content I post on LinkedIn these days because yes, the old place that Andy left, I have not left yet, but I’m still, I’m about to maybe because it’s untenable. So I moved most of that technical content to LinkedIn. I’m always hoping that I can blog again or will blog again. It will be disciplined enough to do it. But if I do, then I will also link my posts there and you can go find me very easily on LinkedIn because my name is not ambiguous. So find me Clemens Vasters or @clemensv on most platforms.

Kate Holterhoff (51:31)
Fantastic. I’ve really enjoyed speaking with you. And again, my name is Kate Holterhoff, senior analyst at RedMonk. If you enjoyed this conversation, please like, subscribe, and review the MonkCast on your podcast platform of choice. If you are watching us on YouTube, please like, subscribe, and engage with us in the comments.