What exactly is WebSphere MQ – queue talk with Andy Piper

While at the IBM Impact Unconference, I talked with Andy Piper about WebSphere MQ and the range of products around queues. We discuss where queues are used, who’s using them, and what IBM is looking to do in the future.

You can also download the video directly, or subscribe to the RedMonk Media feed to get this and other videos downloaded automatically for you.

Transcript

Michael Coté: Well, hello everybody! Here we are at Impact 2011, the IBM WebSphere Conference. We are actually in a sub-conference,. We’re at the Unconference that the Impact Unconference — is that the official title of it, the Impact Unconference?

Andy Piper: Yeah, it’s that. Yeah, absolutely, Unconference.

Michael Coté: And as always, this is Michael Coté of RedMonk and I’ve got a guest with myself. You want to introduce yourself?

Andy Piper: Sure, I am Andy Piper. I work in Hursley Laboratory which is in the UK. I think my job title at the moment, because it kind of moves around a bit — well, it’s WebSphere Messaging Community Lead.

Michael Coté: All right. Basically, that’s MQ, it’s queue stuff, right?

Andy Piper: Yeah, absolutely.

Michael Coté: What falls under WebSphere Messaging?

Andy Piper: So this stuff — that stuff is originally invented and developed from Hursley which is where I sit. So yeah, whatever we’ve we got. We’ve got WebSphere MQ, which is queue stuff. We’ve got a family of products around that. So we’ve got the WebSphere MQ File Transfer Edition which enables you to move files over MQ.

We’ve got WebSphere MQ Low Latency Messaging for very high-speed messaging. We’ve got WebSphere MQ Telemetry which is about mobile and sensor networks. I am going to miss something, WebSphere MQ Advanced Message Security which is for encrypting messages and we’ve got the Message Broker, which is our Enterprise Service Bus.

Michael Coté: There’s this resurgence of queues in architectures. I am not even going to say enterprise architectures, because what’s interesting is that all sorts of architectures for software, they are sort of rediscovering, I guess, using a queue or to put it another way, having an event-driven architecture, a bus or something like that.

Andy Piper: And in fact by doing so, you can actually start to parallelize workload and that’s an amazing thought. Yeah, absolutely, the idea of feeding some work to a process over a period of time and not having to have it instantly respond, but knowing that it will be done.

Michael Coté: Having it be asynchronous, I guess.

Andy Piper: Yeah, and that word, again, we’ve been talking about in last couple of days. I’ve been a consultant and often people hear the word asynchronous, and they go, that sounds slow. It’s like no, it’s just the difference between what we are doing, having an immediate conversation and an immediate response or may get a very false response, but it’s not time dependent. That’s what it means; asynchronous is something that’s not time dependent.

Michael Coté: I feel like at some point asynchronous programming fell out of use. I don’t know if it fell out of favor, but it kind of — I guess, maybe it all came down to web applications which is a very synchronous sort of bus. A request comes in and — I am drawing a line here — a request completes, like you expect the client to be waiting.

Andy Piper: Well, it does, I mean that’s right. It basically times out and actually, the request will time out.

Michael Coté: Right, and then Ajax came along and A being Asynchronous if I remember and I guess that kind of started to change things around a little bit. So what exactly is a queue, like how does a queue operate?

Andy Piper: So I think if a queue as a bucket into which I can throw something and then take it back or something else, somebody else can take it out every now and then when they need it. So what queuing and Message Queue, MQ, lets us do is to say, here is a bunch of data I am creating and I am just going to put it over there and I may come and get it back later myself as a process or actually another process may take it from there.

It’s just a place to put pieces of work or pieces of data. The important thing to note is it’s not a database. It’s not somewhere you are shoving data to store; you are putting it there temporarily until somebody comes along and takes it away and does something else with it.

Michael Coté: And so does that imply that it’s sort of not searchable if you will?

Andy Piper: It’s not searchable in the same way as a database. What we can do is we can do things like assign a load of unique IDs and actually that’s exactly how message queuing works is that every message has a unique ID and we can go and find that ID and get it from a pile. That’s the extent to which it’s searchable, but you are not expecting to, do you like a full text.

Michael Coté: A bunch of queries.

Andy Piper: Yeah, full text index on a bunch of data.

Michael Coté: So why don’t you just use like a shared directory, I mean that would kind of the — that’s a bucket, right?

Andy Piper: It’s a bucket, which is very secure and stable and assured. So the goal is to say, let’s take away the needs to give stuff to a file system which could crash or become corrupt. So let’s give it to a thing which would put it into this queue for us and give it back to or just give it to something else.

And by the way, that queue may not be local to me; it may be over a network somewhere. So yes, you could put it in a shared directory in the sense of a network drive. But then, of course, you need the same file system technology.

Michael Coté: What we are getting at, correct me if I am wrong, is that if you use a directory then you are depending on the file system and the storage as part of your architecture rather than depending on a higher abstraction depending on its service.

Andy Piper: That’s right, exactly. And one of the — I’ll try not to do the chest-beating sales pitch. That’s not what I am here for at all

But one of the things about MQ and one of the reasons why it has been very widely adopted over the last 10, 15 years is because it does run on a large variety of platforms. It has provided the same API. You are not having to think, okay, I am on Windows now so I need to use the .NET API to get to my file system. I am on a Mac; I need to use C or Objective-C to get to my file system. It’s a totally different way of approach. It’s the same API on any platform that MQ exists.

You have got the ability, as I mentioned, to choose to send data over a network to some other queue. So it’s a parsing mechanism.

Michael Coté: And then inherent in networking, I mean, once you bring some network thing, there is security gets involved and actually, giving access.

Andy Piper: Security, SSL channels and things like that. So what we have in MQ is a concept of a channel between two queue managers who own all of the queues on that system. And then between them we have a channel and we can encrypt that tunnel using SSL.

So yeah, you have security. You have the ability — and this is another important aspect of MQ — is assurance. You want to just rely on the fact that you have something reliable moving the data for you.

Michael Coté: And in the case of MQ and maybe queues in general, is that assurance the same thing as sort of transactional integrity in a database or is it more assurance that your stuff won’t disappear or is it both?

Andy Piper: Yes.

Michael Coté: And what I mean by that is, in database terms, like a transactional thing, is that there will never be two states of the data that you are accessing.

Andy Piper: Yeah, absolutely! And there is another aspect, of course, which is, when you are talking about databases, you talk about two-phase commits and all that kind of stuff and the ability to say, I want to do these operations, and either the whole set of operations commits or fails. And actually I might want to do that with two databases and a file system.

Michael Coté: Right, like transferring money between two accounts is a classic example.

Andy Piper: Exactly! So MQ is very commonly used in those kinds of scenarios to say, I need to get this message from this queue, update those two database tables belonging to two different systems, and put this message to this queue, and do all of that within a transaction. So that’s what our Message Broker and our MQ technology does for you.

But yeah, you are absolutely right; the assurance is, I am giving my stuff to some other process to look after for me and I want to make sure that it arrives.

So when I get told locally by the queue manager that it’s got my data and put it and the put has succeeded, then I can rely on the assurance that my MQ network will get that data somewhere else, once and once only. That’s another important concept here. You are not getting duplication; otherwise, you might end up doing the same operation twice.

And then if something crashes, if my local queue manager or remote queue manager crashes and we have put that message in what we call a persistent state, because we can do things in memory as well, in being non-persistent, that we could recover to a known state, right?

Michael Coté: Yeah, yeah, yeah.

Andy Piper: Yeah, you don’t want to lose work, especially if you are updating bank accounts.

Michael Coté: You must know better than I do how old MQ is, but it has definitely been — it’s one of the older IBM software products?

Andy Piper: Yeah, it used to be called MQSeries, it’s about 15 years, I think, now.

Michael Coté: Right, right. So to give you the chance to not so much to be a salesman, but kind of say what that gets you, like the differentiation that MQ has, I mean, what is it about MQ that makes it something beyond, like an open-source ESB that you might get or an open-source queue?

Andy Piper: First of all, it gives you stability of function. So we have been around for a long time and actually the upgradeability has been really good as well. So you take different versions of MQ and they can talk to one another.

Michael Coté: So there is sort of cross-platform interoperability and you mentioned that earlier.

Andy Piper: Exactly, cross-platform and upgradeability. So that’s the first thing.

Michael Coté: Longevity of the technology.

Andy Piper: In terms of feature, we also are constantly working to improve performance. You’ve got the high performing system if you need one, if you need to tune it that way. You have got things like clustering which do let you distribute work over a number of different systems and actually it’s one of the things that we come back to around this rediscovery of the idea of queuing.

Did you hear that news about Twitter rewriting their search engine?

Michael Coté: Oh, right, right.

Andy Piper: And I was reading that they have actually now got a thing which dispatches your query to one of several engines so they can suddenly workload balance all these queries that are coming into their search engine. That’s what clustering does for you.

You can start to move work around really easily. You have got — oh gosh, you have got loads of things, security. You have got emergence of this new family of products in the last two or three years; the File Transfer, the Advanced Message Security, the Telemetry, these have all been added on to MQ, because the vision for MQ from an IBM perspective is universal messaging.

So the idea is that data — your data, anytime, anywhere, if you like over whatever is the appropriate quality of service to give you at the point that you are at. Are you in your transactional datacenter or are you in your mobile device? How fast do you need to go? How reliable does it need to be? How secure does it need to be?

Michael Coté: So you sort of customize the footprint or the profile down to the endpoint.

Andy Piper: In a way, and that’s particularly what Telemetry is actually. Telemetry is absolutely for smaller devices customized to the network capabilities available, that kind of thing.

Michael Coté: One thing I am always curious about with middleware — and I use that phrase tentatively here — is, would you say when some organization is using MQ, is it more of a middleware component in their software stack or is it a system that they are running? Does that distinction make sense?

Andy Piper: Yeah. Okay. So I would say it’s a middleware component really.

Michael Coté: Right. So it’s kind of — it’s not a service, it’s not a universal sort of service?

Andy Piper: No, I mean, services would usually themselves use MQ or run on MQ. So you program your service, your application may use MQ to do its interface to the world.

If you think about an application, we were talking about this in the IBM Conference this morning actually, and a guy was asking about files and how do I move away from the file driven batch to SOA services. And actually, people are always going to be sending files in some form or another. So you might be able to reduce the number of files coming in, but you are probably not going to eliminate them.

Michael Coté: Yeah, because files are just so comfortable.

Andy Piper: Well, they are, they are. And everybody has Notepad and we can just hack away. But more to the point —

Michael Coté: That’s how I submit my batch jobs, Notepad.

Andy Piper: Nice! But it’s just another interface. If you think of a program that takes input — and you mentioned shared directories –maybe, it could be reading files, it could be reading from queues, it could be receiving HTTP calls, it could be coming over a TCP. You’ve got a range of different ways. So you can see files and queues as different interfaces to your services.

Coming back to your question, therefore, I would say queuing is a component, if you like, middleware component, a feature, a capability of your system.

Michael Coté: Right. So it’s not something bigger than the applications that use it that you kind of integrate with. It’s more, you pull it in as part of your application.

Andy Piper: Yeah, I mean, one of the ways that people often talk about it is as plumbing. It’s the pipe work that goes underneath your house.

Michael Coté: Right, right. Having taken a kind of technology first approach, so I think a lot of people, including myself, until I was beaten over the head with it, like to hear things like batch job and jobs and queues, and frankly, it’s sort of like, I have no idea how that stuff is used, like what the business applications of that are.

And then like you are saying, people ask like, we have got these files transferred to us and what do we do with that? And it would be great to hear sort of like what — I am always hesitating because I want to come up with something that doesn’t sound like a stupid business term, but like what business services are queues being used for?

Andy Piper: I mean every one. I have been a consultant and I have worked with retail and finance and government and every system can conceptually use them and many companies and organizations and industries do use them.

Michael Coté: Yeah, I mean the need to transfer data between point A and point B or point A and points B, C, D, E, and F or whatever, every application needs that. But it seems like there is a certain class of applications that use queues a lot more than others. Like a lot of financial applications use queues to basically synchronize payments or transfers between accounts — things like that.

There are things that I think if you are writing this kind of application, you probably use a queue just without thinking about it. Whereas, as you were saying, someone like Twitter is not going to be using MQ.

Andy Piper: I mean, the other thing is — so web services. Web services are great. I know you guys of RedMonk are really big fans of the web services and all their XML stuff.

Michael Coté: Yeah.

Andy Piper: So HTTP is great as a transport for web services, except it’s synchronous. So if the other end isn’t listening or isn’t available or is too busy, then your call will time out and you will have to handle that.

So hey, what a great idea! If that guy is too busy because it’s got loads of incoming requests, then why not instead of having HTTP code, put a queue there. So you can do things like web services over JMS. So your calling application that’s invoking the service can just do that by sending a message somewhere. It doesn’t have to be synchronous. And then it can wait and come back later. So you have got things like the WS-Addressing Standard which lets you say, call me back on this address later when you are done.

Michael Coté: Right. And I am imagining, correct me if I am wrong, but situations like retail data might be nice for that. If you have like hundreds of stores that need to, for whatever reason, send their current inventory needs to someplace.

Andy Piper: Well, I mean that kind of thing is also trickle feed. The idea of trickle feed is kind of very powerful in retail, that you just want to — as people buy stuff, come through Best Buy and are going through transactions at the register, then those just maybe get fired or —

Michael Coté: So it’s a lot more real time than like 24-hour batches.

Andy Piper: It can be real time, absolutely. You don’t have to have all of those transactions batched up in the store and then overnight FTP them to home. You could just start to literally feed them.

If you go in the Apple Store, they will say, do you want me to send an email to you? And magically, they email it to you and yet, you can also go online and see what you have purchased.

Michael Coté: So there is probably a queue involved in there somewhere.

Andy Piper: Who knows?

Michael Coté: I guess kind of the last question, so I mean, MQ is like 15 years old. I mean what are you guys thinking about doing in the next five years or whatever? I mean, like something that’s as old as MQ, at some point —

Andy Piper: Stop calling it old. It’s not old.

Michael Coté: Something that’s mature.

Andy Piper: It’s rich and mature, right? It’s very capable.

Michael Coté: There you go. Well said! Like there is always this question of —

Andy Piper: Let me grab my blue hat and blue collar.

Michael Coté: That’s right. There is always this question of, is there anything left to do?

Andy Piper: And there is, and there is.

Michael Coté: What are the things that are left to do in the queuing space?

Andy Piper: So there are emerging models and emerging programming patterns and ways of accessing data. So I mentioned our Telemetry protocol which is very optimized for IBM’s Smarter Planet story, the whole Smarter Planet. Let’s make use of this instrumented plantet we have built and start to receive data from those centers onto our data backbone, which is MQ.

Michael Coté: To trickle in perhaps as you were saying.

Andy Piper: Yeah, trickle in maybe or maybe you need to actually do kind of two-way communication. So that’s a nice and interesting model.

There are also things like some of the newer web push models, and I can’t talk to you much about specific plans, but we are very aware of things like the ability to push data to web browsers through new technologies.

Michael Coté: The sort of new endpoints that emerge that you want to integrate with.

Andy Piper: Right, absolutely. Mobile is obviously huge and it’s about not abandoning the existing data backbone customers, because they are absolutely core, but to extend the ability to interact with the queuing fabric from new programming models, new languages, and new protocols potentially.

So yeah, you can see Telemetry as an example of one of those, the ability to plug into your existing MQ infrastructure with a specialized protocol for the lightweight networks.

Michael Coté: Right, right, right.

Andy Piper: That’s where all of the cool stuff is happening. And also, some of the data encryption around compliance situations that people find themselves in, we have been looking at that. I mean, you have just got to look at the releases over the last few years that we have been doing and that’s where they are geared.

Usability is always important as well; the ability to more effectively manage things in a nice usable way. I think it’s a great product from a usability perspective anyway, but there’re always nicer things you can tweak.

Michael Coté: Well, great! Well, thanks for taking all this time to go over MQ stuff.

Andy Piper: You’re welcome.

Disclosure: IBM is a client and sponsored this video.