\n
While at the IBM Impact Unconference, I talked with Andy Piper<\/a> about WebSphere MQ and the range of products around queues. We discuss where queues are used, who’s using them, and what IBM is looking to do in the future.<\/p>\n You can also download the video directly<\/a>, or subscribe to the RedMonk Media feed<\/a> to get this and other videos downloaded automatically for you.<\/p>\n Michael Cot\u00e9:<\/strong> Well, hello everybody! Here we are at Impact 2011, the IBM WebSphere Conference. We are actually in a sub-conference,. We\u2019re at the Unconference that the Impact Unconference — is that the official title of it, the Impact Unconference?<\/p>\n Andy Piper:<\/strong> Yeah, it\u2019s that. Yeah, absolutely, Unconference.<\/p>\n Michael Cot\u00e9:<\/strong> And as always, this is Michael Cot\u00e9 of RedMonk and I\u2019ve got a guest with myself. You want to introduce yourself?<\/p>\n Andy Piper:<\/strong> Sure, I am Andy Piper. I work in Hursley Laboratory which is in the UK. I think my job title at the moment, because it kind of moves around a bit — well, it\u2019s WebSphere Messaging Community Lead.<\/p>\n Michael Cot\u00e9:<\/strong> All right. Basically, that\u2019s MQ, it\u2019s queue stuff, right?<\/p>\n Andy Piper:<\/strong> Yeah, absolutely.<\/p>\n Michael Cot\u00e9: <\/strong>What falls under WebSphere Messaging?<\/p>\n Andy Piper: <\/strong>So this stuff — that stuff is originally invented and developed from Hursley which is where I sit. So yeah, whatever we\u2019ve we got. We\u2019ve got WebSphere MQ, which is queue stuff. We\u2019ve got a family of products around that. So we\u2019ve got the WebSphere MQ File Transfer Edition which enables you to move files over MQ.<\/p>\n We\u2019ve got WebSphere MQ Low Latency Messaging for very high-speed messaging. We\u2019ve got WebSphere MQ Telemetry which is about mobile and sensor networks. I am going to miss something, WebSphere MQ Advanced Message Security which is for encrypting messages and we\u2019ve got the Message Broker, which is our Enterprise Service Bus.<\/p>\n Michael Cot\u00e9:<\/strong> There\u2019s this resurgence of queues in architectures. I am not even going to say enterprise architectures, because what’s interesting is that all sorts of architectures for software, they are sort of rediscovering, I guess, using a queue or to put it another way, having an event-driven architecture, a bus or something like that.<\/p>\n Andy Piper:<\/strong> And in fact by doing so, you can actually start to parallelize workload and that\u2019s an amazing thought. Yeah, absolutely, the idea of feeding some work to a process over a period of time and not having to have it instantly respond, but knowing that it will be done.<\/p>\n Michael Cot\u00e9:<\/strong> Having it be asynchronous, I guess.<\/p>\n Andy Piper:<\/strong> Yeah, and that word, again, we\u2019ve been talking about in last couple of days. I\u2019ve been a consultant and often people hear the word asynchronous, and they go, that sounds slow. It\u2019s like no, it\u2019s just the difference between what we are doing, having an immediate conversation and an immediate response or may get a very false response, but it\u2019s not time dependent. That\u2019s what it means; asynchronous is something that\u2019s not time dependent.<\/p>\n Michael Cot\u00e9: <\/strong>I feel like at some point asynchronous programming fell out of use. I don\u2019t know if it fell out of favor, but it kind of — I guess, maybe it all came down to web applications which is a very synchronous sort of bus. A request comes in and — I am drawing a line here — a request completes, like you expect the client to be waiting.<\/p>\n Andy Piper: <\/strong>Well, it does, I mean that\u2019s right. It basically times out and actually, the request will time out.<\/p>\n Michael Cot\u00e9:<\/strong> Right, and then Ajax came along and A being Asynchronous if I remember and I guess that kind of started to change things around a little bit. So what exactly is a queue, like how does a queue operate?<\/p>\n Andy Piper:<\/strong> So I think if a queue as a bucket into which I can throw something and then take it back or something else, somebody else can take it out every now and then when they need it. So what queuing and Message Queue, MQ, lets us do is to say, here is a bunch of data I am creating and I am just going to put it over there and I may come and get it back later myself as a process or actually another process may take it from there.<\/p>\n It\u2019s just a place to put pieces of work or pieces of data. The important thing to note is it\u2019s not a database. It\u2019s not somewhere you are shoving data to store; you are putting it there temporarily until somebody comes along and takes it away and does something else with it.<\/p>\n Michael Cot\u00e9:<\/strong> And so does that imply that it\u2019s sort of not searchable if you will?<\/p>\n Andy Piper:<\/strong> It\u2019s not searchable in the same way as a database. What we can do is we can do things like assign a load of unique IDs and actually that\u2019s exactly how message queuing works is that every message has a unique ID and we can go and find that ID and get it from a pile. That\u2019s the extent to which it\u2019s searchable, but you are not expecting to, do you like a full text.<\/p>\n Michael Cot\u00e9:<\/strong> A bunch of queries.<\/p>\n Andy Piper:<\/strong> Yeah, full text index on a bunch of data.<\/p>\n Michael Cot\u00e9:<\/strong> So why don\u2019t you just use like a shared directory, I mean that would kind of the — that\u2019s a bucket, right?<\/p>\n Andy Piper:<\/strong> It\u2019s a bucket, which is very secure and stable and assured. So the goal is to say, let’s take away the needs to give stuff to a file system which could crash or become corrupt. So let’s give it to a thing which would put it into this queue for us and give it back to or just give it to something else.<\/p>\n And by the way, that queue may not be local to me; it may be over a network somewhere. So yes, you could put it in a shared directory in the sense of a network drive. But then, of course, you need the same file system technology.<\/p>\n Michael Cot\u00e9:<\/strong> What we are getting at, correct me if I am wrong, is that if you use a directory then you are depending on the file system and the storage as part of your architecture rather than depending on a higher abstraction depending on its service.<\/p>\n Andy Piper:<\/strong> That\u2019s right, exactly. And one of the — I\u2019ll try not to do the chest-beating sales pitch. That\u2019s not what I am here for at all<\/p>\n But one of the things about MQ and one of the reasons why it has been very widely adopted over the last 10, 15 years is because it does run on a large variety of platforms. It has provided the same API. You are not having to think, okay, I am on Windows now so I need to use the .NET API to get to my file system. I am on a Mac; I need to use C or Objective-C to get to my file system. It\u2019s a totally different way of approach. It\u2019s the same API on any platform that MQ exists.<\/p>\n You have got the ability, as I mentioned, to choose to send data over a network to some other queue. So it\u2019s a parsing mechanism.<\/p>\n Michael Cot\u00e9: <\/strong>And then inherent in networking, I mean, once you bring some network thing, there is security gets involved and actually, giving access.<\/p>\n Andy Piper:<\/strong> Security, SSL channels and things like that. So what we have in MQ is a concept of a channel between two queue managers who own all of the queues on that system. And then between them we have a channel and we can encrypt that tunnel using SSL.<\/p>\n So yeah, you have security. You have the ability — and this is another important aspect of MQ — is assurance. You want to just rely on the fact that you have something reliable moving the data for you.<\/p>\n Michael Cot\u00e9:<\/strong> And in the case of MQ and maybe queues in general, is that assurance the same thing as sort of transactional integrity in a database or is it more assurance that your stuff won\u2019t disappear or is it both?<\/p>\n Andy Piper:<\/strong> Yes.<\/p>\n Michael Cot\u00e9: <\/strong>And what I mean by that is, in database terms, like a transactional thing, is that there will never be two states of the data that you are accessing.<\/p>\n Andy Piper:<\/strong> Yeah, absolutely! And there is another aspect, of course, which is, when you are talking about databases, you talk about two-phase commits and all that kind of stuff and the ability to say, I want to do these operations, and either the whole set of operations commits or fails. And actually I might want to do that with two databases and a file system.<\/p>\n Michael Cot\u00e9:<\/strong> Right, like transferring money between two accounts is a classic example.<\/p>\n Andy Piper:<\/strong> Exactly! So MQ is very commonly used in those kinds of scenarios to say, I need to get this message from this queue, update those two database tables belonging to two different systems, and put this message to this queue, and do all of that within a transaction. So that\u2019s what our Message Broker and our MQ technology does for you.<\/p>\n But yeah, you are absolutely right; the assurance is, I am giving my stuff to some other process to look after for me and I want to make sure that it arrives.<\/p>\n So when I get told locally by the queue manager that it\u2019s got my data and put it and the put has succeeded, then I can rely on the assurance that my MQ network will get that data somewhere else, once and once only. That\u2019s another important concept here. You are not getting duplication; otherwise, you might end up doing the same operation twice.<\/p>\n And then if something crashes, if my local queue manager or remote queue manager crashes and we have put that message in what we call a persistent state, because we can do things in memory as well, in being non-persistent, that we could recover to a known state, right?<\/p>\n Michael Cot\u00e9:<\/strong> Yeah, yeah, yeah.<\/p>\n Andy Piper:<\/strong> Yeah, you don\u2019t want to lose work, especially if you are updating bank accounts.<\/p>\n Michael Cot\u00e9:<\/strong> You must know better than I do how old MQ is, but it has definitely been — it\u2019s one of the older IBM software products?<\/p>\n Andy Piper:<\/strong> Yeah, it used to be called MQSeries, it\u2019s about 15 years, I think, now.<\/p>\n Michael Cot\u00e9:<\/strong> Right, right. So to give you the chance to not so much to be a salesman, but kind of say what that gets you, like the differentiation that MQ has, I mean, what is it about MQ that makes it something beyond, like an open-source ESB that you might get or an open-source queue?<\/p>\n Andy Piper: <\/strong>First of all, it gives you stability of function. So we have been around for a long time and actually the upgradeability has been really good as well. So you take different versions of MQ and they can talk to one another.<\/p>\n Michael Cot\u00e9:<\/strong> So there is sort of cross-platform interoperability and you mentioned that earlier.<\/p>\n Andy Piper:<\/strong> Exactly, cross-platform and upgradeability. So that\u2019s the first thing.<\/p>\n Michael Cot\u00e9:<\/strong> Longevity of the technology.<\/p>\n Andy Piper: <\/strong>In terms of feature, we also are constantly working to improve performance. You\u2019ve got the high performing system if you need one, if you need to tune it that way. You have got things like clustering which do let you distribute work over a number of different systems and actually it\u2019s one of the things that we come back to around this rediscovery of the idea of queuing.<\/p>\n Did you hear that news about Twitter rewriting their search engine?<\/p>\n Michael Cot\u00e9:<\/strong> Oh, right, right.<\/p>\n Andy Piper:<\/strong> And I was reading that they have actually now got a thing which dispatches your query to one of several engines so they can suddenly workload balance all these queries that are coming into their search engine. That\u2019s what clustering does for you.<\/p>\n You can start to move work around really easily. You have got — oh gosh, you have got loads of things, security. You have got emergence of this new family of products in the last two or three years; the File Transfer, the Advanced Message Security, the Telemetry, these have all been added on to MQ, because the vision for MQ from an IBM perspective is universal messaging.<\/p>\n So the idea is that data — your data, anytime, anywhere, if you like over whatever is the appropriate quality of service to give you at the point that you are at. Are you in your transactional datacenter or are you in your mobile device? How fast do you need to go? How reliable does it need to be? How secure does it need to be?<\/p>\n Michael Cot\u00e9:<\/strong> So you sort of customize the footprint or the profile down to the endpoint.<\/p>\n Andy Piper:<\/strong> In a way, and that\u2019s particularly what Telemetry is actually. Telemetry is absolutely for smaller devices customized to the network capabilities available, that kind of thing.<\/p>\n Michael Cot\u00e9:<\/strong> One thing I am always curious about with middleware — and I use that phrase tentatively here — is, would you say when some organization is using MQ, is it more of a middleware component in their software stack or is it a system that they are running? Does that distinction make sense?<\/p>\n Andy Piper: <\/strong>Yeah. Okay. So I would say it\u2019s a middleware component really.<\/p>\n Michael Cot\u00e9:<\/strong> Right. So it\u2019s kind of — it\u2019s not a service, it\u2019s not a universal sort of service?<\/p>\n Andy Piper:<\/strong> No, I mean, services would usually themselves use MQ or run on MQ. So you program your service, your application may use MQ to do its interface to the world.<\/p>\n If you think about an application, we were talking about this in the IBM Conference this morning actually, and a guy was asking about files and how do I move away from the file driven batch to SOA services. And actually, people are always going to be sending files in some form or another. So you might be able to reduce the number of files coming in, but you are probably not going to eliminate them.<\/p>\n Michael Cot\u00e9: <\/strong>Yeah, because files are just so comfortable.<\/p>\n Andy Piper:<\/strong> Well, they are, they are. And everybody has Notepad and we can just hack away. But more to the point —<\/p>\n Michael Cot\u00e9:<\/strong> That\u2019s how I submit my batch jobs, Notepad.<\/p>\n Andy Piper:<\/strong> Nice! But it\u2019s just another interface. If you think of a program that takes input — and you mentioned shared directories –maybe, it could be reading files, it could be reading from queues, it could be receiving HTTP calls, it could be coming over a TCP. You\u2019ve got a range of different ways. So you can see files and queues as different interfaces to your services.<\/p>\n Coming back to your question, therefore, I would say queuing is a component, if you like, middleware component, a feature, a capability of your system.<\/p>\n Michael Cot\u00e9: <\/strong>Right. So it\u2019s not something bigger than the applications that use it that you kind of integrate with. It\u2019s more, you pull it in as part of your application.<\/p>\n Andy Piper:<\/strong> Yeah, I mean, one of the ways that people often talk about it is as plumbing. It\u2019s the pipe work that goes underneath your house.<\/p>\n Michael Cot\u00e9:<\/strong> Right, right. Having taken a kind of technology first approach, so I think a lot of people, including myself, until I was beaten over the head with it, like to hear things like batch job and jobs and queues, and frankly, it\u2019s sort of like, I have no idea how that stuff is used, like what the business applications of that are.<\/p>\n And then like you are saying, people ask like, we have got these files transferred to us and what do we do with that? And it would be great to hear sort of like what — I am always hesitating because I want to come up with something that doesn\u2019t sound like a stupid business term, but like what business services are queues being used for?<\/p>\n Andy Piper:<\/strong> I mean every one. I have been a consultant and I have worked with retail and finance and government and every system can conceptually use them and many companies and organizations and industries do use them.<\/p>\n Michael Cot\u00e9:<\/strong> Yeah, I mean the need to transfer data between point A and point B or point A and points B, C, D, E, and F or whatever, every application needs that. But it seems like there is a certain class of applications that use queues a lot more than others. Like a lot of financial applications use queues to basically synchronize payments or transfers between accounts — things like that.<\/p>\n There are things that I think if you are writing this kind of application, you probably use a queue just without thinking about it. Whereas, as you were saying, someone like Twitter is not going to be using MQ.<\/p>\n Andy Piper:<\/strong> I mean, the other thing is — so web services. Web services are great. I know you guys of RedMonk are really big fans of the web services and all their XML stuff.<\/p>\n Michael Cot\u00e9:<\/strong> Yeah.<\/p>\n Andy Piper:<\/strong> So HTTP is great as a transport for web services, except it\u2019s synchronous. So if the other end isn\u2019t listening or isn\u2019t available or is too busy, then your call will time out and you will have to handle that.<\/p>\n So hey, what a great idea! If that guy is too busy because it\u2019s got loads of incoming requests, then why not instead of having HTTP code, put a queue there. So you can do things like web services over JMS. So your calling application that\u2019s invoking the service can just do that by sending a message somewhere. It doesn\u2019t have to be synchronous. And then it can wait and come back later. So you have got things like the WS-Addressing Standard which lets you say, call me back on this address later when you are done.<\/p>\n Michael Cot\u00e9:<\/strong> Right. And I am imagining, correct me if I am wrong, but situations like retail data might be nice for that. If you have like hundreds of stores that need to, for whatever reason, send their current inventory needs to someplace.<\/p>\n Andy Piper: <\/strong>Well, I mean that kind of thing is also trickle feed. The idea of trickle feed is kind of very powerful in retail, that you just want to — as people buy stuff, come through Best Buy and are going through transactions at the register, then those just maybe get fired or —<\/p>\n Michael Cot\u00e9:<\/strong> So it\u2019s a lot more real time than like 24-hour batches.<\/p>\n Andy Piper: <\/strong>It can be real time, absolutely. You don\u2019t have to have all of those transactions batched up in the store and then overnight FTP them to home. You could just start to literally feed them.<\/p>\n If you go in the Apple Store, they will say, do you want me to send an email to you? And magically, they email it to you and yet, you can also go online and see what you have purchased.<\/p>\n Michael Cot\u00e9:<\/strong> So there is probably a queue involved in there somewhere.<\/p>\n Andy Piper:<\/strong> Who knows?<\/p>\n Michael Cot\u00e9:<\/strong> I guess kind of the last question, so I mean, MQ is like 15 years old. I mean what are you guys thinking about doing in the next five years or whatever? I mean, like something that\u2019s as old as MQ, at some point —<\/p>\n Andy Piper:<\/strong> Stop calling it old. It\u2019s not old.<\/p>\n Michael Cot\u00e9:<\/strong> Something that\u2019s mature.<\/p>\n Andy Piper:<\/strong> It\u2019s rich and mature, right? It\u2019s very capable.<\/p>\n Michael Cot\u00e9: <\/strong>There you go. Well said! Like there is always this question of —<\/p>\n Andy Piper: <\/strong>Let me grab my blue hat and blue collar.<\/p>\n Michael Cot\u00e9: <\/strong>That\u2019s right. There is always this question of, is there anything left to do?<\/p>\n Andy Piper: <\/strong>And there is, and there is.<\/p>\n Michael Cot\u00e9:<\/strong> What are the things that are left to do in the queuing space?<\/p>\n Andy Piper:<\/strong> So there are emerging models and emerging programming patterns and ways of accessing data. So I mentioned our Telemetry protocol which is very optimized for IBM\u2019s Smarter Planet story, the whole Smarter Planet. Let\u2019s make use of this instrumented plantet we have built and start to receive data from those centers onto our data backbone, which is MQ.<\/p>\n Michael Cot\u00e9:<\/strong> To trickle in perhaps as you were saying.<\/p>\n Andy Piper:<\/strong> Yeah, trickle in maybe or maybe you need to actually do kind of two-way communication. So that\u2019s a nice and interesting model.<\/p>\n There are also things like some of the newer web push models, and I can\u2019t talk to you much about specific plans, but we are very aware of things like the ability to push data to web browsers through new technologies.<\/p>\n Michael Cot\u00e9:<\/strong> The sort of new endpoints that emerge that you want to integrate with.<\/p>\n Andy Piper:<\/strong> Right, absolutely. Mobile is obviously huge and it\u2019s about not abandoning the existing data backbone customers, because they are absolutely core, but to extend the ability to interact with the queuing fabric from new programming models, new languages, and new protocols potentially.<\/p>\n So yeah, you can see Telemetry as an example of one of those, the ability to plug into your existing MQ infrastructure with a specialized protocol for the lightweight networks.<\/p>\n Michael Cot\u00e9: <\/strong>Right, right, right.<\/p>\n Andy Piper:<\/strong> That\u2019s where all of the cool stuff is happening. And also, some of the data encryption around compliance situations that people find themselves in, we have been looking at that. I mean, you have just got to look at the releases over the last few years that we have been doing and that\u2019s where they are geared.<\/p>\n Usability is always important as well; the ability to more effectively manage things in a nice usable way. I think it\u2019s a great product from a usability perspective anyway, but there\u2019re always nicer things you can tweak.<\/p>\n Michael Cot\u00e9: <\/strong>Well, great! Well, thanks for taking all this time to go over MQ stuff.<\/p>\n Andy Piper:<\/strong> You\u2019re welcome.<\/p>\n Disclosure:<\/strong> IBM is a client and sponsored this video.<\/p>\n","protected":false},"excerpt":{"rendered":" While at the IBM Impact Unconference, I talked with Andy Piper about WebSphere MQ and the range of products around queues. We discuss where queues are used, who’s using them, and what IBM is looking to do in the future. You can also download the video directly, or subscribe to the RedMonk Media feed to […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[11,12,23,27],"tags":[86,349,350,367,516,629,791,832],"class_list":["post-6606","post","type-post","status-publish","format-standard","hentry","category-conferences","category-enterprise-software","category-programming","category-redmonktv","tag-andy-piper","tag-ibm","tag-ibmimpact","tag-impact","tag-mq","tag-queue","tag-unconference","tag-websphere"],"acf":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/posts\/6606","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/comments?post=6606"}],"version-history":[{"count":0,"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/posts\/6606\/revisions"}],"wp:attachment":[{"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/media?parent=6606"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/categories?post=6606"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redmonk.com\/cote\/wp-json\/wp\/v2\/tags?post=6606"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Transcript<\/h2>\n