Last September I was at a Microsoft virtualisation event watching a presentation by Gartner’s Thomas Bittman when something struck me. Its been bothering me on and off ever since. Bittman’s pitch was solid, but it was pretty much identical to Microsoft’s, which was near identical to an IBM deck I had seen the week before.
The consensus can be summed up roughly like so:
Cloud Computing = Virtualisation+Automation+Software as a Service (SaaS)+SOA evolution
But if Gartner, IBM, and Microsoft all agree on something… that’s a risk factor. there is a danger in consensus. WS-*, WS-*, WS-*.
In the intervening few months I have seen little to disavow me of the idea that many in the industry are over-complicating things when it comes to Cloud Computing. I have spoken to any number of CTOs, general managers and representatives of major systems vendors about Amazon Web Services. They invariably say things like AWS are “too simple”, or even “don’t really contain much intellectual property”. That, in short, not only would Amazon’s AWS EC2, S3 and SQS be easily replicable but they don’t tackle the right problems; that is – the really tough problems.
Amazon’s initial cloud offerings are instance rather than fabric-based and as such don’t involve a change in the programming model or style. An app isn’t going to scale linearly just because its deployed to AWS. Of course you’d have to be high to believe competitor claims that their own approach does allow transparent scaling of any app. To hear competitors talk though, Amazon might as well have an army of trained monkeys in a data centre somewhere running around provisioning servers and storage. As we all know though monkey armies don’t scale, even if they do create the odd work of genius. 😉
I have heard a very similar line before: IBM systems executives dismissing VMware, for example, or Windows before it, or Oracle before that, whatever the new technology was that would be replaced as soon as the going got tough. AWS just isn’t enterprisey enough. If you talk to web developers today though most would agree Amazon is an essential tool. And web methods are coming to the enterprise.
Incidentally, there are many similarities in the emergence of AWS and VMware. Both were developer-led. Both were reactions to a lack of responsiveness of operations staffs to the needs of developers. Both were initially positioned (by enterprise vendors at least) as test and development frameworks, rather than production environments.
The Complexify Pattern at REST
While I am not of the opinion that Service Oriented Architecture (SOA) is dead, clearly as an industry we got carried away with the attempt to establish a specific, common set of standards interfaces and protocols, defined from the top down, for integrating all types of applications. Everything was to be transactional, secure, ready to roll back.
We should probably not forget that XML Web Services began as a reaction to Corba-generation enterprise middleware complexity. Simple Object Access Protocol (SOAP) was straightforward enough to get things done. But as the arabesques of the WS-I stack became ever more baroque, building layer upon layer on top of SOAP, simplicity was left well behind in the desire to have an answer to everything, to model every use case, to provide interfaces for all seasons and reasons.
While there is a place for Big Architecture in IT, it shouldn’t be a necessity. You wouldn’t use a combine harvester to mow a lawn. You wouldn’t use a space shuttle to get to the shops. You wouldn’t build an airport to land a helicopter. But as an industry we are awesome at complexifying things.
Some things work though because they are simple, which brings us to Representational State Transfer (REST) an alternative development model to XML Web Services. While enterprise architecture astronauts were delighting in complexity, REST was taking hold. REST treats everything as a resource stored at a uniform resource indicator (URI, or a web link to you or me), which can be manipulated with a limited but powerful set of verbs (Get, Put, Post and Delete).
REST is peer to peer in terms of contract and web as agreement. Unlike WS-* Web Services the interface doesn’t tell you everything about how it can be used. REST really only needs one standard- HTTP.
But what is the equivalent in Cloud? What is the REST of the Cloud?
That’s what I have been trying to work out. I had a pretty funny run in with IBM Tivoli CTO Alan Ganek at IBM Pulse earlier this week about the question of REST of Cloud. I tend to hold a number of positions in my head simultaneously as I try and work out my position on things. But Ganek kept catching me contradicting myself. Alan- i am still thinking… forgive me 😉 My problem is that I don’t know the answers, but I do know there is a simplicity at work in AWS, which we are going to lose as we build out enterprisey clouds. What do we need to retain though?
Realising Simplicity: WOOing WOA
It has been intriguing to watch the evolving positions of Burton Group and Gartner with respect to SOA. Both firms were aggressive proponents of SOA but have recently been adjusting their market narratives. Gartner increasingly positions SOA as not delivering on its promises, largely because of the calcifying impact of the WS-* stack, putting forward Nick Gall’s Web Oriented Architecture (WOA) as an alternative. Burton’s Anne Thomas Manes recently declared SOA dead, although to be fair her argument was somewhat more nuanced than the headline. Please go to those sources – I certainly can’t claim to speak on behalf of Burton or Gartner.
So what if we run with a the idea of WOA in the cloud context?
Cloud is to Enterprise Operations as WOA is to SOA. What is cloud if not web oriented operations (WOO)? The question though remains- what is the REST of Cloud? That is, the simpler model which we will come to realise is just as appropriate as the more complex versions we see in traditional enterprise computing. Its hard to know how long it will to see the rise and fall of cloud complexity, but Amazon will likely continue to dominate WOO.
Is AWS the Rest of Cloud, or am I guilty of reductionism? Do I give Amazon too much credit for driving the Cloud model? I would say Amazon is the canonical instance-based service, while Salesforce.com is the canonical Cloud fabric, with its associated Apex language (a Java for the cloud era).
If we are to keep it simple stupid (KISS) what can WOO learn from WOA? [blimey- can’t you talk in English? Ed.] Rather than finishing this post with a grand conclusion I would rather maintain this as an open thread.
Talking of open threads, I had to chuckle last night when I saw a tweet from @jmettreaux. It seems Stu Charlton had transcribed a quote on a video podcast in which I put my position rather more strongly than I have here. This is what I said:
“If you think of the post-SOA term, from Nick Gall… Web Oriented Architecture, clearly this is somewhat different from SOA, although there are some patterns common to both of them….. Is the cloud Web-Oriented Operations, or WOO? (We have WOA and WOO)… and what IBM is saying is definitely not WOO, it’s business as usual, it’s just about flexible delivery of application — all the stuff that is goodness, all the stuff that Tivoli has been talking about since 1995. That stuff all has value, but it’s not Cloud. Cloud involves difference. Business as usual, that’s just provisioning service, and automation and virtualization, which is all good, but… if I hear another person tell me that CLOUD = SOA + VIRTUALIZATION + AUTOMATION, I’m going to ignore them and rubbish the idea as much as I can.”
Well why not say what you really think James? I tend to be a strong believer in Joel’s Law of Leaky Abstractions. I don’t think you make things “simple” and “seamless to the end user” simply by adding more abstraction layers. Simplicity requires that you cut stuff out, that you use simple stuff. Simple needs to be inherent. And instance-clouds are certainly simple.
Yesterday IBM announced deployment to Amazon Machine Instances, on which more later, but a twitted question from Dennis Howlett about portability really got the wheels turning. Instance Clouds are portable. Fabric Clouds are not.
Is there a REST of the Cloud? You tell me.
disclosure: Before I sign off I just want to thank my office share pals at Bash Studio – Matt Patterson and James Stewart for inspiring this blog. The REST of the Cloud- they provided the title.
eric norlin says:
February 12, 2009 at 4:59 pm
james- great post — highlights a lot of the thinking we’re doing around Gluecon (www.gluecon.com).
stu says:
February 12, 2009 at 5:48 pm
sry for the name misspell, fixed 🙂
more thoughts to come..
Mark Masterson says:
February 12, 2009 at 6:12 pm
Well, as you surely know, I certainly agree with your essential premise here. Do you know the old Dijkstra essay about “GOTO considered harmful”? I’ve had the ghost of potential blog post kicking around in my head for several months now, along the lines of “PaaS considered harmful”. I increasingly think that locking oneself into (and accepting the limitations of) a thing like Force.com or GAE would be an insanely stupid thing for an enterprisey entity to do. The equation comes out differently for a startup, but for enterprisey, it’s just a Bad Idea ™.
As for the REST of the Cloud (is that like the Tao of Whatever?), you seem to think we don’t already know what it is. I think we do. And if we don’t, we’re certainly quite close to articulating it. It looks like this: http://highscalability.com/canonical-cloud-architecture
Sean Dague says:
February 12, 2009 at 6:28 pm
I think one of the critical distinctions between REST and WS-* is that REST is data oriented, and WS-* is function oriented. While either gets the job done, the data oriented approach is easier for people to come to grips with. It always reminds of the Linus Torvalds quote:
“I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”
Joe Gregorio says:
February 12, 2009 at 7:31 pm
“””
Some things work though because they are simple, which brings us to Representational State Transfer (REST) an alternative development model to XML Web Services.
“””
The better way to phrase that would be:
“””
… brings us to XML Web Services, an alternative development model to *the web*.
“””
James Thomas says:
February 12, 2009 at 11:07 pm
I always thought “Cloud Computing” was the “REST” version of Grid Computing.
Grid computing seemed to share many of the same principles of what we know term “Cloud Computing”.
Computing as a utility, ease of scaling in processing power or storage, heterogeneous systems, lives “out there in the ether”.
However, it seemed to be surrounded in ever increasing specifications and middleware, similar to WS-*, and never delivered on all the promises.
Stu Charlton says:
February 13, 2009 at 8:19 am
I am for simplicity that recognizes the natural complexity of a problem domain, and doesn’t introduce too much artificial complexity. And I fundamentally think that the freedom of “on demand”, “self service”, and “virtualization” are all driving up the natural complexity of IT, just like PCs did 30 years ago. There are moving parts now, and I don’t think you can eliminate them.
The “law of leaky abstractions” always seemed to me a restatement of Saltzer, Reed & Clark’s “End to End Argument” for system design, but with a misleading conclusion. The conclusion is not “abstractions are bad”, it’s that “abstractions are never perfect” and “prefer fewer”. The Web as we know it wouldn’t exist without the leaky abstraction known as TCP (or even IPv4 in a world of pervasive NAT).
Simplicity is in the eye of the beholder. Instance clouds are simple — for the developers that are the current users of them. But there are many that would prefer to consume services that are simple to them — email, calendering, CRM, maybe a billing system, etc. Must these be delivered as a PaaS black-box, or can they / should they be described in a way that ensures visibility to the natural complexity of such things?
So, we tend to need many layers of abstraction – what sucks is when we can’t throw one of those layers out easily without the whole edifice crumbling down. WS-* and SOAP started falling apart arguably because they tried to pass all networked computing interactions into the XML Infoset as the End of All Syntactic Representations (ignoring how much internet traffic is dedicated to porn. 😉
When we’re talking about Web Oriented Operations, or Web Oriented Architecture, to me, we’re looking at how people and clouds interact and share information. And that’s the “hard problem” – it’s not just instances a matter of standardizing an API, it’s about providing layers of abstraction that can be peeled back if need be. Or can be substituted if you don’t like your neighbor’s abstraction.
So far, the Web Architecture gives us this, if we use it properly. For example, I think IBM’s Jazz Architecture and OSLC is a great example of how a cloud architecture should evolve — a global hyperlinked, declarative set of descriptions for software development life-cycle collaboration tools aren’t much different from a global hyperlinked declarative data center, just that the latter is probably more complicated.
Mark Masterson says:
February 13, 2009 at 1:00 pm
@Stu: “Must these be delivered as a PaaS black-box, or can they / should they be described in a way that ensures visibility to the natural complexity of such things?”
Yeah, but there’s a big difference (as I’m sure you would agree) between *consuming* such services, and writing glue code (or full blown applications) in Apex on force.com or Python on GAE. And to the extent that I make assumptions, as a consumer of some such PaaS service, that’s where the “leakiness” of an abstraction manifests itself, doesn’t it?
The conversation tends to bog down, at this point, into obsessing over APIs, but for me, a big part of the attraction of REST is that it seems to be a holistic approach. If I do it right, and decompose my system / domain into resources that properly apportion state, and which are related to one another in a graph of links, then my decomposition (my architecture) *is* the surface area abstraction (ie. the “API”). Like many people, James focuses a bit too much on “the verbs” in his post, and not nearly enough on HATEAOS, and the implications for apportionment of state and the graph of links that accompany it. “If I do it right” is glib, of course, and I know as well as anyone that it’s hard work — arguably harder than more traditional ways of architecting systems, certainly different. But if I *do* happen to get it right, that obtains me a solution to “prefer fewer” abstractions, as you put it. Doesn’t it?
I agree with you that “on demand” and “self service” are important new elements that the architecture has to address, and I also agree that “elasticity” makes things (vastly) more complex. But I also think that if we project those new elements onto what we’ve come to understand about web architecture, then we get pretty darn close to the picture in Hoff’s post I linked to above. If you disagree, why?
Stu Charlton says:
February 13, 2009 at 4:00 pm
@Mark. Regarding PaaS, yes, that’s sort of what I was trying to say. When I consume this thing, if it makes assumptions that screw me over, I have few options.
In general, that seems to be the difference between enterprisey and SMBs: larger entities expect things to go wrong, but they tend to demand lots of options to mitigate their risk, others adapt to their reality, or just switch providers (because they’re smaller & nimbler).
Regarding REST, yes, I completely agree with you. “If I do it right”. The problem is that this can get bogged down if there are too many cooks in the room. I wrote a rather long report from the Cloud Computing Interop event that RedMonk’s Stephen O’Grady hosted with David Berlind a month back, related to this problem. One subtext I got out of that meeting was that there’s a 15+ year history of IT Services Management standards that are culture-colliding with with this push for “cloud standards” . Are they ALL bad over-complex things that we should chuck for a high-level API? Or should we help provide a bridge to Web Architecture that links the higher-level media types to the more detailed media types that, say, reflect what the DMTF CIM Schemas provide? That would certainly be “holistic”. To the point that it probably needs a serious look at RDF and Semantic Web. (Cue the joke about being able to control your cpu-fan RPM speed via a cloud API).
Regarding Hoff’s canonical cloud, I don’t think you can call this canonical unless you believe “Cloud = Architecture for Parallel Job Execution” (wouldn’t the analysts love THAT equation ;). Not all applications are decomposable in this way.
The other thing is that it seems to be a systems architecture, i.e. it’s a solution to a functional problem. Whereas there’s a real need for a “system of systems” architecture that solves a collaboration problem, i.e., what is in the space between services? IBM views “SOA” as the solution for the latter, though we know that’s pretty meaningless. The Web Architecture could be used as the holistic glue across things, but arguably shouldn’t be used at all system layers, since it has its drawbacks.
Stu Charlton says:
February 13, 2009 at 4:39 pm
A footnote to my last response:
Please don’t interpret that I’m advocating that we create some kind of uber/holistic standard that covers every possible element of clouds at all layers. I think there might be room for innovation in _products_ to do that kind of thing (CMDBs, etc.), but there’s little consensus to create a _standard_ of that nature.
What I am advocating that Web Architecture should be the way that various pieces of cloud metadata (standard or proprietary) are shared and linked, if we’re going to make Cloud “different” from SOA+AUTO PROVISIONING+VIRTUALIZATION.
Mark Masterson says:
February 13, 2009 at 7:51 pm
@Stu: “Regarding Hoff’s canonical cloud, I don’t think you can call this canonical unless you believe “Cloud = Architecture for Parallel Job Execution” (wouldn’t the analysts love THAT equation ;). Not all applications are decomposable in this way.”
Hmm. I see what you mean, but… Consider: what if I just change the wording a wee bit, and out of your text, obtain “Cloud = Architecture for Parallel Resource Resolution”. Because, where you (seem to) see a “job being executed”, I see an abstraction beneath it, where the “job” is just a resource resolution (or chunks of same). Now, you can then promptly accuse me of reading too much into that model, and I will immediately concede the point (I’ve had tuplespaces on the brain ever since reading Carriero + Gelernter, years ago). But, for the sake of argument, if you accept my interpretation for a moment, then…? Do you see what I mean?
Quote: “Whereas there’s a real need for a “system of systems” architecture that solves a collaboration problem, i.e., what is in the space between services?”
Yeah. Is there a real need for that, though? Can you quantify it for me? Let me put it another way: if we say that the model that Hoff’s “canonical cloud” represents is a pattern for the *implementation of services*, then the point you are raising here is “what lies between them”, correct?
So, why can’t I just respond with “whaddya mean, what lies between them? The same thing that lies between any two other URIs.” How would that response (minus the mild snark) be different from “Web Architecture should be the way that various pieces of cloud metadata (standard or proprietary) are shared and linked”?
I sense we are nearing the dangerous ground of debating the merits of a priori service contracts…
a work on process » Selected Saturday links says:
February 14, 2009 at 11:24 pm
[…] The REST of the Cloud […]
alexis says:
February 16, 2009 at 8:20 am
At the risk of appearing the glib latecomer…
REST is the “REST of The Cloud”.
Give us URIs and we can make use of REST/HTTP. As @Mark reminds us, the key point here is HATEOAS — “Hypermedia as the engine of application state”. In the case where “hypermedia” refers to content deliverable using HTTP, this means that *web application execution* works in the same way as *web content navigation*. It has been shown that this is quite a good way to provide web apps and web integration, over the internet.
It’s entirely reasonable to provide this model for cloud applications provided that every resource in the cloud can have a sensible URI that can be resolved through DNS. I’m not sure how easy that is yet — it is not by accident surely that @Stu mentions pervasive NAT. Unlike SaaS, applications provisioned in the cloud, whether instance- or fabric-based, may change location (eg for scale-out, failover). So, some address management is needed and it needs to work with DNS. This can be facilitated by the cloud provider, and/or by using other application protocols than HTTP for some work.
I appreciate that ‘REST’ may not have been meant literally by James et al. It’s about simplicity, right? @Mark talks about a simple reference point: a “canonical cloud architecture” from Hoff. Ironically this architecture is based on queues, which are hard to provide RESTful interfaces for without creating an abstraction leak. @Stu makes the point that any platform/architectural choice (Hoff’s or other PaaS) leads to commitments.. which may *simplify* the end user developer’s cognitive experience, while screwing them over in other ways (eg price/performance) that they cannot ‘get out of’.
So why do customers say things like Amazon is ‘too simple’? It’s not because they care about programming style, even if they share @Mark’s and @Stu’s concerns. It is because they can’t see how to migrate their existing (“complex”) infrastructure, maybe including 100s or 1,000s of applications, over to someone’s (“simple”) cloud. To do so would require change, and change is *always* complex, no matter how “simple” the destination.
Developer-driven cloud adoption? « CloudPundit: Massive-Scale Computing says:
February 17, 2009 at 6:22 pm
[…] Top Clicks internap.comspamhaus.org/ROKSO/l…en.wordpress.com/tag…cloudpundit.wordpres…redmonk.com/jgoverno…cis.poly.edu/~angela…vectortuts.com/desig…research.microsoft.c…en.wordpress.com/tag…ww1.distributionclou…gartner.com/DisplayD…gartner.com/DisplayD… […]
msiersema says:
February 25, 2009 at 1:33 pm
Some Cloud food for thought about REST http://is.gd/jj8k
This comment was originally posted on Twitter
Laurence Faux says:
February 26, 2009 at 7:14 pm
I think the idea of a regualr, simple and consistent cloud API is a tempting idea…and not for the first time; especially as we all get older and become more frightend about how complex this area is becoming…I certainly like to come up with simple explanantions for complex things. However like all things that start off simple they soon have extensions bolted on the side….so mixed mode solutions are a reality so why fight it….so having said that they perhaps consider this:
When I talk to a web app I use Get, Post etc;
When I talk to a Database I use SQL
and when I talk to a Queue I use JMS stuff
Within each of these mechanisms, clearly, the implementation detail varies and industry standards take over wrt data schemas.
My point is as simple as this. I see a place, in the cloud for REST but I also see it for every other such protocol it’s just a question of agreeing them; but hasn’t the industry being trying to unify major boundaries like this since the beginning of time….the cloud is just the next, in vogue battleground that highlights the fact that we have failed thus far…..
erikabele says:
March 4, 2009 at 4:57 pm
@Yeliz & @jimjag: LOL, “Cloud is to Enterprise Operations as WOA is to SOA”, in “The REST of The Cloud” by J. Governor — http://tr.im/h0Hm
This comment was originally posted on Twitter
William Vambenepe’s blog » Blog Archive » IT automation: the seven roads to management middleware says:
May 20, 2009 at 8:53 am
[…] We’ve been doing computer integration almost since the second piece of software was written. There are plenty of mechanisms available to do so. IT management is just another integration problem, so let’s present it in a way that allows us to use our favorite integration tools on it. That’s the driver behind the use of Web Services for management integration (e.g. WSDM/WS-Management): create interfaces to manageable resources so that existing middleware (mostly J2EE application servers, along with their WS stack) can be used to solve the “enterprise IT management” integration problems in a robust and reliable way. The same logic is behind the current wave of REST-based IT management efforts (see this presentation). REST is a good integration approach, so let’s turn IT resources into RESTful resources so we can apply this generic integration mechanism to enterprise IT management. Different tool, but same logical approach. Which is why they can be easily compared. […]
EMC and the Cloud, Or Le Nuage et Le Petit Prince says:
December 4, 2009 at 4:49 pm
[…] consensus now appears to be Cloud Computing = Virtualisation+Automation+Software as a Service (SaaS)+SOA […]