dev/ops in action, with Ernest Mueller – IT Management & Cloud Podcast #085

At SXSW this year, we finally got a chance to talk with Ernest Mueller about the dev/ops work he’s been up to at National Instruments. He’s written about the topic over at his blog TheAgileAdmin.com and spoken about various aspects – it was great to get our own chance to grill him about the topic.

Download the episode directly right here, subscribe to the feed in iTunes or other podcatcher to have episodes downloaded automatically, or just click play below to listen to it right here:

Transcript

As usual with these un-sponsored episodes, I haven’t spent time to clean up the transcript. If you see us saying something crazy, check the original audio first. There are time-codes where there were transcription problems.

Michael Coté: Well, hello everybody. This is another special South by Southwest edition of the IT Management Cloud Podcast. As always John Willis and I are here. We’ve got a guest with us, a local Austinite and why don’t you introduce yourself?

Ernest Mueller: Well hello, I’m Ernest Mueller. I’m a Web Systems Architect for National Instruments here in town.

Michael Coté: What do these National Instruments guys do? I know they are sort of like Goldman Sachs best place to work or something like that. No, no not Goldman Sachs, whoever does that.

Ernest Mueller: Forbes.

Michael Coté: There you go.

Ernest Mueller: That’s right. Well we make both hardware and software for data acquisition, industrial control, kind of test and measurement. One of our big, our big software product is called LabVIEW which kind of most engineering students over the last however many years have used in college. And some of the trivial uses of our dark powers are the LEGO MINDSTORM NXT, a control block that makes the LEGO Robots, we do the software that the kids used to program.

Michael Coté: Okay, so you do controlling stuff and measuring the stuff I guess like –

Ernest Mueller: Exactly, exactly.

Michael Coté: And, I mean you are a technology company, but you are sort of like a old school technology company not like a software company.

Ernest Mueller: That’s right, that’s right, desktop software, and hardware, and both embedded hardware like few J stuff, and parts to put in your PC and dedicated chassis and then that sort of thing.

Michael Coté: And have you sort of worked there all of your professional life or?

Ernest Mueller: No, no, when I graduated long, long ago, I would work in FedEx Corporate IT for number of years. Then I worked for an internet startup in Memphis, Tennessee. And, then I moved up to Rhode Island for a little bit, did some consulting and then I got opportunity, moved back down here on the Texas —

Michael Coté: So before we get into the technical stuff, like you know few episodes ago I went off on a tangent about burgers and fanny packs. So, you know let me ask you about Rhode Island. So now maybe John has something quite sincere from that neck on the woods if you will. But the only impression I have on Rhode Island —

John Willis: Yeah, yeah, I mean yeah.

Michael Coté: The only question I have about Rhode Island is from this one episode of the Sopranos, where they have the cousins who live in Rhode Island. And, ever since then it’s stuck in my head that Rhode Island is a little weird. And, like so is Rhode Island kind of like weird?

Ernest Mueller: Rhode Island is a little weird they take inordinate pride in how many times their governor gets indicted. Of course, I guess we are not all that different down here. But, yeah it was an odd place, but good Italian food.

John Willis: We love the burgers, plenty good burgers there?

Ernest Mueller: No, not really.

Michael Coté: What’s your impression of Rhode Island?

John Willis: I don’t know; it’s a place that you cut through sometimes when we are going up North.

Michael Coté: All right, all right, so we got that.

Michael Coté: When I’m going in to office venue, I have to go through — but of course and you know that whole Nantucket Moby Dick, it’s a very odd scene, its one of the — that’s perhaps the best part of the whole book is the whole Rhode Island scene like what you are going to do. So I know John, you have — why don’t you launch and do it John?

John Willis: Yes, so I was thinking about this, so, you know, we’ve been following each other for a while here you’ve been in DevOps, since DevOps was put through but the —

Michael Coté: All right, go ahead.

John Willis: The, I guess I was thinking about just kind of introducing what are your thoughts about DevOps, but I guess I was thinking more about, since you do work for a large enterprise what does DevOps really mean, and with a spin from an enterprise view in your view?

Ernest Mueller: Well let’s see, from an enterprise view I don’t know that the over all view is any different than the normal DevOps view. It is somewhat more complicated to implement sometimes. But, the whole concept of bringing your development folks and your web operation folks together and actually embedding on a team together to deliver a product is really the core concept there. Scale becomes the issue, just like with Agile in general. In Agile methodology it sounds all simple, you got your small gang of developers, you embed the business on a — if you are in the DevOps you embed an operations person on them and you go, and if you have an eight person company that’s — it’s reasonably straight forward to understand how to do that.

Maybe you are dealing with 5000-person company like we are, the path to get there is little more oblique. And, to be honest we — the way we got our feet wet was like green fielding an internal team with both developers and operation staff and who are on it, to get knowledgeable enough in it that we could then perhaps figure out how to spread it to the rest of the organization. You have to do a small project first.

Michael Coté: You have do small projects first.

[0:05:02.6]

Michael Coté: Right, right.

John Willis: So, Michael thinks that we were all full of crap when you were to talk about culture, at the previous episode if you will. So like — well school about why DevOps, what actually in more like his point as we tend to say, like in 0:05:23 world we use culture as this kind 0:05:26 okay, I am going to stop talking now, we are going to throw the culture out there, like why is this — why do we 0:05:31 about kind of behavior and culture in DevOps?

Ernest Mueller: Well, because I think it’s the grease that makes the wheels turn right. There are many, many opportunities for friction in any organization and the more moving parts you have for example, in bigger organization more opportunities for friction you have.

If people approach a process from an attitude of collaboration, then things are probably going to work out all right. If they are approaching process being, kicking and screaming or hostilely or reluctantly, they will innovate ways to make it not work, and I think one of the difficulties especially with operation staff is historically we’ve been trained to accomplish certain goals. Those goals are availability, but unfortunately sometimes more importantly cost efficiency, and reduction of risk, and once you do that for a while that becomes part of your DNA.

Well, so if you just get thrown on to a team with the goal of let’s deliver shipping products, you really have to be bought into that from a cultural point of view or else you’ll just drag your feet and that’s the big, I mean having had to go through that transition myself as an operations person, I understand if it is — it is difficult that it’s not nothing and there was a interesting post early on OpsOps.

It’s like before DevOps we need OpsOps, because the nature of the burden that’s been put on IT infrastructure departments, it’s not like they’re all happy and collaborative with each other and hate developers, right. They pretty much have been siloed, and sub siloed, and thought to defend their sub silo and that’s a hard thing to get over it.

Michael Coté: Yeah, I mean that’s the point that I like to make with a lot of people is the moral lapse as I would call this. They are only the creation of their situation, I mean they’re doing what they’re been asked to do.

They are how they are because yeah they were made that way, people fall to that. Yeah, I mean I think, I think that’s one thing that John got me thinking about when we are talking about the, my hatred of the cultural argument is that it is like thinking about, and you are kind of touching on this, this is thinking about instead of doing service management, doing a product management, kind of having that by and to like own the stuff.

I think that is, it’s like, more so than just like subtle trickery with words. It does like change what you think about as an ops person, what it is you are doing like —

Ernest Mueller: Absolutely.

Michael Coté: You are not just — you are not just satisfying SLAs and up time, you are not keep something up and running, you are looking at how I can make this product better, this thing better than on running.

Ernest Mueller: That’s a very good point because we had experimented with ITIL and then eventually visible ops, and had started to try to bring that sort of structured service management approach to our work when that 00:08:51 in our R&D department, run a team that’s developing SAAS products, but I was in our IT department heading up our systems side of our corporate website and we tried to push the service management aspect. We have a lot of friends in the business, the Director of Web Marketing was very supportive of setting FLAs, and figuring out how to accomplish them.

But, in the end we only achieved success to a certain degree despite the fact that, we have a lot of great people working there in all the different sectors, and it became apparent to me, our team was always, we were all respected, but we were always the bottleneck. Eventually I kind of got sick of being the bottleneck, I said, look there has to be a different way of doing things, we have optimized our processes as much as we can.

We developed the systems development process that we would use to engage development teams, very waterfall.

[00:10:02]

But brought together many years with expertise and streamlining it, but even so we couldn’t achieve the pace of delivery that the business wanted and then when you stepped back and looked at it was not unreasonable.

Michael Coté: And I mean, can we go into like why, if you’re taking a service management type of approach where and correct me where I’m kind of wrong, where you’re sort of focused on providing some SOAs that you have, and providing the infrastructure that something runs on. You’re kind of providing, I don’t know, the infrastructure stuff runs on, I’m repeating myself. But I mean, what would happen that would cause you guys to be a bottleneck like because you’re doing like ITIL or Visible Ops — a service oriented —

Ernest Mueller: Well, yes, I don’t think it’s, I don’t think we were a bottleneck, because we were doing ITIL and Visible Ops is just those things were not sufficient to stop us from being the bottleneck, because we had — we have like a 100 programmers to a team of like six ops guys.

Michael Coté: So, it’s not that they were slowing things down. It was more than they weren’t sufficient to give you the tools to move at the speed the development wanted?

Ernest Mueller: That’s right, they absolutely held. Right, we had a great kind of internal initiative where we sat down with our senior development managers and the business managers and we put together these, kind of, core goals and SLAs.

It was great because it aligned a lot of the things like hearing the business people say that performance and uptime are two of the top five things they care about, right, helped us make that case to the development teams, and to other business teams that frequently were pushing for release even if they knew something was terribly wrong.

So, it was very helpful, but we just kept bumping up against a kind of glass ceiling where we just — we couldn’t hit that higher rate of speed that when you stepped back, and looked at the pace of delivery, you still said, hey that some just seems wrong there, why it does take that long.

Michael Coté: I guess, I mean it probably is actually a metric in various processes or whatever. But it’s not a metric you hear too about it. It’s the — I don’t know time to deliver new features —

Ernest Mueller: Well, absolutely.

Michael Coté: Or something, I don’t know what you’d call that.

Ernest Mueller: That was the source of one of the — kind of, more bitter arguments surrounding that whole effort. We came up with this kind of five killers that were important to the business was performance, availability, total cost of ownership, and then fairly we call that became proportion between new development and maintenance, and agility.

One of the senior development managers that kept trying to axe Agility, he is like well, how do you put a metric on that, how do you like — and I said, we go back and talk to the business director, I guarantee if we asked him which one of these five things was most important, he would say that one. So, even though we can’t come up with a sound metric for it, I don’t really care.

So that’s true, it’s hard because what — you start getting, you start going down the rabbit hole of feature points, lines of code, it’s all BS right and in the end you have to kind of wet your finger and stick it up to the air and say, so feel like the wind is moving fast enough, no it doesn’t.

Michael Coté: And have you come across people measuring something like that John like its sort of like, I mean I don’t know I would almost think it’s sort of like business guy wants something, never mind the scale of it necessarily, and is kind of like some measurement of how quickly relative to the scale of it that you can get into production.

John Willis: Well, I think if you start looking at like the companies that like are boasting their deploys per day. I mean that’s definitely, I mean that’s a sign of agility. I mean to some who aren’t — we talked about it in the flash podcast and some who are not ready to fulfill all the pieces of that that could be a disaster like, it’s just like, okay we’re going try to do 30 deploys a day, right. That ain’t going to work if you don’t have faster developing, if you don’t have some type of goal-oriented monitoring or just some, all the things implies that make that type of pipeline work.

But, I think that becomes — but I’ve been a big fan of the finger in the air, feel where the way wind is blowing. I’ve been saying it like we used to talk about clouds in the early days, and I’d say, well, it’s just a cloud, it’s just a cloud, I tell you. I can’t tell you what makes the cloud but I could tell you when I see one.

[00:15:00]

And I think that — I think that that’s the — like you will know you’re agile, you will know that your business is agile, when you’re agile, I mean it will be clear, right. I mean look at 00:15:13 you look at — and a 00:15:15 it goes into velocity of innovation. The stories where these companies can, like a marketing guy can get on a phone with the developer guys and say, I got a big deal right now, and then you can show these guys this feature and they can put it together on a couple of hours, and they launch it with the feature flag, the only that customer sees it. You know, that’s our job, right. I mean that’s our job, —

Michael Coté: Yeah, yeah. I mean that kind of scenario is — it always drive developers kind of by rate, but I think that’s because kind of joins back into the thread of conversation, it’s because traditionally that’s a very dangerous thing to do, to have the sales person developer connection.

John Willis: Yeah, right traditionally, but, I mean, this is all about like breaking down a lot of tradition —

Michael Coté: Yeah, exactly.

John Willis: Like doing 30 deploys a day gets people sick, right 00:16:13 when you here the first time. But, having a sales guy — a marketing guy call a development guy and say I want this, that like breaks all the rules of traditional. But in a new world, it’s pretty freaking full — I was telling Ernest earlier today, I was in this trading company recently, where they have the developer’s and the trader’s path, they sit next to each other all day long and a developer like he says, why did you do that, the trader says, well, normally I have to do three steps to get this. It is like hold on and it’s a new world, I mean I think that’s –

Ernest Mueller: That’s a good point. I think the problem that someone is faced with from an enterprise point of view is how much does that scale, right, because when you are — so many of the start ups are very narrowly focused and, they are being encouraged to be very narrowly focused as at a king of the, Lean: King of the Apps Show down — South by Southwest, people are going up and showing their apps and giving their spiel and getting fatigued.

Michael Coté: Like, this is an app that tracks the growth of a potato plant.

Ernest Mueller: Well, exactly and so,

Michael Coté: Not any plant, potato.

Ernest Mueller: So, one guy showed his app and Robert Scoble was like, Yeah, you need to take half of your features out. I said, well, okay, hold on. I could see he needs a simpler interface right but I have got 20 Apps on my phone to do one thing, that when I step back, I think why don’t I have one app that does all these things. How many Google maps mash ups do I need?

Michael Coté: If any thing the apps that I really like, I really want them to do a lot more, like something like Evernote, like I wish you would just — there is this great, this fun little app called Momento that you hook up to all your 00:17:59. It’s like pulls it all and you can see like, for this day you take to these places in first grade put this, took these photos and its fun, right. But like that’s kind of thing we were like, oh, I wish, I wish Evernote gets that and like stored it, like so I would like it to do a whole lot of stuff.

Then there is another example, there is this Instagram application. It’s fun, it’s like a fun way to take pictures and it’s kind of like I want that to integrate with everything. I don’t want it to be it just takes some pictures anyways.

Ernest Mueller: Well and that’s a — and that’s the dilemma so.

Michael Coté: It’s the individual enterprise requirement process. I wanted to do everything right now.

Ernest Mueller: Right, because, it’s just on our website, right. We have like 300 applications in production, right. It’s not we don’t do one thing; we do a lot of things and too much segmentation right, breeds chaos. In fact, that was one of the challenges that my web systems team faced, was the business team was siloed by, here is E-commerce, ECRM support community on down the line, and they would all do things differently, and so it’s hard to get together standards, and make things work. But on the other hand, I think corporate IT has gone too much the other way, right, there is always a balance between centralization, and standards, and innovation.

Michael Coté: Well, let me float this theory that I just was thinking this — as you were talking about this 00:19:28. It almost seems like, so in the past, let me over contextualize this as always, in the past enterprise, I mean like high scale and fast and performance and WebSphere MQ, and all those kind of stuffs, whereas nowadays that’s not the exclusive domain of enterprise, like Twitter does enterprise scale stuff with, and that is not enterprise. I mean whatever, and so it’s almost as if nowadays to the point of what you are saying enterprise kind of means complex.

Ernest Mueller: Absolutely.

Michael Coté: And lots of stuff.

[00:20:00]

Ernest Muller: That is a great insight. So, on our website we didn’t have more than four servers that did the same thing. Internally we might have had somewhere a group of ten servers that did the same thing but, that’s not our — but that’s not the problem we have, we don’t have the Google problem, the Flickr problem, when me and my colleagues go to Velocity, and hear about, oh, here is how we solved this problem for Facebook, like we are jealous that our problem would be that simple, right.

Michael Coté: Right, exactly. That’s kind of something I even thought about till you’re going over it. Is it is, it’s almost a cultural, a business culture problem of — the business doesn’t want to have a simple business, or can’t have a simple business, they have a complicated business.

Ernest Muller: Well, exactly like it, the 00:20:48 coding the scale presentation yesterday, they came out, admitted that they’ve made their application more monolithic so that they can support that higher rate of deploys per day.

They used to be service oriented and basically they’ve smashed it together, and said well, we’ll worry about that later. Then when you have a very large organization where you have you know Oracle ERP systems, and all these other things, it’s really the complexity which is the challenge. We still have the Lotus Notes users, and not because we love it for email, like for just for email be out there tomorrow and we’ll have something else then, but it’s all the little applications.

We have hundreds and hundreds and hundreds of little applications because Lotus Notes solved that pluming problem for you, it’s like, hey, I can write this little app. I can distribute it you know from here to Aachen, Germany to Hungary to Tokyo all the places we have offices, great done, right and still nothing has actually emerged to do that as well.

John Willis: Right, right, right. Yeah, I mean that is the reason just to stick with notes. Sure.

Michael Coté: Yeah, I mean that’s interesting — yes, please, that’d be great, thank you. Yeah, so what should we be talking about next John?

John Willis: Well I’m a big fan of your open source projects if you can or your stuff that you’ve been kind of contributing, you want to talk about that?

Michael Coté: Oh yeah, Let’s talk about technology.

Ernest Muller: Sure.

John Willis: Technology, yeah, sure.

Ernest Muller: Sure, well, so we are still working on the open sourcing part, but one of the things that we did when we started this new team to develop software’s as a service products for National Instruments, I made the decision early on that it need to be all Cloud based. The nature of the products that we were developing, for example, let’s say FPGA Compiled Cloud, they all needed to be multi-tenant, they all needed to be scaled very well, and in certain cases with fairly custom algorithms. We knew we needed it to happen quickly.

So we decided that we needed to write a Cloud based provisioning environment that would allow us to really hit that, both the velocity but also the flexibility that we needed.

Michael Coté: To be agile.

Ernest Muller: That’s right. That’s right because —

Michael Coté: Not to be agile, to have agility.

Ernest Muller: To have agility, it’s all different —

Michael Coté: Yeah, to add the L1.

Ernest Muller: Absolutely, because these products, we didn’t know which ones are going to go over well, not go over. We didn’t know how we were going to develop on this true, kind of, agile development story like it’s not like somebody came to us with a clear business case, and they wanted these things to do.

It’s like we’ve, I don’t want to have this thing and is going to run on the cloud, and let’s figure all that out. Right so like one of our major tasks in every sprint is going and grabbing the business guy by the 0:23:58 and shaking him until he tells us what the licensing model should be right.

Alright, it’s like we are doing this utility billing, or what you know shake, shake, shake. So, we were developing the systems infrastructure and code and the business process all at the same time.

So, what we ended up doing, we have this thing called PIE and, my colleague Peco Karayanev is the main developer on it. You’ll always see us wondering around as a unit, he’s not here because he at a wedding today. But it stands for Programable Infrastructure Environment and what we wanted to do and especially because we just gotten into DevOps, and we are trying to internalize what, how we could really maximally collaborate with our developers we said, hey we need our provisioning to not simply be automated, right.

To be programable it needs to be something you can code and you can code too and so —

[00:25:00]

Michael Coté: And who is the “you” in that case?

Ernest Mueller: The developers writing the more kind of the functional parts of the system and ourselves, right. So, the operations guys have always wanted to have more automation, but we tend to want automation in ways that makes sense to us, but not ways that make sense to a developer, and in the end that sort of self service is the goal. We did — it’s in the side, but in our role in IT, we did two big implementations where we really pushed a lot of stuff to self service to our developers. There was this Splunk Log Management Implementation, and then OPNET Panorama like APM tool implementation where we put them in and we pushed the maximally to the developers instead of having them do the Ops tools and in each of those cases —

Michael Coté: Like the SA guys are talking about.

Ernest Mueller: Exactly, so a huge response like actually saying, hey, this isn’t a thing we’re going to do anymore, we’re going to expose it to you, so you can do it. Like that’s when we got into it — that’s when people were seeking us out instead of us finding them.

Michael Coté: So like rattling on that a little bit, like what is it that it made the developers interested in that, what excited this. That direct, that direct access, and direct ability to see how their own handiwork is working in the production environment. And so was it they — like what you’re saying there I just want to sort of observe their handiwork, but was it bad or was it also like so that they could guide in those problems?

Ernest Mueller: Absolutely, I mean that was our goal, right. I mean our goal was to get them to diagnose problems, but that’s not what got them to open Gooey and click on it in the first place.

Michael Coté: They were just curious.

Ernest Mueller: That’s right. It’s like, oh, I wrote this thing, I cast it over the wall. The admins occasionally yell at me about it, what is it really doing and, if I pull open live log-tailing and then go hit it, what am I going to see because — they don’t necessarily no, right.

Michael Coté: What were you’re saying?

John Willis: I was saying, I think it’s like giving them a lot of a control, I was down and self services is about breaking that barrier of — we throw it over the wall, ask question on the wall, get the –it’s the e-mail version of the communication, right, whereas give me the control —

Ernest Mueller: That’s right and a lot of it is just enabling my people to be doing value-added work, right. Carving up log files, and sending them to a developer is one key-work, that’s why we invented computers, right. That’s not something that a highly skilled person should be spending their time on, and by doing that sort of self service we didn’t free up people’s time to hopefully innovate a little bit more, yeah.

Michael Coté: It’s like when people asked me where something is, and I just want to tell them to look it up on Google Maps.

Ernest Mueller: Exactly. I mean move on to similar high value activity in their life. So recurring back up to PIE, so we device the system, and essentially there is some XML based system model, which is somewhat familiar to those who have looked at Amazon Cloud Formation. It’s kind of like that, except it, except it’s more generic. It’s not just for Amazon, right, it’s more a generalized system model. From that we use that as a — as kind of documents for our developers to collaborate with the system administers, all right. It’s like you’re building out your actual system, and how the applications are distributed and all that.

Michael Coté: Because, the developers are saying, here is the type of server that needs to run, and here is the kind of resources, the speed when it’s in production, which it gets to the point of that programable infrastructure or whatever it is. It’s you got the developers to specify the infrastructure, their applications needs to run on.

Ernest Mueller: That’s right.

Michael Coté: And its part of what they’re coding.

Ernest Mueller: Well, exactly and as part of that also we have kind of quickened replacement, so that if an application needs something that’s box specific or environment specific that can all get filled in from configuration that’s part of that model.

So, we got into a point where you could spin up an entire, we call them cloudlets because nobody has really come up with a better word, right. It’s a specific multi-server, multi-role, instance of one of our applications, maybe it’s a dev environment, and maybe it’s a test environment, right. But, spin up an entire cloudlet off of that model without manual prevention. Both, kind of bringing up the cloud systems and getting core software on them and doing application deploys, kind of getting it all, already to run.

[00:29:55]

Then we have a — we use a ZooKeeper based registry which when all the systems, as they come up, they register with it and is used to hook events, and do orchestration amongst the servers. Finally enough like — this is one of the things we’ve been not entirely of our own volition but we’ve been moving some of our applications to Microsoft Azure.

Those are the two things that they have that really are very forward thinking when you create your app, there is a model that describes, and then there is a runtime registry they call it the Fabric that knows what’s going on, and you can hook for events and those two concepts —

Michael Coté: The AppFabric stuff, right? Oh yeah, that’s more like the bus that runs.

Ernest Mueller: Yes, well, yeah with their branding and sometimes a little hard. Yeah, but there is essentially a Azure Fabric API that, it’s like Amazon Instance Metadata, but then also dynamic stuff, also events.

Michael Coté: That’s right, yeah, I forget like Fabric is one of the, Fabric is a term they used in 00:31:07 they kind of mean everything. So, basically they are using ZooKeeper as sort of a combination, a combination asset database catalogue and orchestration engine.

Ernest Mueller: Well, exactly so the concept of a CMBB is all been busted, right like –

Michael Coté: I mean you told, right, I don’t know.

Ernest Mueller: We tried, we poked around with them in IT for a longtime, and finally we just ran screaming. So we are somewhat, a 100% of our systems aren’t cloud, we actually have a couple on from a system integration with this but because they’re mostly cloud. We said, hey, we really need something that it just —

Michael Coté: You’ve got a hybrid cloud. That is fantastic.

Ernest Mueller: We do. That is just super highly dynamic so instead of, yeah instead of writing things down in the database, or sort of running discovery or whatever, we have a model that describes what the system should look like. We spin it up, and registry knows what the system does with the client. Sometimes people confuse those two things, right. When you only have half of that equation, well here is my config and certainly my tool eventually get it to that config, it’s like yes well maybe but maybe not. There’s monitoring right, the monitoring is a course-grained.

John Willis: Who is that?

Michael Coté: It’s Roger Ebert.

Ernest Mueller: Oh, yeah Roger Ebert.

Michael Coté: Awesome, there he goes.

John Willis: I’ll see if he wants to come in DevOps.

Michael Coté: I don’t think he can talk anymore unfortunately, yeah. But otherwise I’m sure he would.

Ernest Mueller: He’s got more opinionated.

Michael Coté: Sorry to be interruption.

Ernest Mueller: No problem –

Michael Coté: But, see you have the real state of the things, and the desired state of things. And, again it’s because when you are — nodes or whatever come up, they register with the ZooKeeper, for instance you have —

Ernest Mueller: Exactly, especially because of the dynamic scaling in the cloud right. So the model is probably not the right place for how many app servers do I want, like because that’s not something I care about our priority, I want there to be as many as there is traffic, right. So, that’s something within the registry can do and that’s —

John Willis: Well, I mean this is the problem, I think you guys, now when I see this all over, this idea that people are, I mean the products like Chef and Puppet, cfengine deal with the model state, but like so you decide okay, there is thing called the web server, configuration then can be involved in having instances of that, and typically they don’t, right. And then even the state of 00:33:53 of things that come and go, right. So, I think there really is a second home, for a new kind of solution, like you guys are doing, it has to sync with the model space, but also with a separate problem solution, which is state awareness you know.

Ernest Mueller: Well, exactly. And I think like we wouldn’t have to wrote our own kind of simple provisioning as part of PIE, but eventually I see it integrating with, integrating with Chef and Puppet were not, because it’s, it tries to abstract things one level higher, right. So, we kind of referred to it as the tool belt for the tool chain, because you end up getting all these tools, and they all have their own —

Michael Coté: That’s a good metaphor.

Ernest Mueller: They all have their own models. They will have their own ways of interacting with them, but I want one canonical, right. I’m never going to teach my developers to read Chef recipes, and Nagios configs, and all the, I mean that’s why we have ops guys, right, who are experts in a hundred different 00:34:56 apps config format.

[0:34:58.9]

So, I want to be able to construct one model that, and the thing that we like about our model, it goes down to the application level, like hey, there is — there is this service here, it talks to this service over on this other tier, on this port, right.

Michael Coté: Yeah, yeah, yeah.

Ernest Mueller: We automatically generate Amazon Security Groups out. It’s like this needs to talk to that on port 443, okay, that will be automatically opened and —

Michael Coté: I mean, you guys are going to have like the dream of a scene to be on those, right that you’re modeling everything including the actual state of things, down to which ports are connected to what he does and 00:35:37.

Ernest Mueller: Exactly, and so it’s been good, like it’s — it’s not as fast as it could be, although some of that is because we’re using –

Michael Coté: And, then so you guys are going to source that, is what the –

Ernest Mueller: Well we’re going to — we’ve gotten kind of management 00:35:51

Michael Coté: That would be the first thing as open source or?

Ernest Mueller: Not the very first but probably the most substantial item like they’ve actually been working on open source like a little stream capture tool, that one of our programmers in Hungary, wrote which is pretty cool. But, this is a kind of a bit larger than that.

Michael Coté: And, then so also, if I remember you guys are kind of window-shop right —

Ernest Mueller: That’s right.

Michael Coté: — like how would you describe the makeup of your platforms?

Ernest Mueller: Okay, that’s a good point. So, on our IT side, right, our website it was 90% Linux, Java et cetera. But all of our desktop development is fundamentally Windows based, right, lot of .Net, a lot of stuff like that. So, for these Software as a Service products, we are having to bring both worlds together to a degree. So for example our FPGA compiled cloud, the thing that powers it is LabVIEW FPGA, which primarily –-

Michael Coté: Is that something like C and C++ or something like that?

Ernest Mueller: Right, right.

Michael Coté: So, which of course you can only work here on Windows.

Ernest Mueller: Yes, well, and that’s another reason that we ended up needing to develop our on solution was we needed both UNIX and Windows support. And, that’s considered to be a myth for whatever reason in the industry. I know that Chef has recently gotten decent Windows support, but —

Michael Coté: Yeah, I know, I mean, I see this all the time in the management start up innovation spaces that Windows is the last thing people think about, or one of the last thing for whatever reason.

Ernest Mueller: Well, absolutely –

Michael Coté: I think it seems to be that you have to do Microsoft program in the Windows, and a lot of the people who are programmers and startups don’t do Microsoft coding.

Ernest Mueller: That’s right, I mean, it’s a little bit of a, it’s kind of religious — because I made this point at the Austin Cloud User Group, I took a show of hands, it’s like okay, who here has to deal with both UNIX, and Windows systems, and everybody raises their hands, right. That’s a reality. You’re a six person startup, great, you can afford to pick the one specific platform, and do whatever you want, but that’s not with the rest of the world —

Michael Coté: I think, that’s what startups deal.

Ernest Mueller: Exactly, Mac client, Linux server are good to go. So, we needed both, and now that we’re bringing up a product on Microsoft Azure, our path to do that, because actually been simplified, because we’re going to take our model and essentially compile it down to the Microsoft Azure model. Now that Amazon has come out with cloud formation, we’re like, oh, well we can essentially perform an extract of our model to various domain specific model and great, we can use it.

Michael Coté: Are you finding that kind of cross-platform support is working out with sort of across EC2 and Azure and like you’re kind of alluding.

Ernest Mueller: Yes, so right now, we have Windows and Linux support on Amazon and we’re working on adding support for Azure. The most, really the most challenging part was just understanding Azure in the first place, which the majority of that is cutting through the Microsoft marketing message to –

Michael Coté: You got to cut through fabrics.

Ernest Mueller: Yeah, to what it actually is, and once you understand what actually is, and you’re like, oh, okay, yeah, that seems doable, right. So that’s where we are right now, we’re building that up.

Michael Coté: I guess, another way of asking the question is, in doing sort of SaaS and Cloud stuff, how aware are you having to be — I’m running Linux, so I’m running Windows, like where does that bubble all the way up to the top so to speak or is it more like I’m running on a service somewhere.

Ernest Mueller: Well, so something somewhere has to know about it, but we try to keep that encapsulated as lower level as possible0. So yes, in the model there is a service and then there is a machine spec.

[0:40:04.5]

That machine spec says, hey, this is Linux or Windows because you probably going to want to know whether you are 00:40:11 to it, or not doing something not offered to it, but it allows the developers do not have to worry about, about that when they’re — because the developers are the ones that you really want interacting with the model most of the time.

Because we set it up, you are like okay, server tier, memcache tier, database tier, and then we don’t have to do that much fiddling with it. It’s the developers that then fiddle with it time after time again, oh, new version, new version for my application, oh, I need to switch it from HTTP to HTTPS, because I finally got around to doing that, and we are trying to give them maximum flexibility to do that without really even having to involve us. In fact the HTTP, HTTPS is a fun example we have.

One of the developers, it’s like, okay, I need to change — I need to change my colors for our licensing from HTTP to HTTPS. I said, great, here go right there in the model and change it. He said, oh, okay, then he is like, okay, well now I need to — I need to call the DBAs and get that calls that’s coming up from our Oracle licensing system to be HTTPS.

Well, that’s spawned two weeks worth of phone calls, and meeting and, cajoling.

Michael Coté: This is John’s theory, you’re involved in Oracle DBA, your agility metric down.

Ernest Mueller: Well, you know, cajoling with the DBA is trying to get it figured out, what about the Oracle Wallet Manager like and that’s for two weeks, and the developer and he is like, oh, okay I see why we’re doing this, now like click it, completely clicked with them all, like I see you are all just doing weird crazy stuff. I understand the value now.

Michael Coté: Yeah. So, somebody asked another interesting measuring question is, well interesting to me I guess, is how often it does a developer’s — how often does a developer drive the infrastructure towards 00:42:03 like a developer like put something in the model or says, You know 00:42:06 like, oh, I guess we need get this new kind of infrastructure like, oh! Now, we need to get a new server for I think, I need just for the profile or something.

Ernest Mueller: A good but we’ve tried to keep it at a pretty, pretty collaborative level with the group with that we have greenfielded. We’ve been very lucky, we essentially have an Applications Architect; a Systems Architect myself, and then a number of developers, and App Developers, and then head of a Systems Developer right. Because we are all working as part of the same sprint and same step review process with all of that —

Michael Coté: The same team more or less.

Ernest Mueller: Exactly that, that when somebody starts coming up with something and needs a spec to be reviewed, myself and the application architect look at it. And I say, hey, that’s not going to fit on our current one, and sometimes that gets complicated.

I mean we recently had a engaging discussion right about, we are working on a product that does a lot data upload to the cloud, and we had conflicting dev and ops requirements about it, right the developers are like, hey, I really want one Rest API, that I can, that I can get in post and delete the same API.

And I said, well we really have to split them out into different servers because we need the data collection to be extremely high up time, extremely low late, and its going to be stable.

Michael Coté: Yeah, now that’s a good example.

Ernest Mueller: And, we are going to be adding other stuff to the reading, and that’s going to be very.

Michael Coté: Yeah, and that’s exactly along of the lines of a potential example. It’s sort of like, well, if you architect the system this way, we are going to need to get CDNs and we got to put a load balancing thing over here and like we are going to need a 100 gig Ethernet instead of a 10 thing that we — I mean it’s kind of like — that would be the kind of collaboration between development, and operations, I would expect to see a lot of this, sort of development people being kind of not aware of the infrastructure that their architecture is driving.

You know exactly what you’re saying, if you have you’re reading and deleting on the same service that implies a lot, and depending on your throughput about your enterprisingness of that service. But if you separate them, then maybe you could get some leeway that an operations person might understand this better a lot more than a developer would.

Ernest Mueller: Absolutely, and so our ability to really collaborate on those issues, and sometimes they’re very sticky. I mean we’ve been talking through authentication issues, and how are we going to do multi-cloud authentication, and ideally integrated with our legacy user repository.

Michael Coté: That thing, you know, are using 00:44:51 or something.

Ernest Mueller: Exactly that’s where we are going, and that takes a lot of iterating, right because a lot of times the developers think, well, why can’t it be simpler, why can’t I just call from Azure to Amazon across the Internet, every time somebody makes an API call, all right.

[0:45:14.9]

It’s like, hey, so, multiply the availabilities together, and you’ve got extra performance like you have to continuously explain it especially we’ve — one of our biggest challenges, we brought web developers over with us to see this team, but we’re working with our traditional desktop software developers for a lot of the product functionality. They’re right, it’s historically very much been a, you just test in your lab.

It’s not well what about if you’re over a low latency connection, what about fault testing, if the network just goes down for 20 seconds and then comes back, does your thing completely freak out because cloud rebooted, is not —

Michael Coté: And it will, whatever.

Ernest Mueller: — as its 0:46:00 you know a solution in a SaaS environment right, and so that’s been one of the biggest challenges to kind of teach and inform, but when people see it they really get into it. One of our LabVIEW FPGA developers happened to cross the video of from Velocity about Facebook, and how many deploys a day they do. And he came wandering over to us, hey, have you guys doing this. Like, yeah, we are at the conference, but we like that you’re excited about that.

Michael Coté: Yeah, now, I was with my friend Charles from the Drunk & Retired podcast, one of his employees here Daniel. And at the 0:46:37 thing last night, they had that same kind of reaction of like wow that looks great. I mean, instead of like thinking about like how I could use that in my job.

So, it is, whereas much as we spend like blowing up, sucking in air about talking about DevOps, it’s important to remember that not every –developers are up on developers stuff, they’re not necessarily up on operations theory and stuff. There is a lot of things to show them that are fun. Well, I only had one more thing to ask, I don’t know if you wanted to —

Ernest Mueller: Yeah, absolutely.

Michael Coté: It was basically, this is the responsible analyst would be asking this question. So we’ve gone over this exciting like technology and practices and stuffs. So, then when you go back to the guys who have lapels, which is going to be one the dress codes for ANI, business people must have lapels, they can be grabbed and shaken.

So, when you go back to the lapel guys like when they look at what you guys have done or see the effects of what you guys have done like — how do they assign business value to it. Like to them what it is like, how do they come to the conclusion hopefully. That all the shaking you’re doing of them and the time and money you’re spending on it is paying off like what’s the business value for it.

Ernest Mueller: Well, actually there is two answers to that, one is that we’re reasonably lucky, and that we don’t have a lot of lapel guys like our CEO is still the electrical engineer that started the company and so our — my management chain is all people with engineering degrees right. So to a degree that streams on some of the communication issues.

Michael Coté: And they probably also have an implicit sort of faith and technology if you will.

Ernest Mueller: Well, absolutely true, and they were willing to let us prove it, right, so —

Michael Coté: Yeah, exactly.

Ernest Mueller: When we did it we — I mean we developed and delivered two software as a service products over the course of one year, and it’s hard to argue with that, it’s hard to argue that in terms of agility. It’s like, hey, we had to spend extra time and effort on making this PIE thing, that directly have a customer impact, but look, we can just, now, we can just stamp these things out.

Michael Coté: So, that was the first thing, is that they gave you the time to establish a foundation.

Ernest Mueller: Yes.

Michael Coté: And, then the way that they rated the success of that is that you could then, is that you then had agility, and that you — in a relatively small amount of time you could put out two releases because you built this. And, so there was business value there and that you could get more stuff out.

Ernest Mueller: Exactly.

Michael Coté: And then I mean is there — and then there’s actual sort of — how are they running their business side of it differently sort of selling the product, or selling whatever it is based on because it’s built this way.

Ernest Mueller: Well so —

Michael Coté: Or is it just that it’s a SaaS, people want to access stuff as a SaaS?

Ernest Mueller: People want to access stuff as a SaaS, like there’s a big cloud, by cloud and we do that too. And customers, to a degree they just want it to work, but on the other hand, they just want it to work right, if you’re overcapacity on a given day, or you’re down on a given day, that spooks people off because with our systems we’re talking about things where people are submitting extremely valuable intellectual property, right.

[00:50:01]

So far FPGA compiled cloud, people are like, hey, you know the things I am compiling, you know, the things that we generally have corporate espionage, people trying to get out of us, right, and I am outsourcing it to you on the cloud, saving my files there, I am saving my data there, is this really, is this really rock solid and secured.

Now, the good thing is we’ve really haven’t had a lot of trouble selling people on that, like you kind of hear that well, like cloud security is the big bug there. Well, it is if you don’t have your act together with cloud security, right, but, and it was the first question out of the mouths, out of our customers, right well what about security.

We can say, hey, we are doing this, and doing this and we have the CISSP on staff, and there is how we make sure nobody ever sees your data, and then they are like, okay, fair enough like they are not just, like – that are not just random spooked.

Michael Coté: They had to make sure you got it covered internally.

Ernest Mueller: Exactly yeah.

Michael Coté: That’s sounds like another — your value, is basically your customer’s base, they want to outsource that infrastructure of compiling things to you and they want some service to do this for, and they want to own it all.

Ernest Mueller: Absolutely it’s humbling having come from IT, but the people that come to us with big dollar opportunities, and they say, yeah and I wanted to do this because, you know I am smart and I program and I could have done it myself, but I would have to go to get a server from IT and I would rather give you all a briefcase full of money. That’s — gets messed up from one point of view, but we’ll take it from the other point.

Michael Coté: So, you are sort of enabling some of your customers, I won’t say all, maybe does all, but to basically do rogue IT to not to deal with their own IT departments to get their job done?

Ernest Mueller: It’s very true, I mean or at least or at least a different sort of engagement, right. If you consume it as a service even if your IT department is engaged and has oversight on that, it has to be less of a bottleneck, it takes less time to evaluate a SaaS vendor and then to build out a system.

Michael Coté: Yeah, the same SaaS dynamics that works in the CRM and application space work in the FTP or whatever.

Ernest Mueller: Absolutely the hardcore engineering people, there is all kinds of mobile apps, it’s like here is an oscilloscope on your iPhone. I mean it’s the exact same social media, mobile cloud, they are just as compelling to propel our heads as to whatever the course 00:52:42

Michael Coté: It’s not FPG, or is that – it is FPGA.

Ernest Mueller: FPGA.

Michael Coté: There you go, it’s not a first person anyway. I am way beyond my theme, once there is three dimensional objects involved. All right, well 00:52:56 is there anything else you guys want to talk about. I think that was great, I mean didn’t wanted to sit down for a while and just kind go over.

Ernest Mueller: Yeah, thanks for having me.

Michael Coté: — what you guys doing, I think it’s nice to talk to someone who is implementing all these technologies and practices and a business of doing it, not just sort of the tooling and all that kind of stuff so, so great and I appreciate it.

Ernest Mueller: Thank you.

Michael Coté: We’ll see everyone next time.

Disclosure: see the RedMonk client list for clients mentioned.