Skip to content

When Web Companies Grow Up They Turn into Java shops

I presented at JAX London 2016 yesterday, with a talk entitled Java for Cloud Natives. It’s a theme that Stephen and myself have been working on for a few years now – the rise and fall and rise of Java. After its inexorable and steep rise Java plateaued, so people started saying it was “dead”. In our culture after all, not growing seemingly means dying. But open source Java has showed surprising resilience – it is the ecosystem that has fostered much of the interesting work in the Big Data space. One intriguing phenomenon is that Web companies have been a big part of Java’s third act.

Because the Java ecosystem was open source, the platform, notably the JVM, became relevant to second generation Web companies as they hit scaling problems. Firms that started with strong biases to Rails or PHP, with a strong Java sucks mentality, discovered over time that Java was kind of useful. JVM optimisation allows for scale. Java language skills are plentiful. Doug Cutting chose to write Hadoop in Java because he felt it was going to be more maintainable over time. Facebook, which was all about the PHP, runs probably the world’s biggest biggest Hadoop cluster today. Twitter is on the JCP.

The Apache Software Foundation has fostered most of the interesting projects in the Big Data space – many of them written in Java,  see for example Hadoop and Giraph, or Spark and Kafka which are written in Scala, a JVM language. Back in April Heroku (originally one of those “Java sucks” companies) announced it was supporting Kafka, and writing Scala code to contribute. Today Twitter is a major contributor to popular projects such as Netty. I have made the point about Web companies before, but it blew up again yesterday.

At least 50 retweets and 50 likes, and it just got shared by @java so we can expect some more. Given the virality, and some useful comments on twitter, notably from @joshbloch

I thought it was important to clarify a couple things. For one thing, when I say Java shops, I don’t mean Java Only shops. I am certainly not arguing that any of the companies above have stopped using other languages or platforms, but rather that as they have matured they have come to see Java as a useful tool, part of the engineering arsenal, rather than something they just filed under “sucks”.

With respect to Google, it can be somewhat opaque, and while it was my understanding there is a fair bit of core engineering in Java, it was pointed out (thanks @chris_gaun) that Borg is written in C++, and of course Google is a major Go shop.

The next order effect that I think is potentially really interesting however is that as industry leaders and Cloud Immigrants such as Bosch and GE, which are using open source methods, culture and software to underpin their digital transformations they start naturally making open source contributions, just as the likes of Netflix do. And where GE leads, other corporations follow.

And enterprises of course write software in Java.

Slide decks can be a bit deceptive of course, without the words being said in the room, but here are mine if you’re interested. Jaxenter did a really nice write up of the talk here – 5 things we learned from James Governor | JAX London 2016.

Java for Cloud Natives at JAX London 2016 from James Governor
GE is a client.

Categories: Uncategorized.

And nearby somebody finally gets it – new interaction programming models

I saw a tweet this morning that caught my eye.

Having written about Conversational IoT earlier this week I thought it was worth a quick follow up. I said then that

The nature of the current dominant user interface – the apps – is breaking down.

The existing screen obsessed mobile first model, where we’re all looking down at our phones like


So from my perspective Google’s Nearby platform is very interesting because it has a design principle based on interaction between a number of devices and end points. Generally development has been about this screen or that screen rather than this screen and this screen. And now we’re throwing voice into the mix with Amazon Echo, Siri, and Cortana.

Even watching TV isn’t a one screen activity any more – we’re tweeting from the other screen at the same time. I am surprised nobody has nailed the mobile for sports data, augmenting the TV experience yet. What’s interesting to me are the new models for interaction between all the devices and services we use. Amazon’s dash buttons are another non screen-based end point.

Beacons have been around a while, obviously, but Apple and Google are in unique positions to get this right. I will have to look at it more closely but at first glance Nearby looks like a really interesting framework, in that it comes at things from the interaction rather than the screen perspective.

Nearby Messages exposes simple publish and subscribe methods that rely on proximity. Your app publishes a payload that can be received by nearby subscribers. On top of this foundation, you can build a variety of user experiences to share messages, create real-time connections between nearby devices, and receive beacon messages.

Nearby Connections enables your app to easily discover other devices on a local network, connect, and exchange messages in real-time. Use Nearby Connections to create multiplayer and multi-screen experiences.

Nearby Notifications is a new feature allowing developers to tie an app or website to a BLE beacon and create contextual notifications, even with no app installed.

I can’t wait for better heads up design, interaction and programming models. The idea we’re redesigning urban signage for example to accommodate our cricked necks seems crazy to me. In Augsburg they’re now embedding signals into the road after a girl was hit by a tram.



Categories: Uncategorized.

Conversational IoT–’SUP?

Conversational IoT emerged strongly as a theme at our Thingmonk conference over the last couple of years – which is on again next week. Nick O’Leary gave a stellar talk on the subject in 2014, inspired by the work of Tom Coates.

A lot of experimentation in the space happened on Twitter with tweeting objects including ferries, bridges and thirsty plants (sadly Twitter didn’t support the interesting work by developers in the space, another potential opportunity not acted upon).

“We no longer appear able to send a robot to Mars, or land a probe on a comet without an accompanying twitter account bringing character to the events.”

But O’Leary also pointed out one of the most interesting ideas behind things that can talk  – the emotion.

“There’s always a sense of excitement when these inanimate objects start to have a conversation with one another. The conversations between the philae lander and its orbiter were particularly touching as they waved goodbye to one another. Imagine, the lander, which was launched into space years before Twitter existed, chose to use its last few milliamps of power to send a final goodbye.”

ironically enough while writing this post this happened

Lifeless body, lifeless body, lifeless body.

We imbue. It’s what we humans do.

So where would we be without the thoughtful contributions on thinking machines by Douglas Adams and the writers of Red Dwarf.  I am fascinated by the notion of authorial “voice”. Any good writer has one. Indeed these talking spaceships are of course humans playing at being machines. My own founding partner did some good work in this regard with @analyticsmonk.

But what of machine voice? Is it really a thing? Well if machines, or rather the software that drives them, are going to be interacting with people, it’s going to be emotional.

Some cultures are I suspect ahead on this stuff – I remember being initially taken aback in a discussion with a couple of Japanese people when they said Japanese generally believe that machines have souls. But of course, we have the same cultural tic. People name their cars, we anthropomorphise everything. Which is one reason I think we are fetishing chatbots right now.  Bots are people too. If companies can be people then machines definitely can.

Jeff is right – we’re being reductive. Chatbots have become our exemplifying vision of talking things. The Developer Aesthetic has emerged from IRC, through Github ops, and into Slack as a way of communicating. Slack rather than twitter begins to feel like the foundational technology for how we’re going to build conversational interfaces. Slack may not have an IoT play yet, but I expect the platform to increasingly be used for customer relationships, rather than internal comms. Part of that support for the customer is going to be engagement with the machine.

You should definitely read Chris Messina on what he calls Conversational Commerce. My only quarrel with the term is the use of “commerce”.

Before I begin, I want to clarify that conversational commerce (as I see it) largely pertains to utilizing chat, messaging, or other natural language interfaces (i.e. voice) to interact with people, brands, or services and bots that heretofore have had no real place in the bidirectional, asynchronous messaging context. The net result is that you and I will be talking to brands and companies over Facebook Messenger, WhatsApp, Telegram, Slack, and elsewhere before year’s end, and will find it normal.

We’ll shortly be talking to things in the same way. This conversational model isn’t only about buying and selling and brands– it’s deeper than that. It’s going to be completely pervasive.

Stephen wrote a good piece about why Microsoft was acquiring Linkedin which plays to these scenarios. He imagined this conversation

“Cortana: Where has Jane Smith worked?”
Jane Smith has spent ten years as a Java programmer for companies in the insurance and healthcare industries.”
“Cortana: What are Jane’s professional certifications or memberships?”
Jane is an Oracle Certified Master, Java Enterprise Architect, a certification that less than 5% of the applications for this position claim.”
“Cortana: What languages does Jane Smith speak?”
Jane is fluent in English, French and Spanish.”
“Thank you, Cortana. Can you schedule an interview with Jane next week?”
Of course.”

I haven’t had any briefings on it yet, but I think this kind of thinking is driving Salesforce to create “Einstein”. and of course IBM has made a lot of the running in thinking machine technology with Watson.

Chatbot startups have raised over $140m in the last 8 months. A good chunk went into Slack startups and CRM.

The nature of the current dominant user interface – the app – is breaking down. I thought this MG Siegler piece – K I Get Uber – was really instructive about how.

On Friday, I was chatting on Facebook Messenger with my wife about where we should grab dinner. Once we decided, I responded with a simple, informal “K I get uber.” Meaning, of course, that I would call an Uber and meet her there. What’s amazing in this mundanity is that Messenger was able to parse my words and figure out that I indeed wanted to order an Uber car, and so it automatically inserted a “Request a Ride” button right below my chat bubble.

I clicked the bubble, went through an Uber ordering flow — all within Messenger — that in some ways was better than the flow within the Uber app itself, and the car was on its way. I got a message from Uber’s Facebook Messenger bot, letting me know the car was on its way and an embedded map to track it. But I didn’t even need to do that because when it got to me, Uber messaged me, again via Messenger, to let me know the car was there, complete with driver name, car type, and license plate number. When the ride was over, Uber messaged me, yet again through Messenger, with a detailed receipt for the ride.

Who needs apps?

Machines that parse intimate slang.

Frankly this stuff is getting existential.

What are some of the disciplines that underpin Conversational IoT?

We spend a lot of attention on integration for automation, but that feels like a solvable problem, given it’s a core competence of IT. We have emerging protocols – notably MQTT (author Andy Stanford Clark will be with us at ThingMonk again this year) that make sense in a listening world rather than always on like HTTP. We have amazing platforms like IFTTT showing that integration doesn’t have to be painful. IFTTT is likely to emerge as a major IoT player. It has started down the connected home route, which is already pretty crowded but it has momentum and a developer savvy user base to tap into.

We can do the automation.

But writing witty, pitchy even plangent replies on the other hand may be a lot harder. Especially if we need to be able to respond sensibly to every request. When I think about conversation done right from a user experience perspective I think of Joshua Porter’s “microcopy”.

Ironically, the smallest bits of copy, microcopy, can have the biggest impact.

Microcopy is small yet powerful copy. It’s fast, light, and deadly. It’s a short sentence, a phrase, a few words. A single word. It’s the small copy that has the biggest impact. Don’t judge it on its size…judge it on its effectiveness.

Short, sweet and to the point. That’s what we need to automate. Of course machine learning is going to end up writing great copy. ML is analysing emotional state in movies and recreating photos using copies of artist’s painting styles.

Of course chat is not the only interface – voice is taking off. Siri, Cortana, Alexa, Leah are all going to make huge headway in the next 5 years. Perhaps we might even get a digital assistant with a male voice. While we associate Alexa Skills with the Echo for now, Amazon plans to make it pervasive, found in all kinds of listeners.

Then there are emojis. Machines are probably going to send each other emoji as status updates. For a while at least. Until emoji become like cassette tapes.

So great copy-writing, machine learning, chatbots.

What about the platforms and changing infrastructure requirements? We talk a lot in tech today about event-driven reactive platforms. The serverless economy is going to come into its own here. An architecture based on waiting, where you only pay at the point of customer value. One reason Amazon acquired 2lemetry was that Kyle Roche had worked this out.

The affinity of IoT and serverless is exemplified by the fact Amazon’s Echo architecture is built end to end on AWS Lambda.

In Conversational IoT machines will spend a lot of time waiting. They will be sitting idly by in our cars, in our homes, our workplaces. Listening, waiting. They will probably get lonely, and then they’ll talk to each other. One of the key insights of Thington is that machine chatter should be in natural language – we need to be able to understand what the machines are saying to each other. This is higher order craziness. Robot drivers arbitraging human lives.

I Robot indeed. We’re going to be needing those laws of robotics.

At Thingmonk 2015 Coates’ cofounder at Thington – Matt Biddulph continued the theme, with Welcome to the Conversation pointing at what their company was planning to build – a conversational model for IoT in the home. He is back for Thingmonk next week, now his company is out of stealth. But Matt is just one of many many great speakers in a diverse lineup.

We will have coding workshops on Alexa Skills, IBM Watson and Splunk data visualisation. We will having coding by candlelight. Great food. Lovely people.

We have some tickets available. You should buy one.



Amazon, IBM, Microsoft and are all clients

Categories: Uncategorized.

Thingmonk and Software Circus hitting their diversity targets


axiom 1 – tech events are simply better when they’re diverse and inclusive

axiom 2 – event organisers should always make an effort to encourage diversity of speakers

axiom 3 – event organisers should always make an effort to encourage inclusion for all attendees

axiom 4 – events need clear policies and Codes of Conduct to support inclusion

We have signed up to the diversity charter created by Emily Weber. We set ourselves a goal for Thingmonk 2016, our IoT conference, of having an even gender representation and we’re pretty pleased that we hit the mark. Unfortunately one woman speaker pulled out, but we’re still very close to our target. We were also super happy when Yodit Standon, founder of, sponsored a diversity scholarship. We’ve now accepted 16 excellent applications from under-represented groups in tech. All in all Fintan and me are pretty happy with where we are, although obviously we can always always always do better, try harder. Please do join us and help make it a super welcoming tech event. We still have some tickets available.

And everyone needs to chip in so I was pleased when discussing Software Circus with Jamie Dobson recently – it’s a cloud native conference I haven’t been able to attend yet, but it gets ridiculously stellar reviews. Of course I am on holiday and so missing it again this year, which is the bad news. The good news Jamie is also working the diversity angle.I love the diversity policy, which is admirably simple.

As many of you will remember from last year, we are heavily committed to promoting diversity in the world of software.

It is with immense pleasure that we can now announce this year’s diversity policy which, due the incredible support of our sponsors, is even bigger this time round with tickets being given away, no questions asked. If there is enough demand we will release more later.

If you believe that you belong to an underrepresented group in the world of software, you can use the code COMEONECOMEALL to claim your free ticket to the conference.

You can claim your diversity ticket on the registration page here:

Marco Abis is also working to create better more inclusive events. Check out, which is putting the hug in hugops.

If you prefer video, here’s me talking about our Thingmonk program.

Categories: Uncategorized.

Spring Won Platform: the Big Switch just happened

protect yourself from mayhem

Avoid Mayhem. Do programming. Since 2002 RedMonk has argued for the primacy of developers and practitioners in driving business advantage. We see ourselves as developer advocates as much as industry analysts. Our thesis concerns The New Kingmakers.

Ten years ago we were outliers – not exactly a lone voice but definitely a rare one. Even as little as three years ago some of our arguments could come across as a little outlandish – “Outsourcing doesn’t work, you say?”. “We should let developer choose their own platforms – can you just run that by me again? This is a bank, you know.”

At some point in the last couple of years the Big Switch just happened. The Fortune 500 has got the memo that the only sustainable business advantage in an age of unprecedented technical change is unleashing engineering talent. I am not a big fan of the way our industry abuses the term “paradigm shift”, but we are evidently witnessing one right now, in a way that Thomas Kuhn would recognise – the Structure of a Business Technical revolution. A couple of weeks ago I attended Pivotal’s SpringOne Platform conference, which really solidified things for me.

I have certainly attended major vendor conferences over the past few years in which enterprise business executives talked about the growing importance of developers in their operations. But SpringOne Platform was the first major vendor conference I’ve been to where the voice of the customer was the voice of the developer.

Julie Burnard, Ford Motor Company. Greg Otto, Comcast. Doug Safford, AllState. Sheralynn Briggs, The Home Depot. Justin Erenkrantz, BLOOMBERG. Ted Tollefson, Kroger. Andy Zitney, McKesson. Brad Miller, Citi. Sebastian Blandizzi, MANULIFE.

I had conversation after conversation with leaders working at companies that aren’t “cool”, showing that they get it. Fortune 500 people seemingly just as attuned to modern developer culture. The passion for developer-driven reinvention was really bracing. You should watch the keynote. If you’re a business that hasn’t got the memo yet make sure your CEO watches some of the videos. Companies are currently building software development competences that are going to lead to sustainable competitive advantage.

Adrian Cockcroft also spoke at the event, but he doesn’t sound like he’s 5 years ahead of everyone else any more because he called it right – 2015 was the year enterprise Cloud became a thing. One of Adrian’s tropes is a great line about hiring, from his time at Netflix. Enterprises would complain they didn’t have access to superstar talent in the way Netflix did. Cockcroft would reply that on the contrary, “we hired them from you, and got out of the way.”

The direction of travel in 2016 is much less clear-cut however. At SpringOne platform I talked to Brad Miller, who left Amazon to join Citi. He isn’t alone.

The war for talent in IT? You haven’t seen anything yet. To Cockcroft’s point the companies where high performing individuals and teams will choose to stay are those that “get out the way”.

But how do enterprises get better at writing and managing software, and learn how to get out of the way? One way is to bring in outside expertise to teach new methods and styles. Which brings us to Pivotal’s founding story. Pivotal Labs was originally a consulting shop, acquired by VMware, which had built a reputation improving already cutting edge engineering operations at companies including Google and Twitter, with a focus on pair programming and a set of practices called the Pivotal Way. Enterprises are buying into this approach. It is the spearhead for most customer engagement. Today the companies adopting Pivotal Cloud Foundry generally start with services engagement. Frankly there aren’t many services firms in the world that truly understand how to teach agile, devops, and CI/CD. Outside Pivotal itself you might like to Thoughtworks, while IBM is building up its own skills with its IBM BlueMix Garage Method. Flattery indeed. But the Pivotal Way is more than a method – it’s a cult inside Pivotal, and customers are evidently buying in.

I met with Doug Safford of Allstate at the event. It was basically meeting a RedMonk exemplar. He argued that the press and media had been missing the point.

You write about Uber and Airbnb all the time but this is much bigger than that. The change is as fundamental as when the assembly line came in.

Allstate is in the midst of a major talent driven transformation, and Pivotal Labs is at the heart of it.

The business leaders said… we’re not doing cubicles any more.

Compozed is the internally branded entity at AllState that covers all agile development on the cloud, starting with training and work environment.

We wanted to change internally and bang up against the mothership. I love seeing quality engineering coming back. it was really disappointing in the outsourcing wave

I want complete delineation between scrum agile and waterfall. i haven’t seen a bigger bang for the buck than pair programming, XP and the Pivotal way.

A big part of my job is not treating people like they’re not good enough.

Before we treated people like they couldn’t do stuff, after the training, now they’re blowing us away.

We hired people to maintain poor code. Now we’re saying create for us, it’s a different comfort level. Management were all bred and hired and rewarded for climbing the ladder, but all along we had this jewel, we were ignoring, the developers.

Safford used the phrase the “frozen middle” to describe process-serving middle management. It turns out business alignment is about getting engineering closer to management.

Amen, brother. I would work for Doug. He’s hiring.

The big switch is happening all over the place. GE was an early investor in Pivotal. Same week as the SpringOne Platform conference CEO Jeff Immelt said all new employees would learn to code. I am not sure this approach totally makes sense, but it’s a great marker for the corporate world. Where GE leads others follow.

Why GE is giving up employee ratings, abandoning annual reviews and rethinking the role of HQ.

Which brings me to how we will get there – our culture. We may be a century-old company, but we need to move quickly, take risks, fail fast and behave like a startup to keep winning. I joined GE 34 years ago, and until recently our management could make every decision in the headquarters. Those days are over. We have to embrace decentralization and use technology to help our people to stay connected and allow more automated decision-making so you can look at an app and see what’s going on inside the company.

But culture is not just apps. It’s a combination of people and technology. If you are joining the company in your 20s, unlike when I joined, you’re going to learn to code. It doesn’t matter whether you are in sales, finance or operations. You may not end up being a programmer, but you will know how to code. We are also changing the plumbing inside the company to connect everyone and make the culture change possible. This is existential and we’re committed to this.

This is the environment in which enterprise tech is now competing, and it’s not clear that everyone is ready for the change. If you’re Amazon Web Services everything looks rosy. But outsourced infrastructure drives the need for insourced development talent. Clearly Amazon has nothing in it’s services arsenal as yet that looks like the change Pivotal is set up for. Cloud Foundry is a fulcrum for the Anyone But Amazon coalitions.

Pivotal can engage in developer retraining and restructuring, then offer PCF as the platform for developing apps without having to worry about infrastructure.

But just before signing off I shouldn’t ignore the pun in the title. While Pivotal has been engaged in hand to hand combat against IBM around Cloud Foundry engagements the Spring team basically rebuilt the platform from scratch in a major refactoring exercise aimed at making Spring easier to use for web apps and microservices. Spring Boot takes the concept of convention over configuration to it’s natural conclusion in the Java world. At a time when Oracle and the JEE community are at loggerheads Pivotal has delivered an elegant platform with deep attention to detail in terms of improving the Java developer experience- something that Java has never excelled at. During the opening keynote the phrase “owning Java” was used. So the gauntlet is thrown down, and IT shops that aren’t ready for a Cloud Foundry-only stateless world Spring Boot looks like an effective choice for transactional applications. I will be writing more on this, so check back in early September.

Before I head off on holiday though I leave you with the best ever quote about devops

And a bonus video about the Pivotal Spring One event. Disclosure Amazon, IBM and Pivotal are clients. Pivotal paid for my T&E.

Categories: Uncategorized.

when going on holiday is marketing. on Splunk, IoT, transport data and out of office replies

deutsche bahn


I sent an email to Matt Davies, EMEA Head of Marketing for Splunk, this morning and got this fantastic auto-reply.

If you’re going on holiday or on a business trip then a) have a great time and b) think about your journey and how you’re interacting with machine data – it really is a case of planes, trains and automobiles:

If you’re flying from Gatwick airport then they use Splunk to make the airport more operationally efficient.
If you’re flying on an Airbus then they are helping make sure their IT systems are secure.
If you’re on a Deutsche Bahn train then they are capturing data from the tracks.
If you’re in an electric VW car then they are making sure their E-Up cars are running properly.
If you booked your holiday through e-Travel then they saved millions of pounds a year by making sure their website is always up.

130 words. So effective. This approach is great on so many levels. It makes a virtue of the out of office reply, obviously. It’s engaging and funny. It tells a story about Splunk’s – ahem – direction of travel. And it means Matt is still working even when he is away.

I now want to know more about Splunk’s push into transportation telemetry. Luckily Splunk is sending some of its engineers to the Thingmonk hackday in early September, to show us more about how VW is using Splunk as it tries to reposition as a green leader, post dieselgate with an aggressive push into electric cars. I look forward to seeing how we can integrate a Pokemon seeking drone with data about electric cars. The conference is looking good, with a raft of great talks – you should along too. Find out more and buy your tickets here.

Categories: Uncategorized.

IoT ops for humans. making things operable. go to this.



Operability 2015 was the best DevOps conference I have ever been to. Perhaps the best tech conference full stop, and i only made it for one day out of two. Operability was great because it wasn’t about tech, exactly, but rather all the human aspects of how and why we keep systems running in a way that makes users feel good about them. It’s fashionable to talk about empathy in tech right now, but Operable made that a simple matter of practice.

Scott Klein, cofounder of StatusPage, recently acquired by Atlassian, started his talk with some mindful breathing exercises. No seriously – and a largely British audience responded with hardly a chuckle. After some breathing Klei laid down some science about how to communicate the status of your service. It was a great talk – we learned about the value of post mortems, with plenty of best practices for dealing with that outage you really wish didn’t happen.

Empathy, psychology, humanity – that’s what’s important in systems and operations management.

Think about Twitter and the fact we celebrated the Fail Whale. People literally have tattoos celebrating Twitter downtime. Human psychology is not simple when it comes to making allowances for services that we rely on and or just love.

Today, curator Marco Abis said some nice things about our internet of things conference ThingMonk

“it’s simply the only event in London you will find where you can interact directly with the people making the scene.”,

He also made a request for proposals. If you’re involved in managing IoT systems and you’re clueful about human factors you should definitely send a proposal to talk at Operability 2016.

One area I think will be critical to managing IoT, and managing systems more generally, is the bot revolution – at ThingMonk over the last couple of years the strongest emergent theme has been Conversational IoT. Machines talking to people talking to machines.

So Like Cog built by Operable (not to be confused with Operability conf) in ops. While mentioning Operable I should say that Mark Imbriaco, Operable founder and “father of chatops” will be speaking at Operability. He will be worth the price of admission alone, and it’s a very very good lineup again this year. I think I have a few discount codes for Marco’s conference – please comment here if you’d like one.

Bots and command lines is only part of the story in conversational Iot and ops however – voice systems is definitely going to be a very big thing. We’ll have Alexas to hack, a workshop on how to do so, and some giveaways at Thingmonk September 12-14.

I would respectfully suggest you come to both conferences.

Categories: Uncategorized.

let’s hear it for the maintainers!



A couple of weeks back Anne Johnson pointed me at a new report from the Ford Foundation – “Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure.” Written by Nadia Egbhal the report examines the surprisingly unsupported foundations of much of the infrastructure we rely upon.

“Nearly all software today relies on free, public code, written and maintained by communities of developers and other talent. This code can be used by anyone—from companies to individuals—to write their own software. Shared, public code makes up the digital infrastructure of our society today.

Everybody relies on shared code to write software, including Fortune 500 companies, government, major software companies and startups. In a world driven by technology, we are putting increased demand on those who maintain our digital infrastructure. Yet because these communities are not highly visible, the rest of the world has been slow to notice.

Just like physical infrastructure, digital infrastructure needs regular upkeep and maintenance. But financial support for digital infrastructure is much harder to come by.

In the face of unprecedented demand, the costs of not supporting our digital infrastructure are numerous. No individual company or organization is incentivized to address the public good problem alone. In order to support our digital infrastructure, we must find ways to work together.”


Many of the open source projects and modules we use every day are maintained by one person – and that person likely spends a lot of their time being abused for the privilege. Open source consumers are generally quite a demanding and yes entitled bunch.

It is all too easy when you’re a consultant like me, or a developer working at a well funded startup, to imagine that everyone gets paid to create and sustain open source projects. At RedMonk after all we built a company that gets paid advising people to do just that. And yet. My business partner Stephen O’Grady wrote a book called The Software Paradox, examining the fact that while software becomes ever more valuable, companies are increasingly unwilling to pay for it – software license sales at major companies are trending to zero. Instead of selling software licenses and maintenance fees companies must build businesses on services and data.

We live in a world when tech vendors, web companies, VCs and startups all support open source projects. So everything is fine, right? Not so much.

I mention maintenance for a reason – it’s notable that in the age of cloud based services we see maintenance fees as some kind of absurd retrograde mark of the old school. OMG remember when software companies used to charge for maintenance??? How lame. Legacy technology sucks. Things are so much better now.

But maintenance has a cost. Of course it does. Because maintenance has value. Because maintenance requires labour.

That’s the beauty of Egbhal’s report – it makes the analogy of roads and bridges – if we don’t invest in maintenance then they collapse.

Andrew Russell wrote a fantastic post – Hail The Maintainers – that gets right to the heart of it.

“Innovation is a dominant ideology of our era, embraced in America by Silicon Valley, Wall Street, and the Washington DC political elite. As the pursuit of innovation has inspired technologists and capitalists, it has also provoked critics who suspect that the peddlers of innovation radically overvalue innovation. What happens after innovation, they argue, is more important. Maintenance and repair, the building of infrastructures, the mundane labour that goes into sustaining functioning and efficient infrastructures, simply has more impact on people’s daily lives than the vast majority of technological innovations.”

I wish I had known about his conference The Maintainers – How a Group of Bureaucrats, Standards Engineers, and Introverts Made Technologies That Kind of Work Most of the Time.

The maintenance problem is not just about software. Capitalism itself is terrible at maintenance. We’re so busy talking about and investing in “innovation” that we forgot the foundations need to be built and maintained. Witness the current fetish for “disruption” as a good in itself.

But I don’t see Uber building roads. Elizabeth Warren got it just right:

“There is nobody in this country who got rich on their own. Nobody. You built a factory out there – good for you. But I want to be clear. You moved your goods to market on roads the rest of us paid for. You hired workers the rest of us paid to educate. You were safe in your factory because of police forces and fire forces that the rest of us paid for. You didn’t have to worry that marauding bands would come and seize everything at your factory… Now look. You built a factory and it turned into something terrific or a great idea – God bless! Keep a hunk of it. But part of the underlying social contract is you take a hunk of that and pay forward for the next kid who comes along.”

I have sat through seemingly hundreds of presentations over the years where vendors explain that IT is broken because it spends too much on maintenance, and not enough on innovation. Consulting companies love to show off the innovation to maintenance pie chart to chide the customer into doing something new.

I have covered mainframe technologies for more than 20 years now, and they were already considered dead when I started doing so. But legacy just means in production. It just means creating value. Here are some thoughts from legacy boy.

Don’t get me wrong – I am certainly not claiming that everything old is good, that every production system should be left as it is. On the contrary – continuous deployment is all about building maintainable systems that stand up to the rigours of production and can be changed and maintained over time.

But understanding the value of the maintainers means taking a broader view, touching on many of the critical social issues we face in open source. Thus for example we celebrate the coders, but not the people that made the patches, or documented the system, or helped manage the community, responded to the pull request politely. The best and most useful open source though has the best documentation, has the best architecture of participation.

Linux kernel maintenance is not a paragon of inclusion and welcoming behaviour. But this post by Linus Torvalds was on point.

So at one level I absolutely _hate_ trivial patches: they take time and
effort to merge, and individually the patch itself is often not really
obviously “worth it”. But at the same time, I think the trivial patches
are among the most important ones – exactly because they are the “entry”
patches for every new developer.
I just try really hard to find somebody else to worry about them 😉

(It’s not a thankful job, btw, exactly because it _looks_ so trivial. It’s
easy to point to 99 patches that are absolutely obvious, and complain
about the fact that they haven’t been merged. But they take time to merge
exactly because of that one patch that _did_ look obvious, but wasn’t.
And actually, it’s usually not 99:1, it’s usually more like 10:1 or

So please don’t stop. Yes, those trivial patches _are_ a bother. Damn,
they are _horrible_. But at the same time, the devil is in the detail, and
they are needed in the long run. Both the patches themselves, and the
people that grew up on them.

Jessie Frazelle argues that closing a patch request that doesn’t get merger is an essential skill for a maintainers and offers some best practices in how to do so. We all need to get better at this stuff.

One reason Egbhal’s research is so welcome is that I certainly underestimated the maintainer issue. As I saw the tweets roll in from what was obviously a great conference – Open Source & Feelings – in 2015 I totally didn’t get it. I had a total empathy failure. I was an asshole. For example, people from the Django community were saying they felt undervalued because they weren’t being paid. My immediate reaction was – get better at building an economic model that works then. Lots of commercial companies rely on on Django ergo you can get paid. Do some business development. Make a commercial foundation.

And yet- I have friends that are Django maintainers. Not my best moment, as I say. Django Girls is an amazing community. It deserves support.

Maintainers come in all shapes and sizes – with a huge variety of motivations and needs. Why should they have to do things in a particular way that suits a particular kind of person or even business model? As Russell points out maintainers are often introverts – and bus dev and hustling may not come easy. And let’s not fool ourselves that everyone finds it easy to get funding – women and black people are at a huge disadvantage by the numbers.

If want to create sustainable and maintainable code we need sustainable and maintainable communities and we need to show a lot more empathy. OSFeels is *awesome*. We need to support the people doing the work. I have a plan in that regard, of which more later.

In the meantime you might enjoy this video in which I try and sum up of these issues in 3 minutes.

Categories: Uncategorized.

thingmonk 2016 – industry 4.0 meets pokemon go



Thingmonk 2016 is shaping up really nicely and it’s coming soon. September this year. We’ll bring you the usual mix of amazing talks, warm and whipsmart people lovely food and drink experiences, and… we’re bringing sexy back. That is – after a hiatus last year, we’re doubling down on our hackday.

Day Zero kicks off with a half day conference within a conference curated by the Eclipse Foundation. I would argue that of the open source foundations Eclipse has done the most successful, targeted job of making itself the home of IoT standardisation. One of the reasons the machine 2 machine (M2M) wave never took was it was trying to use old school standardisation by specification mechanisms. Standards in software today are built differently – by implementation, a revolution that Eclipse helped foment. When a company like Bosch is on the record as saying that IoT won’t succeed without an open source first approach you know things are changing dramatically in the industrial internet space. Check out hawkBit. Bosch will be presenting at the Eclipse day on standardisation via platforms like Hono. Also – How about Running UK railways with Eclipse Paho & Mosquitto? More info about the Eclipse agenda from Ian Skerrett here.

Following the Eclipse talks we’ll have a hackday with workshops on IoT programming with IBM Watson, Pycom (anyone for drones finding Pokemon?) and Amazon Alexa Skills (tbc). We have been talking about Conversational IoT for the last couple of years, so it’s time to start building it. People talking to machines talking to people. We’ll definitely have some Echos to play with, and expect a couple of giveaways too. This won’t be a competitive event, but a great place to learn, meet people and build cool things. We’ll finish the day Coding by Candelight until 9pm.

Day one and two will be the aforementioned excellent talks. Kicking off will be Sam Phippen – his closer in 2015 was so mind-blowing we had to have him kick off this time around.

Bots, data wrangling, Design, startups, industrial automation/Industry 4.0 (Bosch again), quantifying your fitness,

We’re certainly not cheerleaders for thoughtless automation though- we’ll have skeptical, deeply critical talks about creating a culture and making the right design decisions so that the IoT serves us, rather than us serving it. Terry Eden will explain why you should run screaming from the Internet of Things in a domestic setting.

ThingMonk is about tech but it’s also about creating a warm, supportive, welcoming and diverse community. This year we are honoured to introduce a special diversity scholarship sponsored by Please apply if you’re from an under-represented group in tech.

Buy your tickets for the conference here. More sponsors are also of course always welcome.

Categories: Uncategorized.

what kubernetes and cloud9 tell us about the new industry – anyone but amazon

One of the useful lenses to understand the tech industry is coalition theory. It can be surprising that particular vendors collaborate closely at any given time, given they are ostensibly arch-rivals. Generally however there is an outside threat or driver that explains what is going on.

Think for example of IBM and BEA (later acquired by Oracle) competing head to head in the late 1990s. Ownership of the Java application server market was the prize, but for all of the companies in the Java market the threat driving collaboration was of course Microsoft.

We used to talk about the Anyone but Microsoft club. Arguably the triumph of open source in the enterprise was a result of the same driver. Linux was a means to avoid Microsoft operating system domination. Vendors would push an alternative, and customers would support it partly as means to hedge their bets against too much domination by a single provider.

Today the dominant vendor scaring tech providers is clearly Amazon Web Services.

One facet of today’s Anyone but Amazon coalition is OpenStack. AWS dominance led pretty much every other major tech vendor, no matter how competitive to converge on OpenStack, as an open hedge.

When pondering the implications of Microsoft’s hiring of Brendan Burns the other day, it struck me another coalition is forming, changing the fault lines of the industry. Burns is one of the founders of the Kubernetes container cluster manager project. He was at Google but just took a job as product lead for the Azure Resource Manager. He has already publicly declared he will continue to work on Kubernetes. Kubernetes has also been enthusiastically adopted by Red Hat, through it’s OpenShift platform.

So now Microsoft, Red Hat, and Google Cloud Platform are all now aligned around Kubernetes. While at first glance this new alliance of strange bedfellows might seem to be a response to the rise of Docker and the Docker Pattern – and indeed there is no doubt the enthusiastic growth in Kubernetes is partly driven by concern that Docker will own too much real estate of the new infrastructure world I believe the overarching threat is Amazon.

As Stephen has explained – the biggest competitor to open source is Amazon. There is no doubt that Amazon EC2 Container Services is going to gain wide traction. Amazon can afford to be magnanimous about Docker’s rise in a way other vendors can’t. Docker is an implementation detail rather than a potential existential threat to AWS.

This week another shoe dropped, when Amazon announced it is acquiring Cloud9, the online IDE startup. Cloud9 created Ace, which also powers the GitHub editor.

At serverlessconf recently Amazon GM of Serverless Tim Wagner made it very clear that he sees testing moving into the cloud sooner rather than later. An online IDE is a good place to start – a technology we keep expecting to take off… and keep waiting. Developers continue to choose local machine performance and convenience, which partly explains the Docker phenomenon. Networks are never perfect. On the other hand developers already rely on at least one cloud service – specifically GitHub.

The Amazon move is frankly a huge shot in the arm for the Eclipse Foundation, which was born of an earlier coalition to avoid the emergence of Sun with a top to bottom Java stack as too much of a threat to IBM. Eclipse was the original open source IDE, but has recently started to put together a really nice story around Che, based on software from Codenvy. The Codenvy story is really nice- it’s not really an online IDE so much as Web-based tools portability platform. It manages the developer’s environment, with all dependencies, managing Docker machines and other runtimes. It hides operational details unless the operator wants to see them. Being online rather than on a local machine it’s simple to fork and share environments with other team members. It’s pretty cool. Contributors include IBM, Red Hat, Samsung, and WS02.

Finally it’s worth mentioning the Cloud Native Computing Foundation, which was ostensibly founded to manage Cloud Native open source technology, but also has a role to play in managing the new coalitions. Given it is the home of Kubernetes it’s going to become ever more important as the ABA coalition members collaborate and compete around open source technologies such as Docker.

I talk about the Brendan Burns hire here if you like video

disclosure: Amazon, Docker, Eclipse Foundation, IBM, Oracle, Microsoft and Red Hat are all cients.

Categories: Uncategorized.