One of the most pressing issues facing open source today is maintainer burnout. The issue is systemic, and seemingly without obvious solutions. In this talk, open source maintainer Ashley Williams looks at the burdens of maintainers, the economics of open source and asks the question: is maintainer burnout a market failure?
Transcript
So my talk today is about the dogs. That’s what obviously Steve was talking about, very near and dear to RedMonk’s heart. Just kidding. So yes, my journey here was pretty ridiculous. The title of my talk today is going to be 2 wolves. It’s not necessarily about my dogs, but I did drive here from Austin, Texas, with my two pups here, and we spent two days racing a hurricane and losing. So needless to say, I like a little bit of struggle and there’s certainly a lot of struggle in this talk.
So at the 2015 Linux Foundation collaborator summit, Jim Zemlin almost declared that it was the golden age of open source.
He celebrated things like we have finally gotten our Red Hat 2.0 Hortonworks. They were apparently the Linux kernel itself, more contributors than anyone else, and I think most importantly while they were not a sponsor of Linux Foundation, even Microsoft loved Linux and this was huge and we were on the edge of what appeared to be an open source-first software paradigm, the cloud. And so everything seemed super, super great.
Now, you’ll note that in my previous slide, I said “almost,” because, there was also a massive problem that had started poking its head in 2014. Who’s familiar with this logo? Yes. And so this was actually a massive challenge to the Linux Foundation and their message of sending open source into every commercial enterprise they could.
Largely due to the fact that it appeared to be a failure of the very classic Linus’ law, which is to say, “with enough eyeballs, all bugs are shallow.”
And so, if this was try how was it the case that we were having this catastrophic open source bugs that were wreaking absolute havoc on the economy? And what Jim said, is that even in open source, we can have market failures. He said there weren’t enough eyeballs.
And this is in 2015.
So today, I — Steve was like come talk about open source maintainer burnout, and I’ve been asked to talk about the topic a lot. I’ve been asked to talk about open source sustainability a lot, and you know, I think we’ve talked about these concepts for years. I’m starting this talk citing 2015, but the discussion of these topics why has gone way before that. In 2010, there were concerns about there not be enough containers, not enough contributors, this is a tired topic and I think part of the reason is we were framing it incorrectly. And so, despite not being an economist, I’m hoping that we can take a look at this, not from the perspective of we have something we need to support it and sustain it, but that actually we’ve created a market and that market has failed.
And that we need to, if we want to fix this, address it.
So speaking of failures, in 2015, I joined a company — and again, many people were often confused about this, I joined a company called NPM Inc. They used to have a slogan called nice people matter which became very uncool after the Black Lives Matter happened and it suddenly seemed very insensitive. This is me, sitting in front of that sign in the office on my very first day. It’s probably the biggest smile I had in that entire job, because it was very, very complicated. I lived through many of the things.
I’ve actually realized I’ve been aging out of people’s Who remember Left Pad. Does anybody remember Left Pad. So I worked at this company and I was super excited. I had spent all of my career up to this point working on open source, I had worked at the Flatiron School, I was teaching bootcamps and when they told me to close my curriculum, I quit. I was an open source Diehard and this felt like I get to go work in bed right now.
So I used to give this talk and people freakin loved this talk and I always called it the big numbers are big talk. It’s a graph with a rolling 28-day download and the numbers are in the millions and I was like, this is what people in the business world call a hockey stick curve and everybody was like woo! We love this. And so this was a very popular thing. Everybody loved the fact that JavaScript seemed like it was growing, and there was so much success and especially in counting. But it really remembers what you count, because even though big numbers are big, small numbers were also incredibly small. So this is the GitHub contributor insights page from the MCM CLI that I took yesterday. I don’t know how many people know the people who are listed here, but of the top six, four are people who worked at NPM in 2015 when I did, and this is still the core package manager for JavaScript today.
Certainly there are many more, and we could maybe guess about why. But for all of those billions of downloads, the number of people maintaining that CLI was usually around three.
Which is incredibly small. And the number of people who were managing the NPM registry was about the name.
And so despite the fact that NPM looked like it was growing, it was growing very asymmetrically, and this was a very big problem.
So as many people happen to know, NPM ended up not succeeding as a company. It got acquired by GitHub, which was acquired by Microsoft. And not a lot much more was said about it.
You may be worth noting that again, I took that contributor insights page, I took that screencap, two days ago, and so being acquired by a large company did not dramatically change the number of folks who were working on that project.
And if you talk to the folks who are in charge of that project today, they will tell you that that is absolutely the case.
And so NPM as a company raised around $20 million and it wasn’t enough.
So this is around the time where I think a lot of people with joyous contempt, are gonna tell me this. A lot of people. They love to say it. Who has tweeted this? There’s multiple people in this room who have tweeted this.
All right. I think this is deeply unhelpful.
[laughter]
So, I’m gonna take this one up here: How companies misunderstand open source. OK, I’ll be honest, I haven’t met anybody who started a company to try and make their open source project their day job who thought open source was a business model.
I’ve never met that person. I don’t think they exist. Sure, maybe, there’s like a couple, but it’s certainly not the rule. The people who are showing up and trying to turn open source projects into sustainable companies already know this. They know it kind of desperately.
And so when I hear this, I go, why do people say it? Like, what are you trying to communicate when you say something like this?
And on bad days, I think they just want people to stop trying, which I think is terrible. Because, like, oh, we want people to stop trying to make businesses out of open source projects.
And it makes me wonder, like, what do they think the future for open source is if we do that
On other days, I suspect that they are potentially trying to say that we should come up with better business models for open source.
And I think that that’s more constructive. That being said, I haven’t seen any effort in that direction yet from this group.
But when people say this, it makes me like deep Liangry. And I’ll admit that the people who say it, they’re never — they’re never maintainers. It’s never somebody who’s staying up late at night answering issues.
Is there anybody here in the audience who maintains an open source project actively right now?
That’s a really low number.
It’s lower than the number of people who said open source isn’t a business model.
So what about it? What are we gonna do? So yes, open source is not a business model, but I think undeniably it is business. And it’s pretty serious business.
And I know this, because someone at Harvard Business School said so!
[laughter]
So this is a screencap of a paper by — everyone attributes it to Frank Nagle, there are three people involved in the paper, but in this paper they actually did the work of trying to quantify what the value of open source is.
And this is really important, because it’s funny. Today we talk about software supply chain and things like that. The estimated somewhere between 70 and 90 percent of all software in production today is open source software today. We can kind of stop saying open source software now, it’s just software. The components of software today are basically mostly open source. And they’ve validated that number today, with a number that they know is growing in value, at $8.8 trillion, and open source, it’s not a business model, but it’s worth $8.8 trillion, the amount of which is captured is functionally zero.
So, in 2020, I quit any job to serve the Rust foundation. Has anyone heard of the Rust programming language?
AUDIENCE: Woo!
So there was a Mozilla thing and I was on the core team and I’m passionate about bylaws and I was like, sign me up, I want to do this. And of the things about foundations, is you’re in control of sales. And so it suddenly became my job to sell Rutt. And I personally sold more than $2 million worth of Rust to very big companies. And that’s a really interesting experience.
[applause]
Oh, no, don’t clap for me. Because here’s the thing, open source isn’t a business model, but this was the easiest sale ever. Shockingly easy.
And so this kind of begs the question: What are companies buying when they join open source foundations? What do people think? I was going to say, hint … it’s not nothing.
AUDIENCE: Input, scheduling, marketing.
>> Interesting. So I’ve heard a couple classic ones, so people really don’t, when they say influence is one thing that people said. Foundations love to say that no, the project is separate. No influence. I also heard marketing. I’ll be honest, I mean, shout out to the LL, because I think they’re pretty good at marketing, but I think in general open source foundations are actually pretty awful at it. So if that was their primary model, they have no business selling what they’re buying. So I have a different theory about what they’re buying. But I’ll start out with a theory that I think came with RedMonk, with this post by James, which I had many feelings about. That will not be in the talk.
[laughter]
But I think one of the big things that they sell is trust. Like, the point is this article in many ways was the fact that you live in an era of open source licensed rug poles and there’s uncertainty, and you know, when can companies use open source, they’re making a very big investment and they want to protect that investment. And so this trust is something that they can buy, but I ultimately think — and this is maybe a little controversial — but I think most memberships in open source foundations are selling insurance. So this is an expenditure breakdown in 2023, of the Rust Foundation. It’s relatively easy to read, but you can see community grants are kind of the small chunk and then infrastructure, security, membership and — so the vast majorities are stability, security and legal. And those are the things that many people are nervous about when they’re trying to use open source and in many ways foundations are purposely designed to protect companies when they invest their business models in open source.
And this isn’t necessarily bad. But I think it’s worth calling out that this is what they’re paying. And the thing that I think is interesting about this, is that many foundations have just voluminous amounts of content describing what they call the ROI of open source.
Tons and tons of PDFs of this, right? And even though open source is not a business model, open source has been a business strategy for proprietary software companies for more than a decade now, and the foundations are purposely designed to make this the most paved possible path, ever, and they’ve done it super-successfully.
Just super-super-super successfully.
They even have graphics like this. Has anyone seen this? Yeah, I know you’ve seen it. So I gotta admit, when I read this as a maintainer, I don’t feel good.
It’s OK, you can laugh.
But it’s true. Like, look at this.
31% of respondents reported that paying for equivalent software functionality would incur 4X the cost.
What this graphic says to me is that the number one reason — and many, many studies have shown this, there’s infinite numbers of research papers on this — the number one reason that companies use open source is because it’s cheaper.
Why is it cheaper?
We’ll let that linger.
So because corporations have suddenly picked up this incredibly cheap new way to develop software at a rapid rate, there have been some new requirements.
And unfortunately nowadays, producing open source software has become exponentially more expensive of the maintainers. Who’s discovered that they have to do more work now than they did two years ago?
It’s a lot more.
And it’s largely to this. So this is the cartoon everybody knows. And this is the really long mouthful of words that people like to talk about now when we talk about open source.
The supply chain.
How many people here are familiar with the supply chain? Excellent. All right. So we understand that it’s a metaphor, it comes from the manufacturing world, which has very different dynamics. People do not use, you know, metal pipes or roos, you know, because they’re cheaper. They pay for them.
And this is a massive market. Absolutely massive market and it’s growing and you can tell that I’m fundraising right now because I have these good stats.
[laughter]
But yeah, fastest-growing segment of the dev-led landscape, all right? The attacks are going to cost businesses nearly $138 billion. That’s a lot of money. These are numbers that VCs make bets on. Because they’re big.
But there’s a problem with the metaphor. I imagine that many people in this room are familiar with it, is that the supply chain metaphor immediately fails, because there actually is no supplier relationship. There is no relationship at all. So how many people have read this blogpost?
Oh, I’m gonna read the spiciest part but I strongly recommend that you read this. In sent of ’22, this fantastic paper was written, called I am not a supplier where he details where this metaphor comes from and talks about why this metaphor falls apart functionally immediately, because there is no supplier relationship between folks who are consuming open source and the folks who are producing it.
And I’ll call this out, because one of the ways you can create that relationship would be to pay maintainers, but many people are not. And so this is from Tidelift’s recent paper, even the fox who think maybe are paying maintainers, open source foundations, 3%.
The number one at 25%, is donation programs, and a very large percentage of that is open source maintainers giving their donations to other open source maintainers in a circle that slowly siphons off fees until it is actually negative.
[applause]
So this is a pretty sorry state of affairs if we’re going to apply something like a supply chain metaphor to it. And Thomas says this and the word choice is toe perfect, I simply have to read it: So all of your software supply chain ideas? You are not buying from a supplier. You are a raccoon digging through dumpsters for free code. So I would advise you to put these rules in the same dumpster, and remember, I’m not a supplier, because the software is provided “as is.”
Or is it?
So there is something happening in Europe. There’s a specter haunting Europe, called the Cyber Resilience Act. How many people have heard of this? All right, cool, so the goal here — and Europe is good at this, and I hate to sound anti-regulation, but it’s gonna come up — the goal of this act is to move liability from the consumer of software delivered goods, to the producers of it, and they have introduced some pretty strong requirements as well as some pretty hefty penalties to folks who make software that’s put on the market in Europe.
So the software is provided as is, kind of. This rule actually changes things and it does apply to open source.
Now, when the first draft of this came out, some foundations heard about it, and they wrote this letter. I’m not going to read this out, but they were like, hey, I think this is going to have a chilling effect on people building open source software. You really should have talked to us. We are the experts in this. Bring us to the table. We want to fix it.
And they signed off by saying this: They said, the undersigned organizations collectively represent the governance of much of the open source software which industry and society rely on.
And they were able to carve something out. So open source foundations can now go under this new clause called open source stewards, but they carved it out kind of narrowly and it turns out that most open source foundations have no relationship with them and the foundations didn’t carve out a spot for them and so individual maintainers who receive some amount of recurring money from, say, a company are actually going to be held to higher restrictions than the open source foundations. Which is really unfortunate. The foundations took care of themselves in many ways, but again, the vast majority of open source is single, one-off maintainers, and they currently don’t have any real solidarity or any real voice at any meaningful table.
And the reason I know this, is because I’ve been a maintainer.
And so I’m going to kind of wrap this up a little by talking about a project that I really love that I developed and I’m going to get sad about it.
So when I left NPM, I went to Mozilla and I got what I thought was really my dream job. I got hired to work on the Rust WebAssembly tool chain and this was really exciting to me and it was funny — they hired me because they were just like, how do you publish stuff to NPM. I didn’t realize that you could just like put anything up there, but this was great and I think — is there anyone who was used wasm-pack here? Yeah. Sorry in advance. You know why.
So I was super-excited about this project. I love this project. Unfortunately my contract with Mozilla was only one year because Mozilla was obviously going through some struggles and WebAssembly was also having a bit of an identity crisis. People realized that you couldn’t make money from client-side server. Remember that open source paradigm shift. Weirdly everybody was just putting proprietary software up there and charging for it, I don’t know how open-source-first that is. But they were like, yeah, you can stay on and be the one person on client-side WebAssembly and I thought that sounded awful, but I was like, I’m going to keep this project, I’m going to keep working on this project.
I fell in love with this project.
But it didn’t work out. I had to get a job. I went to CloudFlare to work on their WebAssembly tool chain. I actually integrated the tool chain, but it didn’t align with the business goals for me to be spending a ton of time on it and I was doing nights and weekends, it was wrecking my health and I was just devastated, so after a while, a enough issue follow-ups, like, is this maintained? Are you gonna fix my issue? I had to say that I had to give it up. And this is the worst part, too, is I’m on the Rust core team at the time. Ostensibly this product is governed by a working group — how many people here are in open course governance, I’d be curious what your perspective is, but it always looks more complete from the outside.
The biggest problem here is that the Rust WebAssembly grew — had just completely fallen off the map. It looks like there, but otherwise there was nothing happening and you know, the Rust core team didn’t have any resources. We weren’t currently under a foundation. I personally don’t think that it would have changed anything, but I actually had nowhere to put this project. Like, there was no one to take it.
And I frequently asked folks, like, hey, if you’re interested, email me, but remember, I worked at NPM and there had been so many software supply chain security issues with folks transitioning, authors and publishers and this was even before we talked about the supply chain security thing, that I was trying to be really cautious about it.
And so ultimately what I decided — and despite the fact that the zeitgeist at the time is that people were owed the continuance of an open source project.
I said no, you know, I’m just — I think the thing to do is to not give it to anyone. Like, it’s just gonna sit here, and you can fork it if you want. And no one did.
But I was incredibly sad, and the worst part about this is that it actually kinda haunted me for a while. So because I’m a — I love pain and struggle, I went and I Googled some of the Hacker News comments about me and my treating of wasm-pack and decided to put them here and read them out loud to everyone, because I hate myself, but yeah, so … Yeah, I love this one, which was just she acted incredibly disrespectful toward the Rust Wasm team. People don’t remember the beginning. They only remember the end.
This is a screencap of community insights since I’ve been gone. I took this two days ago when I took the other one.
Nothing’s really happened. But here’s the weird thing. So I’m at my own company, we do open source, I see this project in people’s CI workflows all the time. All the time. People continue to use this to this day. And so in many ways, it feels a little bit like this project is haunting me. But maybe one day I’ll bring it back up. But for now, it just sits like this.
And people keep checking in.
And so when this happened, it was uniquely poignant for me. Does anyone know what this is? So for folks who don’t know, this is a screencap from a listserv where the XZ attack was first announced. The XZ attack, it wasn’t a bug. It wasn’t a bug at all. Or it wasn’t a technical bug. So what happened with XZ is that there was a single maintainer on this project, and this maintainer had an experience rather similar to me, where he kept getting all of these issues and he couldn’t keep up. It was a hobby project for him. And people said pretty awful things. Right now you choke your repo. Why delay what your repo needs? But here’s what he did that I’m kind of glad I didn’t do, but I don’t blame him, because I understand why he did. He got bullied into giving maintainership to somebody else.
And that person installed a backdoor into XZ, and it’s so easy for that to happen.
And reading these emails, it just felt terrible, because I knew that if he hadn’t given the maintainership, he probably would have gotten these emails for years. Because I did.
And his project is way more prolific than mine was.
So what about this? What do we do? I’ve told a couple stories but I’ll be honest, I’m mad about it. I’m, like, really mad.
I’m mad the way it seems like everyone else is mad when someone changes a license to a business source license.
[laughter]
So the thing I want to say — and I say dirty, because people love to be grumpy about money in open source — but the dirty secret is that there’s an immense amount of money in open source. An immense amount. For folks who don’t know what a 990 is, google it, because you can get some numbers and I’ll let you look them up yourself. But I’m telling you now: There’s an immense amount of money. The problem is that it’s not evenly distributed. So a lot of people don’t usually think about this when they think about in a market, but the health of a market its whole purpose is to distribute resources efficiently. So when I look at how much money there is in open source, and then how much struggle theres, I can’t help but think this is a failed market.
So I think we need to stop using this term, “open source sustainability” and we need to start saying, this is a market failure, because that sounds a lot more serious, right? That has consequences. If somebody kind of doesn’t do a lot of charity, that’s Locke like, OK, but market failure, you’re like, OK, because market failures have side effects. Not just on the demand side, but on the supply side.
And I think that’s what we need to be paying attention to.
So with all these license changes, all this discussion of, open source is not a business model, I saw this om come across my LinkedIn feed which has just been a torture device recently and honestly it broke my heart. I don’t know if anybody else here had a visceral response to seeing this, but the idea that open source should be celebrated as a privilege of successful businesses made me mad!
Like, extra-mad.
Can I spent my entire early career trying to undo open source privilege.
The original goal of open source was to, like, be good for the world. It wasn’t like being good for making money. It was that it is the right paradigm to protect consumers with software.
And the idea that now it’s actually good and we should all believe that the only people who get the opportunity to do open source are people who have successful businesses is like actually disgusting to me. It makes me, like, very, very angry.
If we want open source to be the future of software, it can’t just be for people who have successful businesses. It can’t.
And the idea is ridiculous.
And so when I saw that I was like, we have strayed so far from the life, I remember teaching — teach it early, let’s learn in the open, like, let’s do all this.
Like, open source is supposed to be the way that we want. It’s not the reward. It’s the path.
So to finish out the 2015 collaborators summit address, Jim Zemlin said this: And it was interesting to me. He said: We don’t want to create an open source welfare state. We don’t want to ruin the market economics that make open source so effective. Now, I’m not here to say that any one person has ruined the market economics, but I do feel confident in saying the economics are pretty ruined at the moment.
So in that talk he announced something called the core infrastructure initiative, and it started out with an initial funding of $6 million, so $2 million a year for the initial raise. This initiative became the OpenSSF, and based on the 2023 annual report that I could find, they had 4.9 million, 2.8 million of which is used in grants. That’s a decade of difference.
It’s not enough. The goal of that initiative was to help single maintainers on projects like OpenSSL, OpenSSH. The funding is the same. It barely accounts for inflation.
If this is something that we care about, we either haven’t found the right mechanism, or we’re not paying enough attention.
We’ve kicked this off, we’ve identified that there’s problems, but that same initiative still can’t raise more than enough money to cover five or six projects.
So I think open source is at a crossroads. This is another graph from the Tidelift survey.
And what it’s showing is the age of maintainers. It’s a bit hard to read, but the message is that they’re gettin’ older, and there’s not new ones showing up.
And who can blame them?
So this was like a big thing in 2018, there was a company called &yet that we put on t-shirts that used to say open source is about people. And I genuinely felt that, nice little slogan from NPM, we thought that mattered.
The way foundations ensure open source today is they ensure the projects. We spent a lot of time protecting projects and we’ve spent almost no time on the people. So these are my two wolves for you, and this is what I leave you with today. So I think there’s a struggle going on in open source right now. I think that there are groups of folks who consume open source, and there are groups of people who produce open source.
And I think there is an actual struggle happening.
attention is a zero-sum game and we’ve been paying a lot of attention to one group at the cost of another. And there’s a saying that goes, he’s a Grandpa and he’s describing two wolves with a struggle and his grandson says, Grandpa, which one wins? And his answer is, the one you feed. Thanks.
