There is a worldview that decided certain things were essential to the success of open source software on the internet, namely a “gift culture” and the ability for “many eyes” to make “all bugs seem shallow” (what ESR called “Linus’ Law”). On the other hand, the current discourse is talking about how maintainers are overburdened and undersupported. This talk is about how open source technology gets made, and what social factors of open source development our CATB-influenced worldview made us ignore, leading us into the way the world is today.
Transcript
All right, so, my name is Christopher Neugebauer, I’m going to be giving this talk about but first a content warning or you two, I’m going to be discussing two key figures in open source at length Eric Raymond, Linus Torvalds and I’m going to be talking about burnout. These are sources of trauma in our community, I encourage you to prepare yourself accordingly, if you do feel the need to leave, take a breather, do I need to be louder? Are you good? Yeah, sure.
OK, so, my usual genre is Python comedy talks. A lot of the talks today have been deeply personal. This one is deeply personal for me, too, but I’m going to talk like it isn’t, because I’m insecure like that, I like to speak with false confidence. We’ll see how that goes.
So, no Python comedy today. I’m here to talk about the historical underpinnings of the stories that we tell about open source more generally and in particular I’m here to talk about what open source isn’t. So you’re going to see some cognitive dissonance in these slides. So the talk. The talk as I say, begins with an obvious statement about open source so let’s get it out of the way. That’s that we have a sustainability problem.
So I’ve served as a director on the board of Python Software Foundation. That’s the capacity in which I’m normally talking about here. Where the global venue home of the Python programming language, we don’t write Python itself, we look after the finance and legal governance side of the community. I started back in June 2018 and in July of 2018, this happened.
The founder and then benevolent dictator for life of the Python programming language abruptly resigned his role, saying he never again wanted to fight so hard for an improvement and find that so many people despise his decisions, and that he wanted to remove himself entirely from the decisionmaking process.
And this took place after a long and protracted argument on a mailing list about a two-character syntax feature in the language.
This particular situation is one of the few cases where that abrupt resignation ended well. The core development team of Python were able to band together to discuss alternatives for how to structure the decisionmaking processes of the language and they came together to build something stronger and which is demonstrated itself to be sustainable through multiple crises.
But the subject of maintainer burnout is important. Open source maintainers are stewards of vital digital infrastructure, any piece of which has generally untold and largely undetectable reach into so many aspects of our lives. And it is at this point where I’m practically obliged to share this comic again. You know, our digital infrastructure is maintained by some random library maintained by some random person in Nebraska and is somewhat of a cliché these days, but it is more true than not.
For this year the first time that we know of, maintainer burnout itself was weaponized. The story goes like this. People like to be able to log into their computers remotely. Frequently remote access is the only way we can get into our computers, thanks, AWS. SSH is a tool that lets people log into those computers remotely. X-zed, that is how I’m going to be pronouncing it — I’m Australian …
Strings of data or anything else that needs compression. It’s not part of SSH, its maintainers do not collaborate with maintainers of SHH but maintainers can have the XZ library.
Now, compression algorithms don’t change that much and it makes its demands for XZ were sufficiently low that a single maintainer was able to successfully maintain the project on its own until June 2022 when suddenly a number of people appeared on the mailing list and flooded the list with incredibly rude messages putting pressure on the sole mainer to cede the list for someone else.
And right now you choke your repo.
Dennis H writes: I am sorry about your mental health limits, but the community desires more. Why not pass on maintainership for XZ and a week later, something remarkable happened. Someone showed up to help. As I have hinted in earlier emails, Jia Tan may have a bigger role in the future. He is practically a co-maintainer already. Smiley face. now, the people who see maintainer burnout before, like, this was a remarkably fortunate outcome.
Over the course of the following two dares, Jia Tan became a reliable maintainer and he released two versions of XZ all on his own. Now, I was doing performance his investigation uncovered that these new releases of XZ contained a back door activated only when XZ is loaded into on certain Linux systems.
So Jie Tan was a state actor who was able to cleverly create a backdoor that targeted SSH which got picked up of by distributors by XZ. It was exceptionally lucky that the backdoor was found before the compromised version was found.
A state actor spent two years building up trust with the maintainer, doing reliable work such that they were able to get the privilege to release their compromised version of XZ and subsequent investigation showed that the name Jia Tan had only appeared in the context of XZ. but something else surprising happened in the subsequent weeks. The name Jigar Kumar, the person who accused Lasse of choking his repo, had never appeared before he appeared on the XZ list. Dennis Ens had never appeared, either. To the best of our ability to tell, all of the actors who created stress and demands on Lasse’s time were, if not the same person, certainly associated with the same threat operation.
The operation that harassed maintainer under stress and then another false persona appeared as a savior who served as a maintainer from buckling under the maintainer and it is reasonable to think that we got exceptionally lucky. We did. But there’s a big question here: How did we get open source get to the point where a deluge of people being rude on a mailing list was so normal that it was recognized as something a maintainer just had to deal with rather than being recognized as something so extraordinary that it must have been the action of malicious actors?
[applause]
Because in any other course …
[applause]
Because in any other course of life, this is not normal behavior and it would not be tolerated.
Open source has got to the point where normal behavior is so toxic that literal state actors presenting as toxic people on mailing lists went undetected and could have brought an international security incident upon us.
In his 2018 post, open source is not about you, Rich Hickey said: Not a mode of collaboration and he uses this frame to justify that it is acceptable for the Clojure team to not directly engage with its users in order to produce its development roadmap. Now, his point is well made. That is how Clojure operates, but this space of a disconnect between the development class and user class in open source. Bruce parens and his collaborator ESR, led the collaboration of the open source most large corporations used to determine whether a license is open source or not. Now, OSI remains a vital organization, but they serve quite a small limited direct audience in that respect.
Now, today open source as it relates to licensing is largely a solved problem. Save some odd rebels over the years.
However, it left the question of collaboration open.
Because the focus of open source at the time wasn’t just licensing. The name of open source was coined in 1998 as a result of was done evangelizing new collaboration that he claims was discovered. In his 1997 essay, The Cathedral and the Bazaar, Raymond was best known for writing EMACS extensions.
Now, his development process in fetchmail was inspired by his name observing the process as an outsider and the essay serves more as a commentary about how Linux work than any of his own work. Given enough eyeballs, all bugs are shallow, it’s possible that the mere existence of publicly available source code was enough to make it attributable to fix bugs and to make sure you didn’t forget it he gave it a catchy name and to be clear this thesis is very compelling. I still hear g this phrase all the damn time and it was compelling back then, as well.
The essay was used by SK Evans by Netscape who I’m told were really important at the time, to release their web browser, and that has been seen as a pivotal moment in the acceptance of open source bye-byes and this is despite his essay, homesteading likening it as a gift culture where it’s given away with no expectation of anything in return.
Works may be given away freely. And so in open source, the abundant resource is code, and therefore relevance is reputation, rather than market share.
I found it somewhat confusing and difficult to follow, but it seems to trace back to a discussion in his first essay. In The Cathedral and the Bazaar, openly mocks the idea of egoboost programming.
Peter describes the primary motivation for participation in the sci-fi fan community to be egoboosting or egoboo for short.
Participating in a fan community, going to a convention allows you to demonstrate how good a fan you are and become recognized as a great fan by your fellow fans.
And the corresponding idea here is that open source contributors contribute to open source in order to gain reputation as great open source contributors, entirely for their own personal self-satisfaction.
And he also features any other motivation as inherently a form of egoboosting.
And it’s in this circular self-justification, that ESR shows motivations beyond his own personal experience. I have a bunch of fans in the science fiction fan community and so I went and asked them how their community works and it’s true, the vast number of people there are to participate getting self-faction and a reputation as another fan. But there are also people in the science fiction community who are convention runners, or conrunners, who are people to make the spaces where the community gatherings. And this is where the community begets responsibility as opposed to reputation, and finally there are people for whom network and conventions provider a livelihood, and these are writers, and merchants. These three distinct roles form an interdependent net, would with whose disk motivations all reinforce each other and that’s the key. Egoboosters only function within a space that’s cultivated by the conrunners.
ESR chose to ignore the part of the community that he wasn’t a part of, and the motivation that those people have. But just because you choose to ignore something, it doesn’t mean it’s not there.
And so it goes open source, because even in 1997, Linux resemble the boosters of science fiction, and the main drive behind those businesses was not altruism. There was a requirement towards reciprocity.
Linux is Linux required and still does require frequent customizations to support new hardware and those customizations needed to function atop a base that went through frequent changes.
Now, at the time, corporate communications of other open source is systems rarely if ever result in open source motivation. Including by corporations who separately contributed to Linux and several of the popular ones remain projects to this day.
Under a copyright license — that’s the viral characterization of how the GPL works if you’ve ever heard that before. Now. The GPL originally envisaged only compel open source at the source of distribution. Indeed this was not something that the GPL’s authors actively sought, there’s a social contract there that technically competent patches would get merged back to Linux. It’s not unreasonable to argue that a key value of Linux to corporates at the time was that by having their catches accepted back to the main line they would be be absorbed — this is hardly one fixing bugs for fun, if self-satisfaction doesn’t pay the bills. Instead it’s a want of externalizing costs, which is a great thing for any profit-driven enterprise to want to achieve.
And it’s also fair to say with the benefit of hindsight that the model was a result of code requiring frequent maintenance on top of an unstable base. In a talk in 2017, describes a process called lithification, which is a process by which code imported in various sources accumulates layer by layer on top of unstable bases.
In the absence of a good reason to contribute to changes upstream, most codebases tend to lithify. And one could argue that — and this is like to the if you must make changes all the time, why not submit your changes upstream, especially if someone will take care of them for you?
So this combination of a legal requirement towards the publication of code, as well as economic incentives motivating contribute to main line may be significantly more — especially when self-satisfaction is backed up by whatever the hell this is.
[laughter]
People don’t do kernel development to help nice-guy Torvalds. Linus was the gatekeeper responsible, and his communication style, even in 1997, was widely known to be toxic.
Getting a patch into kernel main line often meant suffering hostile communication, including personal insult. I have friends — and itch acquaintances who endured kernel development and left because Linux’s communication style was a severe burden on their mental health. ESR recognized Linus as a nice guy. He was not. But by characterizing him as a nice guy whose communication style encourages people to want to help him, ESR explicitly told people coming to open source that they should emulate that communication style as it is a norm in our community. It should not have been allowed to become so.
Fundamentally, you either knew a bit about the context in which this book was written and you could probably see the unjustifiable or the inaccurate parts about the book or you could be a newcomer and decided that the book did a good job explaining the norms about the community and sought to emulate it. The promise is seductive. If source code is available, users are all potential codevelopers. It’s compelling and is true now as it’s always been in open source, however, the book employs sleight of hand.
Fully merged the classes of users and codevelopers as one and the same. And further he asserts that merely characterizing a bug is enough to render it fixed.
That is, he makes a jump from the truth that a user can be a codeveloper if properly developed to the idea that every user is a codeveloper. And this is not me editorializing here. He draws the same equivalence himself. The act of finding a bug is a bigger that will than then understanding it, let alone …
And if you’re a, could staring down the abyss of unviabilities but with still millions of users using your product was probably pretty compelling and it didn’t work out that way for Netscape. It took a number of years to produce a codebase that was worthy of being read by the public, let alone a usable piece of software.
By 2003, Robert Glass’ book sewed that many eyeballs fewer bugs is demonstrably false.
And Mozilla was still fixing open bugs that were open for 23 years. Even a million eyeballs is not going to help you.
But this essay was the basis for selling open source, the collaboration model, to the masses. And the message it carried was that all open source developers are selfish and anyone contributing a bug report is doing harder work than the people actually doing the work to keep the project open and that venerated the communication of Linus absolves the reader who’s in you to open source from the responsibility of engaging with the projects in a meaningful way and it gives permission for people to model their communication on open source from someone who prizes themselves on deliberately toxic communication patterns.
To the best of my research, the essay was seen as a curiosity by people. Actual practitioners developing FOSS in the way they always had and far from transitioning to anarchy, Python followed two years later, and as businesses, IBM, by 1999, had 200 full-time paid developers working on Linux-based technology. As best as I can tell, ESR’s work was pushed primarily by people outside open source who saw themselves as ideologically aligned with ASR.
In the 1995 essay, Californian Ideology … primarily arising in and around Silicon Valley. In it they describe an apparently contradictory view that business and communication networks had a role in stabilizing society whilst further restricting speech and enabling individual expression.
One form of this ideology was in the form of social media focusing on broadening individual speech.
And in that context, ESR’s characterization of speech about code that is, in bug reports being more important than actual code contributions, as well as inaccurate description of open source developers as primarily selfish made his vision in full alignment with his Californian ideology, if only the entrepreneurial class could get involved.
So the many products focused on enabling open source development following the publication of the cathedral and the bazaar given how the first one embraced this social media — this is — I want to be clear, I like GitHub a lot, especially these days, but I want to make clear that pre-Microsoft. GitHub emerged from the Ruby on Rails community. This was a community that emerged independently of the open source community from a community that formed from scratch well after the release of Raymond’s works.
And GitHub succeeded. It succeeded for a number of reasons. They had an open market, chosen by Ruby on Rails has a version of a troll system, as well as being extreme difficulty getting up Git self-hosting and Git, unlike competitors allowed both private or proprietary development as well as a land for open source repositories. Their contemporaries tended to focus on one or another.
GitHub’s acquisition by Microsoft, if you wanted to host open source for free on GitHub, your repo needed to be fully public, and being fully public meant that when GitHub had an issue tracking, you had two options. You could either pay for a private repo, or you could have a public repo on which anyone could create or reply to issues.
Speech about code, equal prominence in the user interface in the code itself, opting to host your open source code in the most popular open source platform, which it was, had equal prominence to the code published by the trusted people in the repo, regardless of whether your official bug tracker was hosted on GitHub itself. And to be absolutely clear, when GitHub issues was released … The first time any reference to editing or deleting or otherwise moderating comments appeared in GitHub’s docs in November, 2016. And the result of this, combined with ESR’s implicit instruction is to communicate in the style of a famous asshole when interacting with maintainers is exactly what you would have expected.
It is a platform effect that continues to this day. The vast majority of open source projects communicate in the same way, in public with the GitHub issue tracker and are discouraged from alternative methods of communication, even when other organizations open source communication spaces for you. These norms, that is, if there is an open source project, you must be able to demand the attention of a maintainer, have been cultivated in GitHub’s early days and those norms are expected, even if the project isn’t hosted on GitHub and even if the maintainers are telling them that their project is not about you. In a 2020 book, Working in Public, Nada … potentially giving them towards the deflect that she focuses her writing on stadiums, which are how thee names the types of projects that have emerged from the platform effects of GitHub and in my opinion the distortions that have arisen from GitHub’s early lack of moderation.
Now, stadium is a place where a relatively small number of contributors have to respond to a large number of comments or by reports in their projects in public.
And while the book deliberately censors these stadiums, she engages in the same sort of rhetorical sleight of hand. And in those projects she only engages with ones with a small number of individual maintainers, which do not have formal governments and that primarily collaborate on GitHub and she claims to own this filtering so as to reduce the scope of the study in the book.
And by the time we reach Chapter 5, where solutions are finally presented, Work in Public starts by representing that the only ones that matter to organize primarily in GitHub. That is, an assertion that all open source projects — and she suggests that funding individual developments is in better alignment with how all open source is written. She then strongly implies that funding individual developers is best left to individuals, because larger projects are more attracted to corporate funders, effectively making a self-contradictory argument that its unreasonable to fund projects because they’re already being funded, which we know they’re not.
And on the way to getting there, she rules with entirely representation that governments are maintained as funding subjects. An analysis shows that less than two years later, the German government would establish a sovereign tech fund and in its first year they allotted more than 13 million euros including one stadium project without the backing of a large organization. And the recipient of those grants certainly did not protest that. And finally as the work suggests, as if there are no alternative options, there are very few people who like the idea of managing a level of formal legal entity. That’s fair, I’m on the board of one of them. It’s tough work, the results of that work are incremental and unfulfilling and thankless and starting one is even harder.
There is an alternative to starting your own legal entity. You can organize with other projects. The Apache Software Foundation collects money on behalf of hundreds of other projects which would otherwise be solely maintained by.
The palace project is a small group of maintainers and now they have their administrative organization is covered by the PSF.
Just because a project doesn’t have the capability to seek resources as it’s currently constituted, it doesn’t mean that it’s not possible to reorganize itself just to make themselves more desirable to funders.
[applause].
So Working in Public absolves corporations from responsibility of supporting the open source maintainers that they depend upon and allows developers to enjoy learned helplessness about their ability to receive support handling the demands on their time, when in fact the solutions are right there in front of them if they want to go looking for it. Now, Working in Public has gained popularity amongst content creators. Who, thanks to the work on the platform with a large audience see open source maintainers as enduring the same struggle as them and they’re now communicating a distorted view of how open source works to their audience, another generation of potential contributors.
And so our community is polluted with false narratives about how open source works, by people with no real stake in the projects they describe and with false claims about how to fix it.
So what do we do about it? First we need to understand that works like The Cathedral and the Bazaar or Working in Public are not for practitioners. They stories about the value of getting in open source are not of value to people who are already in open source, but ignoring them as irrelevant rather than highlighting the harm they cause gives opportunities to others to exist as valuable. And that opportunity exists because there is an unfulfilled need. Open source is a valuable field and for people outside our field, they deserve to now our past, our struggles and more importantly they need to know how it works so they can participate properly. And as long as there’s an opportunity for these stories to be told, they will be told, and we could choose to tell our own stories or instead the stories we’ve told by outsiders who do not have to bear the consequences of their work. And they describe falsely a world of open source where people on the fringes either have no consequences for their participation, or worse still, are told that they make contributions that are outsized and more valuable than the people living with the actual consequences of those interactions.
People with this distorted view of how open source works create tools that impose that distorted world view on people who develop open source, and in turn, the communities that form those platforms are shaped by those tools in turn.
Now, just because our communities are shaped by the tools that we use, it does not mean that we have to accept those tools. Other tools are available to us and you can pick and choose the best features from all of them. You can have a fiscal sponsor look after your overheads. And you can still are your code hosted on GitHub. And that’s the thing. By not being realistic about the consequences of options or the options of available, we’re lying to people when we describe open source, that you can contribute to an open source project without consequences on the maintainers, that merely reporting a bug is more difficult than maintaining a project, and that people who reap actual economic benefit from a project are somehow freed from responsibility to care for it. And there are books out that are written by people who actually spend their time in open source and get tech reviewed by people who are in open source. This one, Forge Your Future, by my friend VM Brosseur, is excellent. Buy one for all your friends it is far far easier to recommend books that free the readers from the responsibility to each other, but that’s not how successful open source works and we shouldn’t be selling newcomers on that utopian dream. Because the result of this dream being more important as code and the work that keeps open source alive is that maintainers of projects, big and small, come to think that unchecked bad behavior is normal. And then rightly make the decision that open source is not for them, and then they leave, and they hand over responsibility.
We got exceptionally lucky this year. The attackers who focused on the social fabric of our community will have learnt from their failure and they’ll be so lucky if we didn’t learn from ours.
As to what for newcomers, well, imagine as can be said of the next generation that making bug reports was the important work, that enough eyeballs make bugs shallow, but they actually learned to contribute in a way that sustained the projects they were interacting with. Given more hands, all work is light.
Thank you and my name is Christopher Neugebauer … Thank you.
[applause].
