In this conversation, Rachel Stephens interviews Kat Cosgrove – a prominent figure in the Kubernetes community – to discuss her journey into Kubernetes, the structure of the release team, and the role of corporate contributions in governing the project. They explore the expectations surrounding open source projects, the importance of sustainability, and the challenges faced by maintainers in the ecosystem.

Links

• Kat Cosgrove on Bluesky
• Kat Cosgrove on The Pragmatic Engineer podcast: How Kubernetes Is Built with Kat Cosgrove
• The RedMonk team on the role of open source foundations: Amok, Amok, Amok; or, the State of OSS in 2025

 

 

Transcript:

Rachel Stephens (00:12)
Hi everyone, I am Rachel Stephens and I have absolutely no chill today because I am talking to Kat Cosgrove and I am delighted to be here. Kat, you’re a force of nature just as a human, but also you have led several Kubernetes releases, so that’s what we’re here to talk about. Do you want to introduce yourself beyond that? What else do need to know?

Kat Cosgrove (00:21)
Sure.

I am the head of developer advocacy at Minimus, a container security company. I own the Kubernetes release team subproject after I led Kubernetes version 130, Uwubernetes which made quite a lot of people on Reddit mad because they didn’t like the name, which brings me absolutely endless joy. And I would consider that a major career achievement for me. I’m also a technical lead for SIG Docs also in the Kubernetes project. So I’m responsible for the Kubernetes technical documentation and the Kubernetes website.

Rachel Stephens (01:06)
Okay, that’s a lot of things under your purview. Maybe let’s start with how did you get started with the Kubernetes project overall and how did your role evolve over time?

Kat Cosgrove (01:09)
Yeah, I used to be a software engineer a long, time ago. And I learned Kubernetes when I was an embedded Linux developer. I needed to run Kubernetes on a really, really small low memory device. So I learned Kubernetes by learning K3S –

Rachel Stephens (01:31)
Yep, that’s outta Rancher yeah?

Kat Cosgrove (01:35)
yep, it sure is now SUSE, which is very, it’s very cool. It’s a very cool project. I love it.

It is not, however, like the greatest way to learn Kubernetes for the first time. You learn some bad habits around security, but that’s okay. So I was just a Kubernetes user for a while.

I slowly, through social media, became friends with some other Kubernetes maintainers who were being sent to conferences and stuff. And one day, the Kubernetes project decided to deprecate something called the Docker shim.

This was a little software shim that allowed you to continue using the entire Docker tech stack as your container runtime. Docker was the original container runtime for Kubernetes before we had other options. But eventually we standardized an interface that Docker itself didn’t comply with. Because it was the original and so many people used it, we introduced a shim to allow Kubernetes to get at the instance of containerd running inside of Docker.

But that shim was an extra thing for us to maintain. It was kind of a pain. It had other implications elsewhere within Kubernetes, and we didn’t want to deal with it anymore. So we deprecated it. And we grossly overestimated what the average person understood about the relationship between Kubernetes and Docker from a technical perspective. So the internet immediately freaked out.

Software developers who use Docker in dev but don’t actually touch Kubernetes themselves, thought that they needed to learn a new tool. Some people had the misconception that Google still owned Kubernetes in some way and was trying to kill Docker the company or Mirantis the company because there was also a misunderstanding about who owned what in the split between Docker and Mirantis. And I was terminally online.

It was 6:30 in the morning. I hadn’t had any coffee. I saw people who were wrong on the internet, and I took it very personally that people were wrong on the internet. So I thumbed out like 12 tweets explaining the history of the relationship between Kubernetes and Docker, what was actually happening, what the implications were for software developers, what the implications were for cluster administrators, how to tell whether or not you were affected by this, what you needed to do. And it went viral.

And I ballooned from, like, 4,000 followers to, like, 11,000 12,000 followers overnight, which was pretty scary. But I also got pulled in as a Kubernetes contributor for the first time because somebody reached out from the contributor experience SIG and said, hey, you explain this really well, can you write some content for us? So I ended up writing the Don’t Panic blog, kind of re-explaining what went on and the FAQ and the history of why Docker was a valid container runtime in the first place, and the technical instructions for what you needed to do to determine whether or not you were affected and fix it. And from there, I just kind of like didn’t leave. I was an effective communicator. I was very online. I had the time to write content. And I knew a lot of the maintainers. So Jeremy Rickard, who is a SIG release lead, asked me to shadow on the communications team for, I think it was Kubernetes version 1.22 or so. And then I never left the release team. I shadowed comms twice, then I led comms. I shadowed docs once, and then I led docs. Then I was a release lead shadow twice. Then I led version 130.

And then they made up a new job title for me. Normally we make release leads take some time off after leading a release because it’s very stressful, but instead they just gave me the whole release team immediately after. And now every one of them is my problem. So the release team, the release lead reports to me now, which is fun.

Rachel Stephens (05:29)
Alright.

It’s a lot.

It’s a lot. So the power of being a good communicator and refusing to leave and look at this, the whole Kubernetes landscape can be yours.

Kat Cosgrove (05:44)
Stubbornness, yeah, stubbornness and soft skills. Yeah.

Rachel Stephens (05:47)
I like it. It gets you a long way. All right. So tell me, you mentioned the shadow and the lead program. Tell me about kind of what does that look like from the inside?

Kat Cosgrove (05:54)
Sure. So the release team is made up of four sub teams. Documentation, which is responsible for ensuring that any enhancement that has user-facing changes has documentation. We don’t allow features without documentation into the release. Communications, which is responsible for working with enhancement owners to get them to write blogs for important features. Documentation and Comms also handle some of what used to be the release notes team. That team doesn’t exist anymore, but documentation also generates our release notes and communications gathers release highlights and writes the actual release blog that everybody reads. We also have the Enhancement subteam, which is responsible for collecting enhancements from throughout the whole of the project and making sure that they meet all of their requirements as for, production readiness review and code review and test review and all that stuff. And then Release Signal, which is to make sure that we don’t have any release blocking test failures when we cut releases. And if we do to chase down the enhancement owner and the SIG leads to make sure they get fixed. Each one of those teams is made up of a sub team lead who has previously shadowed at least once and anywhere between four and six shadows. It’s really up to the lead, how many they select.

We also have a Release Lead, and that Release Lead will have four Lead Shadows as well who are designated jobs to go and help the other sub teams with both their actual work and with managerial stuff. Because this is an open source project, but we do have a very surprisingly well-organized structure that can look a lot like a business.

So we have the same kind of people management issues that you would have within an org at work.  We have to fire Shadows sometimes and that sucks and it feels bad and I haven’t had to do it this cycle, but I have had to do it before and it really feels awful. But getting into the release team is an open application. So we open up an application usually about two weeks after the end of a release cycle. It’ll go out, gets emailed to the dev mailing list and posted in the SIG release channel on Slack.

Kat Cosgrove (08:09)
And anybody can apply. You don’t have to be a current contributor. You don’t have to have org membership. We have lots of students that are contributing to or working with open source for the first time. Usually there’s at least one on each sub team. I will say that we get hundreds of applicants. So don’t be discouraged if you don’t get selected. Competition is just really, really, really fierce. So do keep applying. Yeah.

Rachel Stephens (08:32)
Let’s talk about the enhancements coming from the different SIGs. How many SIGs are there in the Kubernetes project?

Kat Cosgrove (08:40)
37 or 38, I think, and they’re broken up into two categories. We have horizontal SIGs, which are responsible for things that affect the whole project. So SIG security and SIG docs and SIG release are horizontal SIGs. So we touch stuff that all of the other SIGs own. Then we have vertical SIGs, which own areas of code. So SIG Node or SIG Auth or SIG Storage or API Machinery or whatever, they all own, specific areas of actual Kubernetes.

Horizontal SIGs can own code too, but they own things that touch the whole project. And it also means we have a little bit of weight to tell other SIGs how things are gonna go and how they have to behave. We don’t outrank them specifically, but it’s our job to be, ‘no, this is how the project is going to do this in this specific area.’

Rachel Stephens (09:32)
How many maintainers are there overall?

Kat Cosgrove (09:37)
Weeeeel

Rachel Stephens (09:43)
Okay, give me a ballpark. You don’t have to be exact.

Kat Cosgrove (09:45)
150 maybe.

We would classify a maintainer as somebody who is in an owner’s file. So we don’t actually technically have maintainers, we have code owners. So it would be anybody who is a SIG chair or a technical lead. I think technically anybody who’s a milestone maintainer. So anybody who has the ability to add a release milestone to a PR.

And then anybody who’s on the steering committee or the security response council would be in owner’s files. So I think about 150 at any given time. We prune them every year, and we just ran a prune. So we probably lost 20 or 30, but we also have probably gained 30 or 40 over the last year. The release team is a pretty common source of contributors, at least. Becoming a maintainer takes a long time. Getting on the release team is a pretty common way to get org membership and thus be considered a contributor.

Rachel Stephens (10:45)

Gotcha, fascinating. So I think in terms of areas where people are doing the most enhancements what is the most active part of the project?

Kat Cosgrove (10:58)
SIG Node. SIG Node, and it’s not close. Sometimes I think SIG Storage is maybe in second. I can look at the enhancements tracking board and tell you how many they have, but it’s usually most of the enhancements in a given release come from SIG Node – it’s significant. I think they had like 46 last cycle or something out of, like we had 70 something? It is the bulk of enhancements come from SIG Node. Yeah.

Rachel Stephens (11:30)
Okay, gotcha.

And so you mentioned that students are definitely involved, but in terms of maintainership and people who are on governing boards and SIG boards — how does that kind of look in terms of corporate interests?

Kat Cosgrove (11:44)
So most of them do have day jobs. The woman who used to be my co-owner of the release sub project was made the co-owner very shortly after graduating from college. So it happens. But yeah, most SIG leads, do have day jobs. The steering committee definitely has day jobs, but companies can’t buy those positions.

Rachel Stephens (12:09)
For sure, for sure.

Kat Cosgrove (12:15)
It’s not for sale. We actually have provisions in the steering committee’s charter that prevents more than two people from the same company from holding steering seats so there are some protections in place to ensure that, like Red Hat, can’t swoop in and take control of the project by virtue of having enough voting members to get everybody on steering from Red Hat. But yeah, most most maintainers do work somewhere.

I don’t know that most of them are being paid to do open source as their full-time job though. Certainly many of us are. I’m very lucky in that doing Kubernetes is part of my day job. Certainly it’s a thing that looks good on your resume, can’t lie, companies do want to hire you just because they want to say that they have a maintainer employed, but that usually kind of, that comes after you get the position, right?

Rachel Stephens (13:07)
So does the seat sit with you or does it sit with the company?

Kat Cosgrove (13:11)
With me.

Rachel Stephens (13:13)
So if you left the company, it’s still your seat. Okay.

Kat Cosgrove (13:15)
Yeah. Yep, it is still my seat. Still my title. It is me entirely. Same with the steering committee. The governing board behaves differently. The governing board sits above Kubernetes, the CNCF governing board. Those seats are assigned to companies. Aside from the community seat, which is elected, but otherwise, like, Microsoft has a seat on the governing board, or Google has a seat on the governing board, and then gold members all vote to select a company member to go have the gold seat on the governing board, at the Linux Foundation at least. But those are very, very corporate seats and very corporate titles. They’re not open source maintainer titles, but mine. I was unemployed for six months and I still had the title: still mine, still doing the work.

Rachel Stephens (13:57)
Still yours. Gotcha.

Okay, so I’m glad you brought up Google because it’s a segue into one of the things I wanted to ask you about. So you were recently on the Pragmatic Engineer podcast,

Kat Cosgrove (14:16)
I sure was.

Rachel Stephens (14:20)
-which was great. Loved it. You and Gergely were discussing Google’s influence over Kubernetes before they donated it, after they donated it. You made so many great arguments about just the benefits of open source in terms of cheaper to maintain, 24-7 work cycles, all of these wonderful things. But then I have a question for you about one particular quote. So you said,

“if your project is good, people will come contribute to it.”

Which I think is really great for a project like Kubernetes, but I want to tie this back to the NATS/ Synadia kerfuffle from earlier this year, where one of their complaints was that they were not getting the contributions that they thought they were going to get with the project when they donated it. So I just want to hear, in your view from being inside of an open source kind of … Borg — that’s a pun that I didn’t intend to make.
In terms of kind of seeing things from the inside and kind of the disproportionate way that maintainership and contributions kind of flow through different projects, just talk to me about that. What do you see?

Kat Cosgrove (15:16)
Sure.

I think the number one thing people need to consider before they open source something is you’re not Kubernetes.

You’re not, it’s not reasonable to expect that you’re gonna immediately hockey stick the way that Kubernetes did. I don’t think NATS was going to do that. It’s not revolutionizing the way we literally build and deploy applications which are increasingly the bulk of our lives when we’re not sitting in the dark reading a physical book by candlelight, you know? So you need to temper your expectations for of all. It kind of gives me the same vibes as startup founders who try to run their startups like it’s Google because they came from Google and it worked at Google, so surely it will work at my 12 person startup. No, will not.

Kubernetes is unique in that we don’t really struggle to find new contributors. We do struggle to keep them around for a long time, but everybody struggles with that. But we’re the second largest open source project in the world behind Linux. You can’t compare yourselves to us. So if you’re going to open source something, temper those expectations.

People aren’t going to show up out of nowhere. There is still some legwork you have to do. You have to consider marketing is part of your open source project. It can’t just be engineering, engineering, engineering. You need a PM, you need somebody to handle marketing, you need a technical writer, and you need these people to do work for free, and so at first that’s all going to be you.

But if you’re donating to a foundation like the CNCF or Apache or Eclipse or whoever, it doesn’t matter – it should be the same for anybody – you also need to understand that that’s a permanent action.

There is no universe in which you should be able to donate a project to a foundation. Take those free marketing and technical writing and PM resources and then yank it back when you don’t become famous overnight. You know? So that, that just, it felt that whole thing felt like it was, it was undermining the spirit of open source in general. This isn’t like a company where you can just go public and then get tired of it 10 years later and take it private again. It doesn’t work like that.  You don’t have any ownership of that project anymore. It belongs to someone else and you can’t yank it back just because you didn’t become a celebrity.

Rachel Stephens (17:48)
I would say RedMonk shares your spirit of the nature of open source. I think that’s how we view the ecosystem as well. And we’re dangerously close to talking about licensing rug pulls so maybe we should watch out because nobody wants that.

Kat Cosgrove (17:56)
Ugh, I don’t like the licensing rug pulls either

Rachel Stephens (18:16)
But let’s talk about open source sustainability more broadly then. How do you see us as an overall ecosystem making this just an easier place for projects to thrive and for maintainers to thrive.

Kat Cosgrove (18:20)
Yeah.

I know that we’re all tired of hearing about maintainer burnout as a thing; every conference ever has a handful of talks on maintainer burnout, but it is a real problem that is not going to go away, that is in fact just getting worse. I own the release team sub project alone now because my co-owner burned out and that sucks. So now there’s more work for me and I will replace her eventually, right?

I think that there is a moral obligation for companies that rely on open source software to make money with their paid products to give some of that back. And I don’t care whether it’s in the form of cash directly to the project’s maintainers or if it is in the form of donating engineering time as maintainers. And I do mean full-time engineers, not you give your engineers an hour on Fridays to work on open source. I mean, you hire an engineer whose job it is to help maintain this thing that you make money off of.

I think that that needs to be talked about more seriously. It’s it’s common at big companies, right? Like Red Hat and Microsoft and Amazon and Google all have employees that don’t do anything but open source all day. And that’s that’s great. But startup unicorns are not so much contributing back in the same way. And I know you got to your bag, you got VCs to keep happy or whatever, but we can unplug your business and you’re accelerating that.

Rachel Stephens (19:52)
I think for me the other thing is that the CRA is coming around the corner. I think it is very much going to change the way we all think about those investments and our liabilities back to projects.

Kat Cosgrove (20:07)
It is. I don’t think it’s going to hurt maintainers directly because we have no obligations under the CRA. I as a Kubernetes maintainer do not have any legal exposure there, but somebody using Kubernetes. Yeah.

Rachel Stephens (20:14)
Right. But the burnout side for me is if I as a vendor start sending demands to all of my projects to help me be in compliance, I think that maintainer burnout is going to be rough. And I think we’re naive to assume that that will not happen.

Kat Cosgrove (20:30)
That is going to be a problem. Yeah, it’s going to be really rough.

Yeah, that’s gonna happen. It’s gonna happen. I think there will be a slight delay where everybody’s like, ‘no, it’s fine’ And then all of a sudden we’ll get crushed with a wave of issues. And I think it will, unfortunately, largely be from people who have never really interacted with an open source project before. And so they won’t know how to talk to us and they’ll be mean to us.

And our response to people being mean to us, at least my personal response to somebody being mean to me or my team on an issue is to shut it down immediately. I’m not going to do whatever you’re asking for no matter how reasonable I act I think it is because you’re being a jerk and I’m gonna leave it and let somebody else pick it up. So it’ll get dealt with probably but there will be a delay because I’ll see it first and I will ignore it because you were mean.

Because that is how I protect myself from burnout, by not interacting with people like that. And I protect my team from burnout by not making them deal with people like that. So yeah, that’s going to be a serious issue. And I really, really, really hope that companies listen to, I’m not the only one saying stuff like this, that they listen in advance. Don’t do that.

Rachel Stephens (21:37)
Yes, everyone, this is your warning for 2027.

Don’t do this.

Kat Cosgrove (21:45)
Don’t do that.

Don’t do that. Because look, it’s open source, baby. PRs are welcome. If there is a problem in something that you’re relying on that’s open source, open an issue, fine, with all of the detail that the project asks for, but then assign one of your own engineers to go work on it. You can do that. And you should do that because we don’t have an obligation legally, morally or socially to do anything you ask us to. We don’t even have an obligation to gracefully deprecate a project. We can just abandon it. You can alert us of a critical CVE and we can just ignore it if we want to. That’s okay. It would be better if we didn’t, right? And the Kubernetes project doesn’t, but any of the probably hundreds of teeny little projects you rely on, they can and will do that. So.

Rachel Stephens (22:40)
All right, well, 2027 is gonna be a bit of a shit show. Excited to have you back on and we’ll talk about it then.

Kat Cosgrove (22:47)
yeah. ⁓ yeah, bring me back then.

Rachel Stephens (22:50)
Well, Kat this has been an absolute delight. Thank you for giving me some of your time today and let’s talk again soon. All right, bye.

Kat Cosgrove (23:00)
Always happy to.