Amazon Web Services is not always known for its engagement with open source communities, but there are dedicated teams within AWS trying to change that perception. Listen as Tom “Spot” Callaway discusses building open source culture at AWS with Rachel Stephens of RedMonk. Learn more about AWS’ open source teams, how AWS thinks about both adopting and releasing open source projects, and some of the ways that they assess project and community health.
This was a RedMonk production, sponsored by Amazon Web Services.
Rather listen to this conversation as a podcast?
Transcript
Rachel Stephens: Hi, everyone. Like many a good story, the conversation we are about to have today started on a boat. I met Spot last year at Monktoberfest at our welcome cruise. And I was so very enchanted with our conversation that I spent months badgering people both at RedMonk and at AWS to try to get this conversation somewhere official. And so I’m so, so delighted to have this conversation with Tom Callaway, aka Spot today. Spot, can you please tell us who you are, where you work and what you’re building?
Tom ‘Spot’ Callaway: I am a principal open source strategist at Amazon Web Services, and I am helping AWS to build open source culture.
Rachel: Wonderful. So what I really loved about the story is that, one, I loved that Spot was somebody who was cleared by Amazon to actually tell this story. That’s always a good first step when dealing with AWS. But what I also love is that you have such a great insight into the process of building the culture and the processes. So it’s not just the one or the other, but it’s how do all these things come together? But before we dive all the way in, let’s talk just a little bit, because I think one of the things that’s interesting in tech is that there’s a lot of different approaches that can happen to how to open source things in general. So like some people just have, have a policy in place and people can do what they want all the way up to kind of a formal OSPO, which is an open source program office. And I would love to understand what is the structure and system that exists at AWS.
Spot: Yeah. So AWS, like a lot of companies, sort of started with its open source evolution, thinking about compliance, how to understand when and where open source is used inside its environment and how to mitigate the business risk that is caused by some of that. Now, sometimes people hear that and they go, oh, well, they’re just scared of open source. And I think that at the very beginning that was true. And thankfully it evolved into something beyond that where certainly that is always a part of what matters to Amazon Web Services. But also this is something where they started to realize that there was opportunity that existed in and around open source. And so what you saw is that you have this open source programs office that does this sort of, again, the compliance work. It does a lot of the infrastructure management that is necessary to keep track of open source and to help open source have the platforms and the tools that needs to be able to be managed and be successful. But then you start to think about, okay, now when will we open source our own things and what does that look like?
Because that is a whole different conversation than when you are talking about taking existing open source that other people have built, that other companies and people have put together and use that open source. What does it mean to contribute back to those projects? What does it mean to start your own projects? How can you do this well? And at Amazon, there’s always a lot of importance given to the customer. We spend a lot of time trying to sort of work backwards from where the customer is trying to go and to make sure that we are building tooling and services and enabling them to be able to do the things they want to do in our cloud. Open source is no different. What we ended up doing is we created a secondary team that is focused around the strategic thinking around open source. Beyond just tracking where we have open source and going into how do we participate in open source communities, how do we open source our own work? Where does it make sense for us to join foundations, where does it make sense for us to take an existing project and try to merge it into another project?
There’s a lot of different opportunities and the nice thing is that nothing’s ever off the table. And so we spend a lot of time trying to think big thoughts and work with the service teams that are very influential in their own areas. One of the big things that people don’t always understand about AWS is that because of the nature of how it’s structured, service teams are made up of all these little small, what we call two pizza teams. And each of these teams is empowered to make decisions on their own accord. So it’s not a culture where I can just say, hey, this is the open source rules, you all have to follow it. It becomes something where I have to convince these teams on the merit of doing an open source thing in the right way, which is challenging but fascinating at the same time.
Rachel: Yeah, okay. There’s a lot in there and we’re definitely going to have to tease some of that apart. But let’s start. So it sounded like you have two different open source teams. One that’s doing measurement and one that’s doing strategy. Is that accurate?
Spot: I would say closer to compliance than measurement.
Rachel: Gotcha. So what do you call these two different teams and where do these teams sit in the org?
Spot: Sure. So our open source programs office, or OSPO for short, they sit under an engineering org inside of AWS. And then we have the open source strategy and marketing team, which to be fair, was really only named that so that we could have the acronym “be OSSM” so that we could go around saying that we were an awesome team. So the OSSM team, we actually live under product marketing, which is interesting because almost none of us on the team have any sort of product marketing, or marketing even background. But it’s a nice balance because it allows us to be in a place that provides a diversity of opinions around open source decision making. So we have technical folks who can provide insight and wisdom into when and where the right investments are. And then also we have strategists and marketing people who can chime in to provide advice and relevant data so that we can make the best informed decision as we move forward. We almost never do anything in isolation. We have this split, but we work together all the time.
Rachel: Okay, so engineering, product marketing, and then you also have all the two pizza teams around you who are in kind of engineering product worlds, who are… this is my next question then. You mentioned about things being customer focused, and I definitely understand like two pizza teams focused on like those external customers delivering the best possible products to them. For you all, is it also those external customers or do you think of your customers as mostly those internal teams that you’re serving? Or is it the communities or something like, who are the constituencies that you’re worried about?
Spot: Yeah, that’s a great question. I think certainly we do care about the customers who pay AWS money. I mean, those are always front of mind for us, but we also consider all of the participants in the communities that we care about to be our customers. We consider those communities themselves to be our customers. One of the things that we do as a company is we provide cloud credits to open source communities to help them run the infrastructure that they need to be successful. And we treat every one of those communities as a customer that needs to be taken care of. So there’s a lot of different customers. Our scope of customer is a lot broader than most of those two pizza teams.
Rachel: Fair enough. And then my other question is Amazon then is really well known for having kind of a distributed set of decision making. Different teams can pursue their own general strategies. Different products can kind of overlap with one another. How does this work in terms of having like a centralized strategy team in around open source? Like, are they, are you telling — it sounds like you’re not — but like, where is the guidance that’s happening in terms of what you want teams to be using versus what teams want to use? Are you pushing things to them? Are they bringing things to you, like, what does that back and forth look like?
Spot: Yeah. So I think that, you know, the teams are trying to make the best informed decisions they can as they move forward and we act as a source of valuable information for them as they’re trying to figure out what they want to do with regards to open source. So we can help give them perspective on some of the open source dependencies they’re thinking about taking. We can give them insights into the communities that they are thinking about joining. If they’re thinking about open sourcing something, we can give them data that helps them show what the landscape looks like for technologies in that same space. There have definitely been times where teams have wanted to open source something, and we have pointed out that it might make a lot more sense instead for them to work in an existing upstream project to add the features that we’re missing, instead of creating an alternative that would unnecessarily compete with a project that a lot of our other customers are already very happy with. So we act as, again, a strategic advisor for those teams, but ultimately they get to make the final call at the end of the day.
Rachel: Gotcha. And your response, I want to go in two separate directions. I have to remember to come back to my second point. First point that I want to get to is you talked about, like, helping provide strategic guidance and kind of insights into what’s happening. Can you talk a little bit about what kind of measurements you all use, or how are you assessing the health of different open source projects from different communities?
Spot: Sure. So my team has a dedicated data insights team, and what we do is we’ve built a number of dashboards around open source projects that are critical to our service teams, and it helps them to see how active these projects are. It helps them to identify specific metrics of interest to us. One of the big metrics that we don’t really care about is GitHub stars.
Rachel: Fair enough.
Spot: Because they can be very trivially gamed. There are even companies that are out there that will sell you GitHub stars. So you give them cash and your project gets stars. But we do look at things like the sustainability of a project, and we look at specific factors like Elephant Factor and Pony Factor to help us understand how broad the community is in their engagements. And when we see that a community project is really being controlled and dominated by a single entity, then that’s an area of a concern for us.
Rachel: Okay, sorry. So let’s say more specifically, just defining what Elephant Factor and Pony Factor are, can you just give us more flavor there?
Spot: Sure. So let’s start with Pony Factor. So Pony Factor is basically where we look at any single person who is responsible for more than 50% of the commits to a project in a window. For projects that are not early stage, that are somewhat more mature, you don’t want to see one person doing the bulk of the work because that usually means that person’s on the edge of burnout, that they’re sort of bearing all the load for that project, and that if they got on a pony and rode away into the sunset, that project would have some serious concerns.
Rachel: Okay. I like pony better than, like, the hit by a bus factor. That’s actually much more… [laughs]
Spot: We don’t like to think about that. I think too many of us have been in open source for too long.
Rachel: I feel like pony is a much better metaphor. I feel like we as an industry should adopt that.
Spot: Yeah, I’ve heard a number of terms for that, but, you know, at the end of the day, it’s really just a measurement of, we want to see a large set of individuals who are actively participating in a project such that there are no single points of failure.
Rachel: Okay. And then Elephant Factor means?
Spot: Elephant Factor is basically the same as Pony Factor except for individual. It’s not for individuals, it’s for companies.
Rachel: Okay, gotcha.
Spot: And so, you know, when we see a project and 100% of the commits are coming from people who work for one company, it really signals that there’s a concern about the health of that project, because if that company were to go away or the interests of that company were to change, then —
Rachel: Change a license…
Spot: Yeah, exactly. When would that happen? But, you know, that community is at risk. And that isn’t to say that there aren’t opportunities that come if that action happens. But certainly that’s something that we want to be aware of so that we can figure out what actions we want to take as a potential community participant to invest in a space. And if a community is not open to contributions outside of their company, you often see that sort of very, an Elephant Factor of one. And there’s no amount of banging on pull requests, in some cases, that’s going to change that.
Rachel: Right. Can we talk about… so one of the things that’s happened in our industry in the past few years is we’ve seen a lot of these kind of post facto like license changes where what was previously considered an open source project is now closed source or somewhat proprietary, somewhat open/closed hybrid. And I think that Amazon has sometimes taken some heat in terms of your responses here. So, in particular, I mean, about OpenSearch, I think there’s a lot of people who are really excited to see the elastic project have an open de facto way to contribute, but it’s also one of those ones, right? I feel like, and maybe I might be wrong here, but that was kind of one of your first forays into forking. And I would love to hear what kind of lessons did Amazon learn from that? What was the experience like? And then now as we start to get into this happening more, so we are seeing things like today, you might not have things to say at this point because it happened literally today. But like with the Valkey project, what is happening in terms of things that, like Amazon wants to own versus supporting in the community?
And this maybe ties into what you were talking about before in terms of like, contributing things to the upstream versus like pushing your own things out into the world. Like, how is Amazon thinking about that community membership? And what lessons have you learned from just engaging in the community in various ways?
Spot: Well, so I’ll start off by saying that open source is something that Amazon has gotten a lot of value from. And one of the things that has evolved in our positioning over time is that we have realized that investing in these open source communities does return positive results for us and our customers. And when someone, whomever, decides they want to change the license and prevent us from being able to honor our commitments to our customers, we would rather try to do it in open source than to try to take it internally. And there’s a couple of reasons for that. And I think that one of the big reasons that we did OpenSearch was that we understand that large software projects are better when they are built collaboratively. And if we were trying to build it entirely in house, even at the size that we are at, we’re not going to be as effective as we would be if we were able to build an effective community around that. That brings a diverse range of experiences, technical knowledge, use cases to the table. And the best way to do that that we know of is open source.
And I think one of the big things that we learned with OpenSearch is that we really wished we had had the opportunity to have been engaged deeper in the community that we forked before we forked. And one of the big differences between an OpenSearch and a Valkey is that we had invested in the upstream community around Redis for a number of years and we had a maintainer on the core team. And that meant that we were in a much better position when Redis changed its license. And it meant that we didn’t have to do an OpenSearch for Redis, that there was already an existing community that had some diversity in it, that was just as unhappy as we were about what had happened in the Redis space. And instead of having to build something from scratch, we were able to support and embrace a community that sort of organically came into being and saying, what can we do to help you be successful? And what resources can we offer to you to have the best possible outcome? Because as far as we’re concerned, Valkey is a very logical continuation of the open source project that used to be called Redis and is now called Valkey.
Rachel: Fair enough. I think that all makes a lot of sense. One of my other questions while I’m listening to you talk about this then, is, do you have a general position on foundations and working within foundations or projects that exist within foundations? How do you think about that?
Spot: I think that foundations are a valuable factor in success for sustainable open source projects. I don’t think they are necessary for success for open source projects, but I do think that when a project is ready to be under a foundation, then it can be a huge benefit if both parties work together well. It’s not a magic wand by any sorts. You can’t just join a foundation and have all of your problems go away. But I do think that it signals to the world that you are willing to share control on a project and that you are open to a model that is inclusive, so that other people can feel comfortable investing in that project and then leveling up within that project and getting to a point where they become maintainers. And they do have a voice in the direction of the project. And that’s certainly something that is challenging for most companies to be willing to do on a good day, to give up any measure of control over their work. But we’ve been able to see that with open source that there’s a lot of value that comes from having that sort of a governance model, and foundations really help that be possible.
Rachel: Yeah, I love that. So I think maybe to wrap us up because we’ve covered a lot of great stuff. Can you talk about, you mentioned team members who sometimes want to open source something like, what is the process of pushing something out into the world for Amazon? Is it difficult? My other question is, do people, when they’re contributing as maintainers to projects, do they contribute under Amazon or did they contribute under their own GitHub name? How do you think about just letting your team members contribute to open source?
Spot: Sure. One of the things that has been evolving in our model around contributions from Amazonians is that we have gone from an area of risk mitigation and instead we’re going to an area where we’re starting to encourage Amazonians to participate on their own time. Obviously depends on the team, depends on the role, depends on the work. All of these things factor in. There’s not a blanket ‘anyone can give anything out at any time.’ But one of the things that we’ve built is we’ve built a model called Simple Contributions. And for teams that have determined this is a good fit for them, essentially what it means is that for things that are simple bug fixes, straightforward changes, basically anything that is not a giant feature that we’d be trying to drive into a project, then they have the flexibility to just go ahead and do that upstream pull request or send the patch in and they don’t have to seek any special approval beyond that work. We do have that carve out for large features just so that we can make sure that we do our due diligence around legal. But it has definitely improved the experience for a lot of folks who have wanted to contribute to open source projects at Amazon.
Rachel: Yeah, I think one of the things that’s really interesting is I think Amazon has taken a lot of flak in the community for just open source in general. And I don’t think a lot of people have appreciated how your practices have evolved over the years. And so this has been a really great conversation to just understand how you all see it from the inside and how you’re building this open source culture. Spot, thank you so much for chatting with me.
Spot: We have a lot to improve on still. Like, we have a lot to learn. We’re constantly trying to be better open source citizens. And I think that, you know, one of the things that, that we’re always trying to do is trying to better understand where we sit in the open source communities and every day try to be better.
Rachel: It’s always a good goal. One of the RedMonk mottos is make better mistakes tomorrow. It’s a good one. Well, Spot, thank you so much for your time today. I appreciate it.
Spot: Thank you, Rachel.
