Software quality and engineering metrics is a big topic in the industry right now, as companies seek to become more effective and competitive, and better understand how their engineering teams are performing. In this conversation Emilio Salvador, VP of Development at GitLab, and James Governor from RedMonk talk about the impact on generative AI on these topics, and take a look at GitLab’s own framework for better understanding productivity – focusing on Task, Team, and Time dynamics.

What is developer productivity, after all? What is team productivity? What does writing code 25% faster actually mean? What if you save an engineer an hour a week – what is that worth? What is breakthrough productivity? What if we could build 10 times as many things as we could before. What are the implications? Measuring, managing and understanding developer productivity requires us to consider these kinds of questions.

The discussion delves into the complexities of measuring developer productivity, emphasizing that it cannot be reduced to a single number. The conversation explores the challenges in measuring performance, including the need for a combination of data sources and addressing cultural differences within teams. Emilio argues that we must recognise development as a team sport, with a focus on matching skill sets to internal transformations, always fostering collaboration within development teams.

This was a RedMonk video, sponsored by GitLab.

Rather listen to this conversation as a podcast?

 

Transcript

James Governor: It is James Governor, co-founder of RedMonk. I’m here for another RedMonk Conversation. And I’m super happy that today have with me Emilio Salvador from GitLab, VP of DevRel and Community. And we’re gonna be talking a little bit about developer productivity and AI. Obviously those things are coming together at the moment. And I think that’s a key issue. So Emilio, welcome. Thanks for joining the show. What exactly do you do at GitLab?

Emilio Salvador: Hey James, first of all, thanks very much. Super nice meeting you and thanks very much for giving me the opportunity to sit down with you and discuss this, you know, super relevant topic like nowadays. My role as you said at GitLab is the Vice President of Developer Relations. I’m tasked with all our community and developer programs efforts. My team is basically tasked with evangelizing GitLab as the most advanced and secure DevOps platform. And I’m also tasked with getting our community to contribute back to our platform GitLab.

James: Okay, great. So that’s the community side. So let’s jump in. I mean, here’s the thing. I have been watching, uh, you know, a lot of, uh, presentations from vendors, from people in the industry. And they’re saying things like, ah, with our gen AI tool, we’re now 23% more productive and it’s not entirely clear to me that that’s a sort of a, how do we get so reductive a metric?

I mean, A, I’d like to ask like, is that how you see generative AI? And, you know, a little bit of a leading question, given that you have been thinking about this, possibly having a framework for thinking a little bit more broadly about developer experience and developer productivity. Do those metrics make sense? And how are you looking to measure developer productivity and how should customers be thinking about the opportunities around productivity and generative AI.

Emilio: James, it’s really great question. And we’ve put a ton of energy and thought behind it. In fact, we just published a white paper on the topic, trying to explain our customers how we are approaching developer productivity in the context of GenAI. As you said before, we all hear all day long is that, hey, with GenAI, people are gonna increase productivity in general, not just the productivity by 50, 60, 70%. Eventually I’m getting to a point where I believe that, oh, I’m no longer needed. So we ask ourselves that question, how should we think about productivity? And as engineers, sometimes we tend to focus on just one number, one metric, one thing that is gonna solve every single problem and a number that we can play with. And the reality, and especially when you think about developer productivity, it is not a single number. It is not a single metric.

I mean, there are multiple elements. Some of them are not even technical to factor in when you think about developer productivity. So there’s been plenty of research on that topic from multiple companies. There’s a couple of frameworks that people are super familiar with and we are coming up with probably a simpler approach to it that helps us think about developer productivity. But I don’t think there’s a clear answer to that problem.

James: Yeah, I mean, I think that, you know, to a point we’ve clearly got people that we, I think, you know, the appliance of science, we stand on the shoulders of giants, clearly the work in and around sort of the Dora report and the later work that Dr. Nicole Forsgren has done, try to understand the opportunity in terms of developer productivity.

I think, I mean, I think honestly where we are as an industry right now, Emilio, is this is definitely something that we all need to think about, because we understand that developer productivity is important. But measuring that and importantly, like deciding what to optimize for something that is an industry that is up is up for grabs, and or something that we need to work on. So, Emilio, why don’t you tell me a bit about how you’re thinking about the value of developer productivity and developer teams, engineering teams, and perhaps something that we mentioned that you’re publishing a paper. What does that say? What should the end user be thinking about in terms of AI and productivity?

Emilio: So our view on that is simple. I mean, we see GenAI as a tool that is going to help enable and enhance the work that developers do on a daily basis. There’s no question about that. Developed productivity goes beyond that idea. If I could do an analogy, it’s as a painter. How do you measure painting productivity? And you do that by your number of paints, someone can produce on a daily basis. No, there are other things to factor in. So we’ve been thinking a lot about this problem, and then we ended up thinking with a simple framework, framework on three different dimensions, three different access. We call it the 3Ts, because it talks about tasks, time, and team.

James: Tasks, time and team. Those are the dynamics that I need to think about.

Emilio: That is correct. When we think about most of the people, when they think in terms of productivity, they always go to tasks. They try to measure individual things by themselves. Right? If you think about developers, people may think of, hey, the number of pull requests, the number of bugs fixed, the number of lines of code, you know, a developer writes.

James: Right. I think it’s very reductive, but we do. And we’ve been talking to enterprises, and they are doing some of that. It’s sort of sledgehammer metrics. Yeah, lines of code. Is that really the best measure of productivity?

Emilio: It is not, but it, I keep saying one thing. The problem is that it’s not only that you have metrics or numbers, is that how you use them. Because eventually if you think about those numbers and you use them to set goals, then people will find a way to gain the system, right? Tell me what you want me to achieve and I will overachieve it, right? Lines of code is that all.

James: Yeah, 100%.

Emilio: Is my performance based on the number of lines of code that I write? I can write thousands of lines of code just to solve the simplest problem that you can think of. So it is not so much about the task itself and the metrics, but also how you think about them and how you use them effectively as a way to look at performance in a more strategic and holistic way.

James: Okay, so times, tasks, and teams. And how do we measure that? Where are we getting the numbers from? Is this telemetry? Are we doing surveys? Is this data from the platform? What does that look like?

Emilio: First of all, it has to be a combination of all the above, right? Because otherwise you will not be able to get a complete picture. If you want to focus on tasks, whatever tasks you focus on from hey, lines of code, number of bugs fixed or anything else, eventually that is something that the system can provide you. When you think about, for example, team or time, Dota provides an excellent framework, mostly based on time, or number of times, I mean, your team deploys, uptime, change in time. But there are elements that are extremely relevant in the context of team that you cannot obtain unless you run a survey, right? And that also gets tricky because surveys are tricky. You know, there’s a timing component, there’s a difference on the way you frame the questions.

There’s a different way of how people from different environment, different cultures answer the same questions. How do you start this baseline? It gets really complicated. And that’s why the approach to this has to be thoughtful and strategic. It’s not about just coming up with, hey, are you happy about doing what you’re doing? And then expect a one to five answer.

James: Okay. So I mean, I think when I look at this question, and as you say, it’s going to be a mixture of different ways of measuring. Some of this is to a point. The time questions, mean timed recovery. We think about how often we are shipping, how often we are deploying. How are some of the ways, how do you measure some of the aspects around just team performance that are not timed? Talk a bit about like tasks or, if you think about the three T’s.

So we’ve understood that time may be a measure, but what are some of the ways that you’re able to measure some of the other considerations in the framework as you see it?

Emilio: Well, probably the easiest ones are always the ones connected to tasks. Eventually, you can reuse things to individual tasks and then assign a number to them. I gave you a few examples before, although I don’t advise anyone to use them. Like for example, lines of code, number of pull requests, number of bugs fixed, or even number of —

James: Emilio, we had an organization, we were talking to people and they were measuring how many microservices were built. And that seemed like the worst measure I’d ever heard. In an industry that’s measuring a lot of bad things, the idea that we would measure the number of microservices as a value and sort of as a goal didn’t make any sense to me.

Emilio: It doesn’t make any sense because at the very end of the day, what you’re looking at is outcomes versus, sorry, outputs versus outcomes. Eventually, you need to establish a connection to some of the business priorities. Your ultimate goal is either to drive adoption, drive revenue or drive user satisfaction. And then you have to establish a connection with that. That should be your ultimate goal. Your goal is that number of microservices that you’ve been able to build over the last three months, that’s irrelevant, you know, can be one or a thousand and yet you wouldn’t have achieved anything with them. And that’s the entire point about thinking about productivity, not in terms of individual metrics but in turn in a more holistic way that connects that back to the business.

James: I think this is so important, Emilio, because this gets to me, one of the things that we’ve been really surprised by at RedMonk, the idea that an enterprise would be concerned with spending $20 a month per developer in terms of generative AI and the values it will bring in terms of their productivity.

Because to us, if we think about what we’re trying to do in developer productivity, and an application will do that. We’re trying to enable developers to become more productive, more proficient. I mean, there’s a couple of ways of looking at this. One is just the basics of like, wait, if you save somebody 20 minutes because they were able to look something up or they were able to be guided through the creation of an issue, perhaps the interaction between two people in a pull request, doing a better job describing why we’re to solve that problem. The idea to me that we would worry, I mean, even in a large developer population, I mean, I’m assuming I’m a bank, I’m 10,000 developers strong. But the idea that I would worry about spending $20 for developer. Doesn’t really make sense to us at RedMonk because we’re like, well, actually what is the cost of time saved? And that’s one of the ways of thinking about productivity.

Another one is this. There’s a guy called Simon Willison. He is one of the people doing a great job of tracking generative AI and the benefits it has. And he looks at this as a question of like, what are all the things I wouldn’t have done because it would take too much time? So supposing, I mean, he’s building a platform called Datasette.

Emilio: Like the opportunity cost.

James: Yeah, absolutely. Opportunity cost. So he’s building this platform data set, and it’s all about like pulling together data in order that and he started from a citizen journalism perspective, developer to citizen journalist. But his thing was like, look, at any given time, there are going to be five or 10 things that a developer may be thinking, hmm, what if I could build a tool that would make me more productive in building the platform or in supporting the platform? And they’re like, you know what? Oh God, I’m gonna have to go and learn that like, you know, that Swift library, or I’m gonna have to learn that JavaScript framework. That’s not, I don’t quite understand the nuances between Angular and React in this place. It’s an issue. If you’re able to shortcut that, so that the developer can say, actually, I’m gonna have a discussion with the AI. I’m going to begin to solve any problem, you become in an environment where instead of it being a blocker, you’ve got all of these new opportunities.

And for me, Emilio, I’ve been thinking about it as the introduction of the cloud. No need to ask for permission. The developer can get work done. Now, I think that you kind of see it apparently, you think about it a little bit differently. It’s not just the cloud, it’s serverless. Like the productivity enhancement and opportunity goes well beyond just, oh, code to cloud and what that looks like.

Emilio: It does. It does. And I could not agree more with you. If we think about it at the individual level, I’m a thousand percent with you. That’s the beauty of serverless. Lambda is an example. It’s that, hey, write your function and then it runs. Forget about everything else. You don’t need to spend any time on anything else. The productivity increase there is huge. Absolutely. If everything else around you remains the same.

There’s a couple of things that we need to factor in. One is that development, especially in large enterprises, is a team sport. It’s like you can be extremely productive or increase developer productivity. People are writing code by 1,000%. But that doesn’t mean that you’re going to be able to deploy 1,000 times faster. There are multiple links across that journey that you also need to optimize.

James: Right.

Emilio: Google did a piece of amazing research around factors that can help predict developer productivity. And if you go through them, most of them were not technical. They had to be with the work environment. They had to be with the project people were working on, the peer support, access to documentation. So my point around this is, unless you consider all those other elements, yeah, you might be able to increase the productivity at the individual level with one particular task, but then nothing else is gonna change and then your overall productivity is gonna suffer because of that.

James: Okay, so let’s sum up a little bit. We’ve established that some of the metrics are brute force metrics, not necessarily gonna be so helpful in truly understanding whether the organization is being effective from a developer productivity perspective. We’ve got the opportunity of AI and generative AI in making developers and people working with them more productive in various areas, whether it might be describing bugs, describing issues, LLMs are pretty good at this stuff. So if we think about the interaction between what does productivity mean, you’ve got three T’s. If we think about where GitLab is in terms of LLMs and development tools, what should developers and your customers be most excited by, what should they be looking at? What’s your call to action in terms of taking advantage of a better development framework and thinking about that in terms of AI?

Emilio: I think that my advice to existing customers is, hey, number one, don’t believe everything you read. Number two, take this as a strategic project, right? And then is that see the best way and the most strategic approach to apply GenAI along your entire software development life cycle. Number one. Number two, keep in mind that there are organizational factors and team elements that you also need to consider. Right?

And then do that in a thoughtful way, not because someone at the top said, oh, this GenAI is going to be fantastic when you’re going to increase productivity by 50%. No, because that’s going to create basically the opposite effect. And in fact, what you’re going to see is that some pushback from existing teams, because they will not even be willing to try the new toolset, because they may see that as a threat. So I think that the advice here is that

Take this like the cloud. Having a single developer back in the day using serverless was amazing. That person could prove that they could do things in no time. But for your entire organization to take full advantage of the cloud, that required a ton of transformations internally, culturally, and then in terms of skill sets as well.

James: Okay. So skill sets, you know, GenAI needs to match that. Yeah, I think from my perspective, looking around the industry and certainly looking at GitLab in the broader context of the software delivery lifecycle, it’s not just about a developer, it’s about a lot of different constituencies. You’re focusing on development as a team sport rather than just sort of individual developer and making them more productive. And I think basically that has to be the right approach. So I’m looking forward to seeing how the three T’s lands. I’m looking forward to seeing how your customers begin to adopt the generative AI tools that you’re building. And here we are, we are kicking off 2024. We’ve all got a lot of work to do. And I’d just like to say Emilio, thank you so much for joining us today.

And that’s a RedMonk conversation. Thanks everyone for joining. And by the way, if you’ve got a question for Emilio, let us know, reach out to Emilio if necessary, but like, subscribe, share the conversation. If you’re using GitLab, we’d love to know what you think. The new functionality is there. So like, subscribe, get on board. Thank you, Emilio, and let’s crush it in 2024.

Emilio: Let’s crush it James. Thank you very much. Thanks everyone for joining us today. It’s been a pleasure.

James: Thank you.