In the behavioral sciences, the tendency to engage in “mind reading” – assuming knowledge of others’ thoughts or beliefs – can lead to maladaptive practices, particularly in software engineering teams. This phenomenon affects crucial areas like developer productivity, an often misunderstood metric that is inconsistently defined across teams. The presentation introduces a “Three Layer Productivity Framework” to differentiate between production, productivity, and performance. Utilizing this framework, research has highlighted the misalignment in defining and measuring developer success and the preference among engineers for quality-oriented metrics over production-focused ones. The talk provides suggestions for how engineering organizations may align their success metrics more closely with the actual values and insights of their software engineers.
Transcript
Hello, everyone, I’m Morgan, I’m one of the two research scientists in Developer Success Lab, with my colleague, Carol Lee. First of all, it’s Friday. Second, this is our first time here, so thank you so much for the invitation, we’re super-excited to be here. And a quick housekeeping thing. Everything we’re going to throw out to you today is accomplished in four or five research studies, so keep it at a high level. So if you want to go deeper, you can take a screenshot at the end, download all the papers in our talk. So without further ado, I’ll pass it over to Carol.
>> My name is Carol, I am a clinical scientist in the Developer Success Lab at Flow. I conduct researches about how people like you and me who are developers work, cope and thrive through times of stress, which I’m sure we all experience here, right? I want to start with a fun fact about being a psychologist. So just kind of bear with me here. So when you are a psychologist, upon telling somebody that you you’re a psychologist, I’m going to say that we’re going to say for instance, OK, there’s like a 30% chance that they will ask you if you can read minds. I don’t know why it happens, but it does, and I obviously say no, I don’t read minds. At this point people usually pretend that they were joking when they weren’t, like no, I can’t read minds, I’m sorry, it’s dispointing, etc., etc. The funny thing about this is psychologists think about mind-reading kind of a lot, we just don’t think about it in the psychic sense * we think about it in a cognitive bias where question assume we understand what people are thinking, feeling and believing.
Now, the thing about mind reading is that we all do this. We do this about really small things, like assuming that everybody likes chocolate, but we also do this about big and important things, too, like saying that we all agree and know what it means to succeed as being a developer, something that we see on software teams quite a bit, actually. So just to highlight this, I’m going to do a quick little experiment. I want you to answer this question. Which of these is the most important characteristic of a successful developer? A, completes work that is usable and value to the org. B is able to complete a work despite tech debt. C, commits a lot of code, D, has a lot of pull requests, and E, writes high quality code. So raise your hand if you think A. Don’t be shy.
OK.
Interesting.
Raise your hand if you think B. C? D? And finally E.
Interesting. This is so interesting, sorry. OK.
So I’m trying not to analyze you all as I stand up here. OK. Take notes, Morgan! OK. All right, so kind of notice that not everybody chose the same thing here, right? And this is pretty normal. We all have our own definitions and ideas about what success means to us personally. Thinking about that characteristic that you just chose, is this the primary way that your success has been evaluated thus far? So if you said, hey, the important thing is, I don’t know, code quality, has your work been evaluated based on code quality? Raise your hand if that’s the case.
OK. So not a lot of raised hands, right? In fact most of us are not raising hands. So what does that tell us? OK, one important thing that tells us is the way that we define and measure success for ourselves isn’t always the way that we are defined and measured externally in our orgs, right?
And this nicely highlights one of the biggest problems with mind-reading is that when you engage in mind reading, you actually run the risk of being really misaligned on really big and important things, like how you have been successful in your careers and if you think about it, that’s actually kind of a big deal. Because when you think about it, your career depends on more or less everyone agreeing or being aligned on the fact that you have been successful.
And we also see that as our data, as well. So we see some evidence of this alignment in this orange box here, it’s going to highlight some of these major misalignment points here, right so if you look at this, a lot of these say hey, when you are currently being measured by things like number of pull,” number of code reviews, much more than they really want to be. We see more alignment in this box here, where people say they want to be evaluated on things more like code quality much more than they currently are.
We see this broken down by role, where we ask ICs, where they say that hey, we think there’s a lot of focus and usability in value of the work to the organization, as well as the number of pull requests, but what we really want is focus on code review quality and code quality. You see a really interesting misalignment here with the managers, though. Managers say the opposite. They’re like, hm, I think we actually do want code quality and longevity. What we want more of is this usability and value of work to the org.
You see this flip-flop misalignment here. All right, so why is philanthropies happening? One of the reasons is that we tend to choose inappropriate measures success because we haven’t taken the time to find out what success means. At the end of the day your measures are only as good as your definitions. The other is that we tend to engage in mind reading. We assume we know how our colleagues and managers are defining success and we assume that it’s the same way that we are defining and measuring success.
So how do we manage this? You can define success before choosing your measures of success. This sounds really simple, but it’s actually a really effective way to make sure that you’re being aligned on the definitions and your measures of success.
And then second of all, of course, you want to make sure thatoo you are aligning on how you define and measure success across roles.
So what does this look like in practice? So imagine that you have been asked to evaluate your team’s success, most of us have had to do that some point, right? So a common mistake that folks make is they’ll start by making a list of every success measure that they can think of, right? They might write down things like raw counts of code commits and pull requests. They might look at
And then they might look at amount of work given tech debt, team size, and then things like code quality, longevity, etc.
The second thing that we do is assume that all of these measures measure the same thing. You have just written down a list of success measures and so you think, yeah, these all measure success, great, move on.
Because you assume that they all measure the same thing, most people will just choose the ones that seem the easiest to remember. You’re busy, you’re going to choose like the top three things to do and move on from there. This is the mistake, because it turns out all of these measures of success don’t actually measure the same thing.
If you dive into our research, we actually see that things like raw counts of code requests and pull requests that’s actually a measure of something called team production. You’re looking at the quantity of your team’s output, regardless of any resources provided to that team. No context whatsoever, OK
If you take a look at things like tech debt, team size etc., now you’re talking about team productivity. You’re looking at the quantity of output given the resources provided to a team.
Finally things like code quality, longevity, sociocultural factors, this is a measure of team performance, now you’re looking at things like flexibility, adaptability, dependability, stainability beings and quality of output and resources over time. Through a longitudinal lens.
All of this is to say that you really want to define what you mean by success before you actually measure them, right?
Because at the end of the day, things like raw counts of code commits, that is not going to measure team performance, no matter how much you want it to be. At the end of the day, if you want to be measuring things like team performance, you’ve got to be looking at things like the code quality, code longevity, these sociocultural factors.
And then finally of course, you also want to be talking about this within your teams. As we saw, there was misalignment on what these teams thought they were being measured on, versus what managers thought they were measuring. You all should know how your team is measuring success and how you’re actually going to measure it.
All right, so all of this is to say that instead of mind reading you could define success before measuring. Talk to your managers, ICs, leaders, etc.
> Yes, in addition to that, we have as I mentioned earlier, lots of different studies in this. So we also wanted to explore, this might be happening no your developer teams, but what might be happening with your collaborators, so we did a couple of studies and realized that actually as developers you’re not alone. We actually did a study to understand kind of the ecosystem. And we also designed this product triad. and then product folks, think of your Product Managers, Product Leaders, whoever they might be. So in a separate study we wanted to look at one of those relationships. Particularly how developers understand product folks and how product folks understand developers. Primarily as Carol was mentioning, we wanted to see, how does mind reading affect our product partners? Are they seeing what we see in are they seeing something completely different? And to do so, we wanted to focus on the question in particular, what does productivity mean to you?
And doing so, we had 17 hours of conversation with both product people and developers, and shocker! It was different.
So developers are more so thinking from a quantitative lens, using words like speed and time to define how they perceive productivity. Whereas Product Managers were saying terms like clarity or quality to define how they saw productivity. So very clear from the beginning it was looking at the word productivity that both sides were defining it very differently from the start. So OK, that was cool, now, let’s go into it a little bit deeper. If we’re looking at productivity, how do we define developer productivity, and what does that mean to you?
And to do so, we did a little bit of a kind of an innovative way of going about it. So my background is in design research and UX research. One of those designers in the bottom right and so a way that we were testing this perception was in the concept that we kind of have all seen it. Is this picture. If you see it, don’t say it, it’s kind of the thing. But raise your hand if you kind of see a more mature woman in this photo. Raise them high. Be proud. Yes, and raise your hand if you see kind of a younger girl in this photo.
So as you can see, same image, two different pictures, and that’s kind of the concept that we wanted to do, to test perception around productivity for developers and engineers. What we did again, same picture, two different images.
We showed three different hypothetical representations of what a working environment might look like for a developer. Team A, showed merged PRs, teamed B showed average churn per commit and Team C showed average cycle time each month. We don’t have to go into the nitty-gritty. But one thing we’ll kind of focus on is Team C. When we showed this to developers and product folks, we had two different completely different answers.
We asked, this chart is considered a normal trend in developer productivity. Developers, 75% of them said yes, this is very normal, day in the life, this is what happens. This is considered a normal trend. on the contrary, the flip-flop that Carol was talking about, product folks were oh, no, something is wrong here, this is not normal, this is completely different. Something has gone awry. And the most important thing to take away from this is in fact you were looking in one picture and seeing two different things. There are two different perceptions in what’s going on. In addition to that, 75% of course is not 100%. So it also points to the fact that there is a bunch of developers who are thinking this is actually not normal. So while there might be misalignment, it’s also true that even developers think differently about their other productivity as Carol mentioned. So yes, it’s very important that we’re aligning on that, to make sure that we are reaching the same goal of success.
So that might be the case of why, not to mind read, so what should we do? In interviews, we talked to those same folks and they suggested three critical things to making sure we’re aligned in productivity. The first is effective communication, not a huge shocker, I’m sure we all know that it’s important to effectively communicate, while in a work environment.
But what this unlocks as a lot of Product Managers and developers talk about, was not only our communication is a lot smoother, but you’re also able to have way more effective planning sessions and not only planning sessions, but the preplanning before you start a trend, before you do anything, is that conversation before you start the project is imperative. For example, one Product Manager said, the pre-planning is important, it’s like a backstage conversation before the big finale. A product developer, said it’s the only thing that’s important. Both sides of the house in over 17 hours of interviews, folks are pointing to the critical alignment in the planning sessions to reach success, and the other thing that we saw in that are not only effective communication or alignment, but how important domain knowledge is and we don’t mean domain knowledge in terms of developers knowing development work and PMs knowing PM work. We’re talking about knowing the other sides of the house. For example, one developer said: When — and developers said, when p.m. Ms understand the developer workflow, they understand infrastructure and what we can achieve within a certain time frame. So let’s take a second for that, not only to make it just very clear in what we’re going towards and having effective communication there. But it’s also important to understand the other side.
So whale productivity might be misunderstood or understood in different ways, defined in different ways, you might be trying to mind read that, communication, planning and domain knowledge are critical to reaching team success.
Now, that all sounds great, it’s all well and good, but my specialty is also in inclusive design research and we would be remiss if we did not talk about the most important thing that came through those interviews is all of this is not possible without trust, and this has been a theme I — listening to these wonderful talks and it’s really important that we’ve been seeing this, is how important this is. It’s not an optional thing. It seems like it might be a “nice to have.”
But where they come from, what they’re doing in their lives, without those, it’s going to be very, very difficult for you all to reach team success.
We dove into this in another setting with nine different developers, to kind of ask them what leads up to success.
And without getting too far into it, because it’s a bigger study, we came up with this thing called the trust ecosystem and at a very, very high level, there are different components to beings that either create bridges to that team success or barriers to that team success. At the very fundamental level what you want to walk away is this very bottom part which is around human context and empathy at the foundational level. If there’s anything to walk away with today, reaching those is critical to reaching team success, reaching alignment, making sure you’re not mind reading.
We had one quote from a Product Manager that I wanted to read out which was so beautiful about that and he said, I had a team that I worked with from 2020-2021 and we had stellar collaboration because we built that trust. How do you make people feel safe to trust? Starts with me opening up. It’s being able to share my experiences. It’s allowing everyone to feel comfortable enough to share their lives. It’s not an overnight process. I’ll repeat that again, it’s not an overnight process. There isn’t just one point where we built trust. You look back on a situation, we’re like, oh, well over the course of the period of time, things started being a lot more cohesive. So if anything, we wanted to share, yes, there might be misalignment, yes, we might be trying to mind read, but as long as we’re trying to understand with the other people on your team, you’ll be able to reach that team success. Thank you.
[applause]
