We hear a lot these days about developer success, productivity, and happiness, but what about developer anxiety? Dr. Carol Lee stops by to talk about her research as Principal Scientist at Pluralsight’s Developer Success Lab, and how her training as a clinical psychologist comes into play. Kate and Kelly also learn how Carol created an Institutional Review Board (IRB) at Pluralsight to clarify processes around the creation and maintenance of ethical standards and to enable scientific rigor. Also of interest: a research project on code review anxiety, and the creative process behind Carol’s webcomic, Thriving Blobs.
This was a RedMonk video, not sponsored by any entity.
Resources
- More info on Carol’s research into code review anxiety: Understanding and Mitigating Code Review Anxiety
- Why I Created an Institutional Review Board at Pluralsight
- Thriving Blobs comics:
- Carol and her colleague Morgan Ramsey speak at Monktoberfest 2023: Using Evidence Science Instead of Mind Reading to Measure Developer Success
- Hyperbole and a Half comics on depression:
Rather listen to this conversation as a podcast?
Transcript
Kate Holterhoff: Hello and welcome to this Docs Are In conversation. My name is Kate Holterhoff, Senior Analyst with RedMonk, and with me is my co host and fellow Senior Analyst, Kelly Fitzpatrick, as well as our esteemed guest, Carol Lee, Principal Scientist at the Developer Success Lab at Pluralsight, a company that is all about technical training and onboarding. Carol is also a friend of RedMonk. She has spoken along with her colleagues at the 2023 Monktoberfest, and her talk entitled “Using Evidence Science Instead of Mind Reading to Measure Developer Success” was an absolute hit. And I have no doubt in my mind that the theme of using science will appear throughout today’s conversation. Carol, thank you so much for speaking with us today. To start things off, can you tell us a little more about who you are and what you do?
Carol Lee: Sure. Thank you for having me, first of all. So I am a clinical scientist in the Developer Success Lab. What that basically means is that I’m a scientist who specializes in physical and mental health related issues because my background is in clinical psychology. Obviously, this is more about the mental health side of things. Yeah, my expertise is in stress and anxiety. More specifically, it’s this thing called mechanisms of change. What that really means is that it’s like the stuff that really drives people to change, even when they’re experiencing stress and anxiety. So kind of relating this back to Developer Success Lab. If you’re a developer, you probably experienced some anxiety at work, I’m guessing. Stress is probably very familiar to you. So, yeah, it’s kind of looking at those aspects and those issues and developers and helping folks kind of grow through that.
Kate: And how did you end up in this position? What was your education and your journey to working at Pluralsight?
Carol: Yeah, I have kind of a very strange journey. I have a doctorate in clinical psychology, and that means that I’m trained to give therapy. My specialty was traumatic and anxiety related disorders. Did that in specializing in sexual trauma for a long time. For like, a few years, it felt like a long time. And then I moved into being a professor, which I absolutely loved, and at the same time just kind of felt like there’s this thing that happens when you’re an academic where the people you’re talking about your research to also research the thing that you research. And so it’s a little bit of this echo chamber where you’re like, mental health is important. And everyone’s like, yeah, I know, mental health is important. And you’re like, yeah, I know, awesome. And so this kind of thing happens.
And then also at the same time, you know, a lot of scientific research is gatekept behind a lot of fees for people so a general, like, person googling can’t actually read a lot of scientific literature because it’s locked behind these publishing fees that are pretty, like, expensive, actually. So I try to kind of manage this by only publishing open access for a while, but then, you know, that kind of got old and so really wanted to, like, make a difference and kind of share science more broadly. I’m really passionate about making science something that everyone feels like they can attain and kind of have access to, and particularly, like, rigorous science.
And then at the same time, I have a really good friend who is a software developer and was kind of talking about their work. And I had this very strange realization that I’m sure everyone else is, is kind of very familiar with already in this space where I was like, hey, the world kind of runs on code. Or, like, my hospital system, like, relies on code. Like, the education system relies on code. So then I was like, well, what do developers — what’s up with developers? And the science on it is pretty dated. Like, really relies on psychology research from, like, the seventies. And it’s research that was based on five old white guys, and they’re like, yay, we did the science thing. And I’m like, wow, we could do so much better. So when Pluralsight developed this think tank style lab, the Developer Success Lab, it was just kind of a no brainer. It was just really fun.
Kelly Fitzpatrick: And it sounds like a very timely thing to have, especially with what everyone’s been going through for the past couple years. I feel like anxiety is like, it’s all our best friend now.
Carol: Yes. Yeah, I mean, I hate the pandemic for a lot of things, and I really appreciate that the pandemic almost gave folks an excuse to say, like, yeah, anxiety is a thing. And I’m like, yes, and you’ve always had it, by the way, right?
Kelly: Yeah, you’ve always had it and here are some better ways to maybe talk about it. And to that end, can you speak a little bit more about what is the clinical in clinical psychology kind of mean? It’s a field that I don’t know too much about, and you, you know, you clearly do.
Carol: Yeah. So if you think about psychology as being, like, studying human thoughts, feelings, behaviors, emotions, et cetera, clinical psychology is taking a look at how to work through, treat, assess, and create interventions for kind of those maladaptive processes. So when those behaviors, thoughts, feelings, et cetera, are kind of getting in the way or are not serving you in a positive way, really, clinical psychology really looks at like, okay, well, how do we assess and treat this?
Kelly: And then also, you spoke a bit about Pluralsight and the Dev Success lab, and Kate gave a bit of intro, but can you talk about the relationship between those two? And funny aside, Pluralsight is one of the kind of training resources that is suggested to you when you are a member of the ACM, which I am. And I was like, I noticed that just this week, and I was like, oh, my goodness, this is very serendipitous.
Carol: That’s so funny. Yeah. So I would say that the best way to describe us is that we are a think tank academic research lab that’s funded by a tech company, which is Pluralsight. So it’s really nice. We do have academic freedom, and so that means that we don’t do any product research. Our participants aren’t users, which is really important to us. Something that I really — something that’s a really big pet peeve of mine, actually, is when folks do research, and all our participants are from the company they work at, because now it’s like, well, now we know a lot about your company, but not about literally everybody else. Right? So that’s not super generalizable to the general population. So that’s a really big value of ours, and we’re an open science lab, so that means that the science that we do, we really freely share with folks. This isn’t something that you have to pay for. It’s not like you have to be a Pluralsight user to access this. We’re just kind of like, we really care about the science. We really think developers deserve science, and we just want to give it to you for free. So, yeah.
Kate: It’s a wonderful motive. And can you define what a think tank is exactly? I think I associate them with, like, DC politics and, like, lobbying.
Carol: No, it’s basically like… a group of people that get to conduct research together to inform policies in the future. In this case, our policy isn’t a political thing. Our policy is changing the tech world a little bit, changing how folks work or how folks see developers.
Kate: So you’ve recently written about the process of creating an institutional review board, or IRB, for short at Pluralsight specifically. And Kelly and I are both super familiar with IRBs from our time at Georgia Tech, where we both went through the certification process to serve as principal investigators. However, IRB is a term that you don’t often hear in the tech industry, even though they can be crucial in setting and maintaining ethical standards. Can you speak a bit about the purpose of IRBs and why you felt Pluralsight needed one?
Carol: Yeah. So an IRB is essentially a group of people on a board, as you say, it serves as an ethics board. And their whole point is to make sure that researchers, people doing academic style research is, one, trained on the ethics of human research principles, so they’re not making any kind of mistakes or accidentally putting people’s data at risk, things like that. And then two, to actually make sure that focus studies are being ethical as well. So every study that a person completes or kind of runs, they need to get IRB approval ahead of time. So it’s not like an ethics board is like, oh, man, you shouldn’t have done that. Right? It’s like you say, here’s everything I’m going to do. The ethics board looks at it and says, okay, here’s where I have questions. Here’s where I don’t have questions. You need to change this. Give me more information on this, et cetera, et cetera. And then you go ahead and do that study. If you have any changes that you make, you actually have to go back to the ethics board and say, oh, sorry, I’m actually going to do this other thing instead. Now I’m going to add this other thing. And the ethics board has to include those changes as well. So it’s really just a board that approves your study to make sure you’re being ethical.
So the reason why I think that Pluralsight needed one is that honestly just kind of the nature of our research. You don’t see IRBs a lot in industry because a lot of folks tend to do kind of very lightweight survey research. So to be fair, a lot of industry research would be qualified, what we call as in the exempt category. That means that an IRB will kind of like briefly look at the proposal and say, doesn’t really need review. Like, go ahead, do it. There’s like no or very minimal risk here. And this is where a lot of industry research falls under. But if you want to start moving into kind of deeper human experiences, right? So things like mental health or job security or fears, things like that, or if you want to start doing more rigorous methods, so moving beyond survey research and moving into things like experiments or intervention work, which is my favorite thing in the world. So if you want to start doing that kind of work, then you do want an IRB. It’s not legally required to have an IRB in an industry kind of research lab, but it is just kind of ethically sound to do that? Yeah.
I think, like, on a personal note, too, I just like the idea of having extra eyes on my work. Maybe it’s, you know, just a result of being a psychologist or maybe it’s my background in being an academic. But people make mistakes all the time. Right? Like, we often don’t know what we are biased against or for because there our own biases, right? So, like, I’ve had, you know, people, I’ve had IRBs come back and say things like, hey, you actually want to explain, you know, that you’re going to ask about these things before you actually ask about it. I’m like, oh, you’re right. Like, I’m so used to asking about these things that I didn’t think about the fact that somebody might be like, whoa, why are you asking me about this right. And so kind of, so having those checks and balances is really nice. I think anytime you’re working with humans in research, you want to be really prioritizing the humans that you’re working with. Right? Like, that’s kind of important. As researchers, we have a lot of power over the people that we’re conducting research on. And so I just think it’s important to respect that.
Kelly: I love that. And having recently gone through, like, I’ve had to recertify myself for the Georgia Tech IRB because you have to do that every few years. And one of the things that struck me this time around was the questions and the kind of guidance around safeguarding people’s data and privacy, which are things that I feel like people care about more and more these days.
Carol: Yeah, I think that something that… Oh, gosh. What do I want to say about this? Okay. Something that I think is not maybe really understood about academic research is that in order to have IRB approval, your data has to be private and secure. What that means is that nobody else but the researchers have access to it. So, for example, with our data, Pluralsight does not have access to our data ever. This is our data. Only the researchers have access to the raw data. Obviously, when you report on it, do aggregated scores and things like that, that goes out to the public. But raw data is researchers only. I think something that often surprises people is that things like IP addresses, that violates IRB rules, right? So, like, we don’t collect IP addresses, for example, because that’s a pretty big no no to the IRB because that’s violating privacy. So I think there are these, like, little things that folks may not think about, but that the IRB does which is pretty cool.
Kate: Yeah. I tend to think of traditional IRBs as protecting, you know, the rights of children and like, neonatal, you know, entities. So, pregnant women, prisoners, things like that. But, yeah, this sort of new realm of digital rights also comes into it, which I think is useful and kind of looks forward. And it probably would inform a lot of the research that you’re doing at the think tank.
Carol: Yeah, I remember when I was in grad school, right, we kept all our patient data in locked file cabinets in a locked room, inside a locked office that nobody except the researchers had access to. And so you didn’t have to think about, you know, these things like, oh, well, we did a digital intervention, and so now all the data’s online. It’s instead just like, oh, you like, have to have the key. And if you lose the key, like the whole thing, it changed the locks and, you know, so it’s a very different kind of thing to be aware of. So, yeah.
Kate: Yeah. I also like when you characterize it as almost like a reader to like a part of peer review where you’re looking for readers who are going to view, you know, kind of before the research is even out, you know, your ideas for study. Your thesis, the methods that you’re going to employ with that critical lens and can intervene and say, hey, you know, I have some suggestions here. I think you’re on the right track, but here’s some ways that I would tweak it. But, yeah, I’m with you. Actually, Kelly and I joke a lot about peer review because, of course, it has a lot of issues, but in general, there’s a lot of merit to it right? And if only for the fact that you have someone that you’re in conversation with about your research. And of course, all of us who have done the academic thing get very excited about the things that we study. And so the fact that at least one person is reading it and they’re willing to challenge you and compliment you on things that are going well, I mean, that’s a real boon.
I mean, that’s something that actually, in my most recent book, I devote some time in the acknowledgments to the blind readers. I don’t know who they are, but I am so grateful to them for taking the time out of their schedules to really dig into my argument. So, in any event, I haven’t really thought about IRB in quite that way. And so I really appreciate your perspective there.
Carol: Yeah, I kind of think of it as, like, I mean, the peer review process can be brutal sometimes, right? Like, sometimes you’re like, who is this person? They hate you. And then other times, you’re like, wow, such an amazing, like, review. I never thought about that. Love it. It’s all about tone half the time. But I often think about our research projects as, like, our babies. And, like, you really hate it when someone’s like, man, your baby’s ugly. You know, that’s what it feels like oftentimes. Right? But you also want to know, like, if you’re doing something really, horribly wrong. And so I kind of feel like the IRB is like, this, like, helpful, I don’t know, expert who’s like, hey, maybe you shouldn’t do that with baby. And you’re like, you’re right. Like, baby probably doesn’t love that, and people may not love that. And it kind of feels like, hey, you can see someone saying, like, hey, you’re a bad parent. Or like, hey, you have an ugly baby. But, that’s not really what’s happening, right? They’re just trying to be helpful. But it does mean that who’s on your IRB board matters a lot. I mean, that is something that we thought about very, very carefully, because sometimes you get a bad IRB board, and it’s just full of people who just want to, like, make you happy, and you’re like, well, I don’t want you to make me happy. I want you to make me better. Right?
Kelly: I love that idea that you don’t want your board to be entirely made up of people pleasers. You really want kind of that rigor. And I think you’ve made — you’re kind of going back to this argument that you’ve made, which I think is great, about how IRBs facilitate scientifically rigorous research in applied settings. Can you speak about how this is playing out in your current or recent research? Like, for instance, I know that you have been working on code review anxiety, which is a topic I think many technical practitioners get. But I feel like your clinical psychology training and adherence to scientific rigor leads to a different approach than what we often get through tech industry thought leadership.
Carol: Yes. Yes. This is a topic that’s super near and dear to my heart. One of my major expertise areas is social anxiety. And you can think about code review anxiety as a type of social anxiety. It’s performance based, evaluative, and there’s a social aspect to it. It’s hitting all the pieces of social anxiety. Something that I think is wild is that when you look at how to — how developers manage code review anxiety, there’s a bunch of research — not research. Literature, I will say, like, books or like, self help, whatever blog posts out there that are like, code review, we all have it. Code review anxiety, we all have it. And then they’re like, here’s how to write better code. Which is, like, a little bit vile to me. Like, it’s kind of like saying, hey, sucks that you’re feeling not great about yourself. Have you thought about being just a better person? And you’re like, that’s like the worst thing you can say to somebody with anxiety, right? Like, supremely unhelpful.
Kelly: Like, why don’t you just not be anxious and be better at the thing you’re worried about?
Carol: Exactly. Like, oh, that’s so weird that you’re depressed. Have you thought about not being sad? No?
Yeah. So it’s kind of like that similar messaging that developers are getting all the time when they’re dealing with code review anxiety. So I really wanted to create something that’s more evidence based, based on actual cognitive behavioral therapy principles. So I made this workshop, right? But to make this workshop, I need to get IRB approval because it’s now an intervention. We’re kind of diving deeper into more intense human experiences, like anxiety, and getting a lot of folks in this study that have actually quite high code review anxiety, for example.
And so that was kind of a big piece of what the IRB lets us do is say, hey, I know you’re going to do this deeper dive into this. Let’s, you know, make sure that this workshop is helpful, not harmful right? Let’s make sure that this workshop actually, like, looks like it’s legit. It’s not just, like, hot takes. It’s a real, you know, based on a real thing, and there’s a good reason for why you’re doing it. And let’s also make sure that when you’re asking people about their anxious experiences, you’re doing it in an ethical way. What’s also really cool is that in most clinical work, you’ll see something called a randomized clinical trial or a randomized controlled trial. So this is also an RCT, we call them. And so having an IRB actually allows you to move beyond survey research, which really only can give you correlational data most of the time, and actually move into kind of experimental data. So moving away from correlation into causation so it’s pretty cool.
Kelly: I feel like I’ve just learned 100 things in that one section of our discussion today.
Kate: I just wanted to point out that one of the things that I loved about your article on this, Carol, is that you actually move from just like, this is a problem to, how do you set up an IRB board within your own organization? So you, like, actually, as Kelly’s point, like, the being solutions oriented, like, giving a blueprint for how to create this and just going beyond the, like, this is a problem. And here’s why you should care.
Carol: Yeah, I appreciate that because I really didn’t want it to be something that was like, oh, look how special I am. I have an IRB. And you can’t have one. Right? But more about, like, if I have an IRB, you should get one too, right? Because I think oftentimes —
Kelly: Yeah, IRBs for everyone!
Carol: Yeah. I’m like, yeah, we all love ethics! Woo! Ethics for everybody! So, yeah, I really, yeah, I did my best to kind of, like, lay out some simple steps. It actually wasn’t super horribly hard. You just, you know, got to get your good board together. Granted, writing the handbook of the manual, I definitely had some prior expertise there from being on being on an IRB board at an academic institution. So I can see that being a harder step for most people. But, yeah, I mean, getting it officially registered wasn’t too bad.
Kate: As we are getting to time, I have got to hear about your work on illustrations here. So published a little bit on that subject myself. Always get excited when some of our clients have architectures on their website. Love digging into visuals, and I just think they’re super important. So I have got to hear about your own illustration, your own forays into visual communication. So you’ve created a web comic called Thriving Blobs. Can you tell us about it?
Carol: Yes. Zooming out a little bit. Have you all heard of Hyperbole and a Half? It’s like this web comic series that was out like 90s, 2000s, maybe, something like that.
Kate: I don’t know that I have. Tell me about it.
Carol: It is really bad art. I mean the person, like, the author admits it’s really bad art. She’s just like, I don’t know, I don’t even care. Clearly done on, like, Paint or PowerPoint or something like that. But she really excels at talking about tough things in a really real and approachable way. So when I would teach Abnormal Psychology. Or, like, Helping Skills, things like that, I’d always make my students read about a comic she wrote about depression, and it was very, very approachable and real and really talks about depression in a way that is kind of like lived experience and matter of fact, but also very funny, right? Like, kind of a dark humor. And that’s always really stuck with me. It’s always been something that my students in the past were like, yeah, I get it now, right? Like, they’re like, oh, I get why you shouldn’t say just be happier. I get why you shouldn’t just throw solutions at people now. Like, I totally get it. Just really amazing work. And so I’ve always had this kind of fondness for this webcomic thing in communicating tough or technical work.
So when we were talking about, kind of communicating, about developer thriving, I kind of had this, like, inkling of, oh, it’d be really fun to do a comic, but I don’t want me to do the comic. I just want, like, the comic to happen because I’m not an artistic person and I don’t really want to draw, you know? So, like, I didn’t suggest it at the time because I was like, I bet if I say someone should do a comic, someone’s gonna be like, well, why don’t you do a comic? And I’ll be like, hmm, good question. So then I, like, secretly started doing this comic, just to see what I could do, you know, very rudimentary art. It’s like, literally blobs with some faces on them and started writing up what I would want to say if I were, like, talking about writing to my friend. Made a few panels and just, like kind of randomly put them in my team slack. And was like, I made this. My manager was like, well, I love it. What is it? I was like, it’s a comic. Do you want me to do it? And she was like, yeah, just go for it. Who cares?
So it’s really fun because I think that as scientists, folks sometimes think that you are in this field and you’re serious the whole time, and you’re just like, oh, yeah, this is science, and I frown the entire time. And I’m not, like, playing with my work. But actually science is a really playful field. You get to play with methods, you get to play around with how you’re going to measure something. You’re going to play around with how you’re going to present something, and it’s a really fun and dynamic process, right? And I kind of wanted to kind of emanate that, like, joy of play in some ways into this comic while I was talking about thriving, like, hey, when I think about thriving and not just thinking about it as a scientist, I’m thinking about it as the human that really loves people and really wants people to enjoy good things and have good things happen to them. Right. So that was kind of the process behind it, I suppose. Turned out pretty good.
Kate: Turned out amazing. Are you planning on doing more webcomics?
Carol: I am. I have one in draft right now. Yeah. This time blobs get a pet, which is something called a sheep cat dog. So watch out for that. Sheep cat dog is sheep cat dog because it’s the only animal I could make on PowerPoint. And it was determined it was a mix of a sheep cat and dog. So I was like —
Kate: I was wondering what your method was. Is it for, like, what tool you were using to create this? You’re using PowerPoint. I love that so much. If our colleague Rachel Stephens was on here, she would use Excel. She would find a way.
Carol: Yeah, 100% on PowerPoint. It’s very [inaudible 00:26:06]
Kelly: I’m here for the Excel created webcomic now, though.
Kate: Yeah, I know, Rachel…
Kelly: No pressure Rachel! Yeah.
Carol: Rachel if we can do a little team up, I would love that.
Kelly: I love this so much. Yeah. But kind of going back, Carol, to the whole idea of the comic making thing is just kind of, like, real and accessible. For me, I think you’ve made the entire field of clinical psychology real and accessible, and I think for many people out there, the concept of IRBs as well. So thank you so much for taking the time to join us today.
Carol: Thank you. I really appreciate it. Thank you both.
Kate: All right. And again, your co hosts today have been Kate Holterhoff and Kelly Fitzpatrick, senior analysts with RedMonk. And I want to thank Carol for joining us today. You can learn… Well, actually, let me ask you, where should we learn more about what you’re up to Carol?
Carol: You can check out our website at devsuccesslab.com, pretty easy to remember. Just check it out and, yeah, see what we’re up to.
Kate: And are you on any social media platforms these days?
Carol: I am. I’m on both Mastodon and LinkedIn, so you can find you there as well.
Kate: Fantastic. If you enjoyed this conversation, please, like, subscribe and review the MonkCast on your podcast platform of choice. If you are watching us on YouTube, please, like, subscribe and engage with us on the comments.
