In this RedMonk conversation, Tom Johnson, co-founder & CTO of Multiplayer, discusses the evolving landscape of system design with Kelly Fitzpatrick, senior analyst at RedMonk. Tom emphasizes the collaborative nature of system design, and the necessity for all team members to understand the architecture of software systems. They explore the importance of documentation, the role of system design in job readiness, and the artistry involved in backend development. The discussion also highlights resources for learning system design and the significance of cost-conscious and secure software design.

This RedMonk conversation is sponsored by Multiplayer.

Transcript

Kelly Fitzpatrick (00:12)
Hello and welcome to this RedMonk conversation on system design as a team sport. My name is Kelly Fitzpatrick and with me today is Tom Johnson CTO of Multiplayer who is guested on the MonkCast previously to talk developer experience and why developers need to stop diagramming and start designing. Tom, welcome back.

Tom Johnson (00:31)
Thanks for having me.

Kelly Fitzpatrick (00:33)
And Tom, for folks who may have missed the previous episode, can you give a quick TLDR on who you are and what you do?

Tom Johnson (00:41)
Sure. I’m, as you mentioned, co -founder, CTO of Multiplayer and long time back -end software developer. Got my start in speech recognition and robotics and segued into telecom and internet apps, building systems for companies large and small.

Kelly Fitzpatrick (01:00)
Well, welcome back. And I’m going to jump right into it. Cause I feel like we might have a lot to say on this topic. In various written content, you have made the argument that system architecture is a team sport, meaning in part that it is a concern, not just of a lone architect or people in architect roles, but folks in various roles on an engineering team. And this of course suggests that maybe things have not always been this way. So I have to ask for folks who are not familiar with the evolution of software architecture.

How are things done, quote unquote, back in the day? Not to suggest that you have been doing this for a very long time, although you kind of have. What changes have you seen in software systems that have necessitated this sporting kind of team approach?

Tom Johnson (01:47)
Sure. Well, yeah. So back in the day, know, system architecture wasn’t even really a thing. The developers were kind of like write stuff. Then you discovered like, well, maybe there should be an overarching view of the software as things started becoming more distributed and more components connected together. You would see a system architect role, somebody who would decree, you know, from high above, the way the architecture would be.

that was okay when things were simple. Teams were simple. Everybody was in the same room. and you know, clear division of responsibilities, but today it’s completely different. You have, complex software. you have, fast changing software. You have teams that are, distributed. you’ve got separate roles for a lot of different people. and it’s just.

You can’t have one person in charge of all that anymore. Plus, it’s better that everybody is involved in understanding of this complex system. It’s just impossible to have it all in one person’s head. So it’s a requirement now. I think that everybody knows how their software, even if they’re focused on little bit, fits into the bigger puzzle.

Kelly Fitzpatrick (03:05)
So Tom, taking a step back, it seems that system design is something that some technical practitioners learn about. For instance, when I taught

At Georgia Tech, my computer science capstone students had to create these kind of like detailed design documents for whatever they were building. And I also know that many professional developers have a lot of like thoughts and feelings about, say, UML diagrams. However, there are many folks in technical roles who, regardless of whether it was part of their education, either to choose to avoid system design or are just left out of the process altogether. To your mind, who in an organization should be participating in the system design process?

Tom Johnson (03:44)
Yeah, that’s a great question. And I think that this is one of the trends we’ve seen in recent years. It’s really everybody. Everybody has to have an understanding of the system. They’re just too complex, for, one person to know everything about and sort of dictate how changes must happen. Things are fast moving. You, as a software developer, you might be focused on one part, but you really need to know how it fits in, what depends on it.

what your dependencies are, scaling issues, security, things like that. it’s, you know, plus there are, you know, the other benefits for, for everybody knowing about, system design and understanding how to do it. You know, you have a better understanding the overall like application you’re working on. You can collaborate more effectively with your peers. you can be more proactive with things and spots.

maybe design flaws or bottlenecks and that requires a higher level understanding of like how software fits together. And you have to take yourself out of just the code for your single component and go higher level 10 ,000 foot view and look down and understand how things work together.

Kelly Fitzpatrick (05:01)
I love that idea of taking yourself out of the code for your single component. Because in addition to thinking of the different pieces of, a project or a product, often what we see is that knowledge across teams about what is happening also can be very siloed. So it almost sounds to me that this approach to system design is also a way of kind of breaking down some of those just communicative or like

Tom Johnson (05:18)
Yes.

Kelly Fitzpatrick (05:28)
very practical silos that we see.

Tom Johnson (05:31)
Absolutely. And, and, and that’s a must have, you know, software these days is also not like all developed in -house by one team. You know, it’s now, if you’re part of a reasonably sized company, probably have multiple teams. You might have multiple platforms. might have the APIs you provide to each other, but you also probably have, you probably didn’t write all the software in -house in that company and you’ve got pulling in a SaaS provider for this and a cloud provider for that.

There’s more assembly and composing of platforms from pre -built services, open source components, other things. And that creates complexity, dependencies, and that’s a distributed system. You have to know how things talk to each other, how things scale. It’s just, it’s a requirement that everybody involved knows about distributed systems, patterns, best practices, and things like that.

Kelly Fitzpatrick (06:31)
And I also feel that in terms of who knows what about what patterns I’ve often seen are that there’s like one person who knows how to do this one thing. And should they decide to move on to a different role, suddenly everybody else is like scrambling to figure out what it is that this person used to do.

Tom Johnson (06:46)
Yes. Yeah. And you know, that, brings up, you know, one of the important things of system design, which is having a great source of truth, having great documentation about your system so that, the knowledge in the person’s head as valuable as it can be, as important it is for, know, for the understanding to be in, people’s heads. If they walk out the door, that knowledge doesn’t walk out with them. So that there, that needs to be out in a, in a place that people can easily go to, find, collaborate on. And that’s one of the big things we’re looking to solve at Multiplayer, is auto -documentation of your system and better debugging tools.

Kelly Fitzpatrick (07:38)
Yeah, even you forget people walking out for another job. I remember having to document things just so I could go on vacation back in the day.

Tom Johnson (07:45)
Right. Absolutely. Yeah. So it has so many benefits to get information out of your head. Redundancy, just being able to take a break, being able to move on to another position, work on something else, share it, not be the sole person responsible for the thing that you just developed, and to be able to hand that off and share the responsibilities. There’s so many benefits.

You know, first of all, everybody knowing about how to do system design, but also having a source of truth, great documentation, great tools for debugging take the manual work out of like discovery and tracking out of it. So you don’t, it’s just not more grunt work for a developer to do that. If it’s grunt work, it doesn’t get done. It gets done quickly, probably mistakes, and we’ve all, this is the, frankly, the current way things are.

for, for teams just manually documenting things sifting through APM data for, evidence of a bug, that kind of stuff. And I’ve been burned by those things and I’ve spent too much my life hours, weeks, months of my life. I’d like to get back because there was no other alternative. So, we’re looking to solve those problems.

Kelly Fitzpatrick (08:59)
And I feel like anyone who’s been in this industry for a while has that list of, I remember when this happened and I’m never getting those hours back.

Tom Johnson (09:06)
Absolutely. boy. You talk about, it’s funny, you know, documentation seems like a boring topic, but once you talk, talk to somebody about like, okay, you know, how do you keep track of you? have a system architecture diagram that’s up to date? They’re like, no. Okay. What, what has happened as a result of that? And there you just, people open up like a therapy session, like the horror stories. Everybody’s got horror stories of, of like changing something, breaking this or not knowing that this was there scaling issues or crashes for not having that great source of truth.

Kelly Fitzpatrick (09:40)
Yeah, and I used to work at a startup where we had, we would have, it was called the beer of shame. And if you broke the build, you would have that on your desk until the next person broke the build. It moved around a lot. But I feel like often these things would happen because of exactly what you’re describing, because of just like lack of updated things and being able to get to.

Tom Johnson (10:01)
Yeah, you know, I’ve seen that too. And you know what? And there’s also a collaborative element to this communication. So it’s like source of truth, plus being able to share the information. So design needs to be a team sport. Everybody should know all the things or be able to access this information easily, know that there’s, it’s factual, but also to be able to communicate about it, communicate, because design and development,

is a lot of the problems I see are communication problems. Design should be like a team sport where communication is really easy. You’re basing your decisions and designs on knowledge that is factual and source of truth. And you can work together as a team.

Kelly Fitzpatrick (10:48)
So I buy into this kind of team view of why everybody should know system design from an individual practitioner point of view. Why is it important to know system design? And is this something that can help folks get a job?

Tom Johnson (11:03)
Yes. So apps, it can absolutely help you get a job. In fact, I think the interviewing process has evolved over time. Yes, you still get those questions about like code this in JavaScript or Rust or whatever, where, know, you’re getting tested on your knowledge of a particular language. But I find that interview questions and I’ve shifted more toward how do you build this app or how do you scale this system?

Or how would you solve this problem that’s high level? And that’s all about system design. You have to take the big problem, split it up into smaller problems, solve those smaller problems. And that’s usually a collection of software talking to you. You have a distributed system. So understanding that is super important to get a job. It will help you get a job. And it’ll help you keep the job because you’ll be good at that stuff. No longer is software just like, tell me what

Tell me the little thing that I’m going to develop. I’ll sit and develop that one component. You’re going to be involved in multiple pieces. And it’s definitely, I think it’s a core skill that every engineer should have.

Kelly Fitzpatrick (12:11)
Yeah, and to your earlier point, software today is a lot of assembling, right? It sounds like this is something that could actually help with integrating all of these pieces together and not just building.

Tom Johnson (12:16)
Yes, yes.

Exactly. Like if you were to say, I’m going to solve this problem by building every single component myself or in -house or the team will, and there’s very good solutions out there that you could just use, they’d look at you like you’re crazy, as you probably would be. So these days, first thing you do is you look for stuff that’s already built that you could use, that you could bring in. Because it’s not

maybe core to your business either. So you want to get, you want to move fast. There’s a lot of, software out there whether it’s open source components or SaaS providers, cloud provider features where you can just say, okay, I’ll use this. I’ll use that. At least the other thing assemble it. Okay. And now I’ve got a couple of microservices that are specific to my problem that I have to write. not something I can just go and get somewhere else, but I’m going to wire it all together. So it’s more about like composing.

and writing as little code as possible to get something up and running. So that has to do with system design. The more you get into it, you’ll see patterns, best practices. There are lots of ways to solve a problem. Should you use queuing here or caching there? You ask other questions like, how is this going to scale? If I had a million users tomorrow, what would break? Or questions like, when stuff breaks,

How do I handle it? Those are all system design questions. And as a software developer, have to have answers for those.

Kelly Fitzpatrick (13:54)
So in, in terms of things that organizations have to care about, but developers often don’t have time to, is systems design something that can lend itself to helping individual developers as it pertains to helping the organization cut costs or build more secure software?

Tom Johnson (14:13)
Yes. Absolutely. Well, you bring two key points there, and they’re probably the most important ones. For the organization, they’re going to care about, you can’t just have this huge distributed system and forget about the cost, and we’re going to pay anything. So as a developer, you have to understand, OK, what are the cost impacts of my design?

And that’s important. And that’s helpful to the organization. If you have good documentation, the people, you know, this is part of why it’s a team sport. You know, it’s not just developers involved anymore. It’s managers and other people who have to be concerned about these things, pay the bills. And they might have to weigh in and say, hey, you’re using this, you know, couldn’t we use that? You know, this is very expensive. Maybe this is cheaper. Or does it have to

cost this much and they’ll have questions. And I think, managers are more technically adept. That’s another change that I’ve seen over the years. So they can participate in these discussions as well. And developers will get that sort of feedback and need to incorporate it. But the organization can benefit from, cost conscious design and security is a big issue as well. So we see dedicated

DevSecOps teams that will analyze architecture, not just internal architecture, but say the dependencies that you have externally for security requirements. And they may be required from a compliance perspective, to meet certain requirements or objectives.

And as a developer, have to understand these things. You have to be able to like communicate about these things, have an answer, for that type of organizational requirement. Definitely.

Kelly Fitzpatrick (15:55)
Yeah, no arguments there. So changing gears a little bit, you have also written an article on the invisible artistry of backend development. That’s a great title. I love a good title. And I agree with you that we tend to see front -end development, especially as this inherently kind of creative field, but we fail to see the artistry in, a really well -designed API.

We tend to think of artists as these kind of like lone genius individuals. So how does the framing of developers as artists also fit with the argument that system design is a collaborative undertaking?

Tom Johnson (16:32)
Sure. Well, yeah. So as you mentioned, the front end gets all the kind of love and thinking about what is beautiful about a system. You can see the front end application, the mobile app or the web app, and you might have a beautiful design. But if you don’t have a back end system that’s as beautiful, and beauty might mean scalable, great APIs, great design.

that front end app might not be able to do anything. Okay. So there’s, there’s this requirement that both sides need to be kind of have that, that beauty. and I I’d say that, looking at a backend system, you know, when something is kind of ugly and, know, so when things are hacked together and sort of taped together and, when it’s not, when it’s well thought out, when it’s well designed, when you can see the logic of it, when you can see the beauty, there’s a beauty to that.

And I think that back end developers, it’s a creative act to do system design. You’re asked to do something sometimes seemingly impossible, solve this big problem. You have to then think, it’s probably never been done before. You have to think, how would I do this? You take that big problem, slice it into smaller problems, solve those individual problems. Those are lot of choices. There are lots of different ways to solve the same problem.

sort of artistry comes down to making choices a lot of times. And you’ve got the creativity for how am I going to solve this? you know, people are going to be using the APIs that you design, the software that you’re developing. So there’s an artistry to that. It’s invisible because, you can’t see into the backend system as you can sort of look at the front end. design, but there’s artistry to doing that work, definitely.

Kelly Fitzpatrick (18:20)
Yeah, and I like that you say, like, scale as something like can be a thing of beauty, even though it doesn’t have the same, like, easily recognized shininess as a good front end.

Tom Johnson (18:31)
Yeah, you know, and things like, I mean, maybe this is just me geeking out as a backend developer, but you know, when you think about like fault tolerance and when things fail and what if this whole set of nodes just went away as things, you know, these things happen, like what is, how would the software respond to that? Like a living organism, you know, how would it heal itself? How can it, can it

survive, it continue to work partially available? And to think about what are the algorithms, what are the design patterns, what’s the design to make that happen? And when you figure that out, it doesn’t always work, but when you can figure something like that out, it’s a beautiful thing to see it working. When your app is up and running and things are on fire in the back end, but the front end, the users don’t even notice. It can be magical.

Kelly Fitzpatrick (19:27)
And I look forward to the day where I see a back -end system and it’s like so beautiful that I cry like tears of joy because I have cried many tears of like anguish over the sadness over like, as you your point, like horribly designed things that just did bad things.

Tom Johnson (19:36)
Sadness.

Yes. Yeah, me too.

Kelly Fitzpatrick (19:46)
So let’s say someone out there is convinced by this very podcast and wants to learn more about system design and also is very interested in tools that enable collaborative design, where would you recommend they get started?

Tom Johnson (19:59)
That’s a great question. So it depends on how you like to learn about things. I I love going to YouTube. It’s a great, but growing resource for systems design content. One great channel is called ByteByteGo. Nice, small, easily consumed videos with best practices.

and also stories about how some systems are actually designed at companies. So you get a lot of real world, knowledge for a small amount of investment in time. If you like books, there are a couple of really, good books, So Designing Data Intensive Applications, by Martin Kleppmann and Designing Distributed Systems by Brendan Burns. There are also some great newsletters like the YouTube content where these newsletter providers have real experience and share it and you get to hear that you get to sort of read about the like the war stories and horror stories.

and things like that that are really, you can only get from real experience and it takes time as a developer to get that yourself. one is a System Design Classroom by Raul Junco and AlgoMaster by Ashish Pratap Singh And I think I’m pronouncing the names correctly, but forgive me if I’m not.

Kelly Fitzpatrick (21:18)
We’ll try to get some links in the show notes too.

Tom Johnson (21:19)
Yeah, that’s great. And Jeff Hodges wrote “Notes on Distributed Systems for Young Bloods,” which is good too.

Kelly Fitzpatrick (21:27)
Great. And final question for you, Tom. How can folks hear more from you and from Multiplayer?

Tom Johnson (21:34)
Sure, so you can find me on X or threads as @tomjohnson3, and that’s the number three at the end. I’m also on LinkedIn at Thomas Johnson. And for more about Multiplayer, we’re at Multiplayer.app or @trymultiplayer on the social platforms.

Kelly Fitzpatrick (21:51)
Well, many thanks again to Tom for another great conversation. My name is Kelly Fitzpatrick with RedMonk. If you’ve enjoyed this conversation, please subscribe and review the month cast on your podcast platform of choice. If you’re watching us on YouTube, please like, subscribe and engage with us in the comments.

A RedMonk Conversation: System Design is a Team Sport with Tom Johnson

Links

Transcript

No Comments

Leave a Reply Cancel reply