In this conversation, RedMonk analyst Kate Holterhoff discusses Embrace’s State of Mobile Experience Engineering Report with Andrew Tunall, VP and Head of Product at Embrace, and Virna Sekuj, Product Marketing Manager at Embrace. They review Embrace’s research detailing mobile end users’ and builders’ perspectives of performance issues and app experiences.

This was a RedMonk video, sponsored by Embrace.

Rather listen to this conversation as a podcast?

 

Transcript

Kate Holterhoff: Hello and welcome to this RedMonk Conversation. With me today are two leaders from Embrace.io: Andrew Tunall, VP and Head of Product and Product Marketing Manager, Virna Sekuj. We have a lot of exciting territory to cover today, so let’s jump in. Virna, can you tell me a little bit about Embrace and the problems that you solve?

Virna Sekuj: Yes, for sure. So Embrace is a data driven toolset that helps mobile engineers build better experiences. And we’ve found that in the mobile space, especially, engineers tend to deal with a lot of frustrations around resolving app performance issues and optimizing their end users experiences, especially because mobile is such a unique development environment with a lot of different variables. So what we do is we tackle that challenge by collecting data via our SDK and provide really critical mobile insights into that through our analytics dashboard.

Kate: So we’re going to be talking about an important study that you put on at Embrace. And I’m curious a little bit about what drove you to conduct this research. And, why is mobile app development really in need of this sort of critical lens?

Andrew Tunall: Yeah. Well, mobile is a pretty fundamental part of our daily lives now. I mean, we communicate, work, shop, consume entertainment over the Internet and mobile is really at the forefront of how people experience it. And it’s only becoming more and more important as we see shifts in people’s consumption behaviors. And our experience was that providing a solution for companies to build better mobile apps has shown us that there’s still a bunch of work that people have to do in terms of creating better experiences that really align with the ever growing demands that their users have. Their users expect apps to be really fast, really stable. They are a lot less tolerant of slow app spinners or bad experiences on mobile than they are on traditional web. I mean, I think I was trying to change a flight yesterday on a mobile app and something was spinning for just a few seconds and in my head I was like, “It’s broken” and ended up force closing the app. And so we really wanted to understand a variety of perspectives in the world. Like not only what are the perspectives of our customers, but what are the perspectives of end users and how do they contrast with what developers think about today so that we could help our customers better align to kind of what shifting consumer expectations are.

Virna: Yeah, that was what sparked us to do this research. And I can speak to the methodology about how we underwent this study.

Kate: Yeah, let’s get into that.

Virna: Yeah. So basically, as Andrew was saying, we really wanted to understand different perspectives: app users plus developers, plus the kind of landscape of the industry in general, because we collect a lot of rich data just through our own product. So what we ended up doing was conducting a three part study. So in the first part we ran a nationally representative survey among about 900 mobile app users in the US. So that gave us some really interesting insights. And then the second part, we actually pulled our own Embrace community of mobile developers and engineers across our customers who represent tons of different industries. So that was again also very insightful to compare and contrast those. And then in the last part we actually looked at all of our customers’ app data in the aggregate and built a ranking system to benchmark how different industries were doing and how they were performing across a few different key metrics like crash free rates or incidences of high memory usage, etcetera: some of the critical things that we cover. So those were the ways that we extracted the data for this.

Kate: The first part of your research delves into perspectives from an app user and what are the findings from that particular demographic?

Andrew: Yeah, it validated a lot of what I think seems intuitive, just to reflect my anecdote from earlier. We found that intuitively, you think if something doesn’t seem to be working — I mean, we’re all mobile app users at this day and age. So like, if something doesn’t appear to be working, it creates a lot of user frustration. We’ll go through the specific numbers or Virna can, but you know, we found that user tolerance for delay — slow running parts of the app — was not particularly strong. People who appeared to see the app running slowly… they didn’t like it at all. And I think that makes a lot of sense. I think what’s interesting is that the underlying causes aren’t so abundantly clear. If you think about the old game where a mobile app was really a reflection of your web experience and was kind of just a different form factor for communicating with a core set of services… I think the old observability game where it’s like I look at first party network requests to my core services kind of works. But mobile apps now are pretty complex beasts. They’re compiled code that live on your device and people get frustrated with simple stuff like, you know, I’m trying to transition to another screen. It’s doing an animation, it’s super janky or it’s trying to process through data that’s locally stored and taking a long time. And so I think it was interesting to see the real users of apps reflect what intuitively our own lived experiences are. But, in terms of data, which is always helpful.

Virna: Yeah. And to just add some context to that data. We found that people in general are experiencing issues all the time. So you have this situation where the expectations that app users have right now are so high because we do everything on our mobile devices. We want things to be fast, we want them to be stable. You know, we’re not willing to wait more than 30s for a key flow to be done. So the expectations are really high. But then at the same time, we’re dealing consistently with performance issues on — for many people — a daily basis. Our data found that two thirds of app users said they were experiencing any of the sort of slew of issues on at least a weekly basis, which can include crashes or slow startup or images or video just loading very slowly. But almost half of people actually said that they deal with these types of issues daily. So there is that situation where we have very high expectations, but apps aren’t always meeting those expectations, which was sort of an interesting part of the research that we found.

Andrew: Yeah, and the other thing that was super interesting was just the consequences, which I think, you know, ordinarily we always thought of, okay, user has a bad experience or something doesn’t work as designed. Maybe you missed the sale. But apps are interesting in that deleting the app off your phone is essentially just like deleting it from the internet, if you will, right? Like if it’s something that you use as a primary conduit of interacting with that brand, the act of removing it from your device is essentially just removing your eyeballs entirely. And so you think about, you know, marketing teams use push notifications as a mechanism to gain your attention in critical moments. If you don’t have the app, you’re not getting notifications. And I think again, it kind of reflects all of our lived experience: the shortening level of patience, but also the consequences for not doing the right things. If you don’t build an experience that lives up to my expectations, that starts to reflect on your brand. And because that’s how I interact with you, I eventually choose another one.

Kate: Right. The stakes couldn’t be higher and 50% of users is a tremendous number. So it really suggests the importance of trying to lessen these annoyances that end users are having. So I guess what I’m curious about is what tends to be the worst of the worst here. What are people the most frustrated by?

Virna: Yeah, it’s definitely crashes. Crashes are the worst of the worst for sure. The vast majority of people we talk to find them very frustrating and not tolerable and would actually delete an app off their phone. If it’s crashing a handful of times, it’s kind of gone. So that wasn’t super surprising as that being the worst. But one of the interesting things we also found was how some other common issues ranked in relation to other ones. So obviously crashes are terrible. But one of the ones that wasn’t as bad as we thought it would be was startup time. So slow startup wasn’t seen to be as frustrating or as intolerable by users as when they can’t interact directly with the app. So if the buttons are unresponsive or the links are unresponsive or the forms you can’t fill in… those were found to be much more frustrating than slow startup. Which is surprising because working in the industry, startup time tends to be one of those measures of performance that people are constantly trying to reach a certain threshold of acceptability when actually there might be other things that your users are getting frustrated about even more so.

Kate: That’s a good segue into a topic that’s near and dear to my heart, which would be developer experience and the perspective of the developers who are actually creating these apps and maintaining them. So you just said that there’s some misinformation, a misalignment between developers and the end users. And so when it comes to addressing certain issues, this misalignment is something that we’ve definitely noticed at RedMonk and continue to follow very closely. And my colleague Stephen O’Grady has spoken about this in terms of the developer experience gap, which just basically means that there is a difficulty in piecing together all the tooling that’s needed for modern infrastructure, and that burden tends to fall on developers disproportionately. So I’m curious how Embrace is taking all this information that you have learned from this research and you’re grappling with the challenges that developers face in leveraging the increasingly complex tool sets. And can you talk more about what you found in this research and how it addresses the developer experience in particular?

Andrew: Yeah, absolutely. So I mean, I think one of the interesting things that I thought about with the startup data was the why. Why I think developers are super focused on it is because every app starts up and also the companies that publish apps into the marketplace tend to punish you if you have apps that exceed certain unknown thresholds in terms of startup time. And so it’s a known problem that people can build a tool for and address. I think what you highlighted, though, is the larger issue, which is we all know that there are these unknown challenges; there’s usability problems, there’s feature problems, there’s things going awry in the app that people can’t necessarily identify. And honestly, it’s a bit like a game of pin the tail on the donkey. It’s like — you get a ticket or you get a customer with a bad app store review and then it’s like, Well, how do I wander blindly through the desert? Swinging around trying to find something that might be the root cause? And I mean, I was never a B2C app dev, but I did — for most of my career — I did web apps in B2B world and we would get tickets all the time. It was like, users experiencing X, Y, Z. What’s going on? And the challenge to recreate the issue or understand why people weren’t completing a certain action was pretty monolithic.

Andrew: I was talking to actually another one of our product marketing managers about this earlier as we think about our narrative and how we talk about what our product vision is and what the company is and how we talk about it. And the amount of toil that gets introduced to engineers becomes just punishing for organizations. And, the old world where you had a few engineers working on your mobile app — and again, it was really just a different form factor for interacting with the web, which was where a majority of your users were — that toil, while it stunk for the small handful of mobile engineers you had, wasn’t crippling for the organization because the critical roadmap wasn’t on that team. But we’ve seen organizations over the past few years go from three engineers on one team to like 50 engineers across seven. And so when you start having 50 engineers experiencing a bunch of that toil… the impacts on your roadmap and the ability to execute on it, the ability to have engineers feeling like they’re highly successful and doing what they came there to do, which is probably not fixing crashes or hunting through a bunch of random data, trying to find an answer so that they can keep somebody happy… that’s not what they signed up for. And so, I think the interesting thing for us is, how do we shape product strategy that hopefully over the long haul gives people more answers. It lets the members of those builder teams, the product managers, designers, even marketing or business analytics and engineers use common data sets so that it’s less pin the tail on the donkey, more, How do we build stuff that really delivers an awesome experience for our users?

Kate: I like that metaphor, but I want to dig down a little bit deeper into this developer perspective problem, since there’s a lot of insight in your research that really addresses the sort of issues that they’re seeing on a day to day basis. So when it comes to monitoring performance in general, not just around certain issues, what did you find that maybe surprised you that is going to affect how developers work?

Andrew: Yeah. Again, it validated some of our lived experience which we spend all of our days talking to mobile engineers. So it’s kind of unsurprising, right? But we found that app performance, generally as a category, was super important. And what we didn’t expect was that it was actually like their biggest concern right now, which was really interesting. I think we hear this a lot, which is, we suspect there are things that aren’t working as well as they should and are taking a long time. And we want to know why. And I mean, this is reflective of what they’re probably seeing in app store reviews in their own customer feedback. And we saw it in the actual user data that was shared with us that we got as well.

Kate: Do you have an answer to that? Why is it?

Andrew: You know, I think mobile devs are fundamentally creators. And so, I think creators who signed up to build great experiences tend to be pretty clued in to their users’ experience as opposed to viewing themselves as operators where they’re just pushing the buttons of the machines. The people that consume their service, if you will, are actual human beings. I mean, whether you’re a web dev or a mobile dev, you signed up because the delight of an end user in the eyes, when they walk through the app, using it, when you see it in their hand in public, is what gets you excited. Not necessarily seeing the line on the graph that shows, you know, p95 latency. Which is just interesting, right? Latency on endpoints is important, but the fact that users are irritated when they use your app because things take a long time and they’re not in love with it? That, I think for a lot of mobile engineers, is something they’re highly attentive to.

Kate: Right. So Android and iOS are very different platforms. Did you come across any major differences in what end users or developers mentioned when they are speaking about that? You know, in these particular operating systems, what are the fundamental differences that came across in your own data?

Virna: Yeah. So we often just speak about the differences between Android and iOS because they’re quite different platforms to build for and everyone is always trying to equate the experiences if you have an app that goes across all different ecosystems. And the research helped provide some kind of insight into that. So in general, we do see iOS apps perform better than Android apps. So in the full version of the report, we actually ranked apps aggregated into industry categories based off of certain performance metrics like I mentioned earlier. So, you know, crash free rates or rates of eating up memory. And we split some of these metrics up by Android versus iOS apps. And in general, we did see that Android apps rank lower when it comes to these critical metrics versus iOS.

Kate: That’s really interesting.

Virna: Yeah, right. So there’s a lot going on behind the scenes with how Android versus iOS actually work under the hood. And it gets reflected in the data and it gets reflected in app users’ experiences as well. So when we split our group between Android and iOS users, the Android users were complaining more frequently about slow startup versus iOS users, and even Android devs were saying that they prioritize dealing with slow startup more. So you definitely do see some performance differences.

Kate: Wow. And do you have a sense of why that could be?

Virna: So the Android ecosystem is quite different than iOS because it’s just so vast and so varied. You have a lot of older devices out in the market, a lot of devices that are running on older versions of either the Android operating system or an older version of whatever app that you have out there. So there’s not the same amount of consistency and control. Whereas, with iOS, you have a much greater amount of control as to what you can do and standards and regulations. So with Android, obviously having a broader kind of system to work with allows for a lot of freedom for devs who are building for it. But it comes with that risk of performance issues because things can be more spread out. And that’s sort of what we tend to see reflected in this type of data.

Andrew: An interesting thing about Android, though, there’s obviously a ton of risk, but there’s also a ton of opportunity. I think if you look worldwide, your average per capita income is substantially lower than it is in the United States. And so affording an iPhone is out of the question. I mean, there are entire countries and continents where Android is 99 plus percent of the usage volume. And so, many times it’s older devices and that means that your limitations that you have on what your app can do — the stakes are pretty high right? You have to be ruthlessly efficient. You can’t be like hogging resources, but it’s not like there’s zero opportunity. I mean, there’s literally billions of people who are unserved by the iPhone market that you can tap into. And there’s ad dollars at play. There’s opportunity to surprise and delight them the same way there is in the United States. It’s just a wholly different group of people that you need to meet. And the only way to really effectively do that is by understanding the impact that your app has on those different kind of cohorts of individuals.

Kate: Right. Talking about older mobile devices is a good segue to talking about the future of mobile development. Did your findings reveal anything about where we’re going to be going in the next five, ten, fifteen years?

Andrew: Yeah, I think we found what will surprise exactly nobody… which is that everyone’s excited and probably simultaneously somewhat fearful of the future of AI. And I think there’s opportunity across the board. I know there’s opportunity for us when we think about AI replicating what human beings do during the troubleshooting or like app performance analysis experience. But if you think about consumer apps or really any app ecosystem, the ability to build highly adaptive and more personal experiences through AI could never be greater. I mean, it just further creates this set of winners that think if they really harness the ability to build something that’s differentiated will set themselves apart from the crowd, right? It becomes well, everyone does this, but *this* thing is magic. And, you know, I’m a big fan of Lenny’s growth Podcast. I listen to that all the time. I think a lot of product leaders get on that. And I’m a big believer in this, too, which is: every great product, whether it’s B2B or B2C, there’s element of magic. And the ability of AI to help us deliver magic that feels incredibly personal to you is totally there. I mean, I think this is where the fear comes in, which is like, I’ve been at this game a long time. I might look young, but I’ve been doing product management for almost 20 years in the tech space. And I’m not going to pretend to be an expert in generative AI at all. The Internet’s going to look a lot different in five years, in ten years. I mean, in ten years, people who grew up using mobile apps probably will have a totally different set of expectations. And that means it’s going to be a crazy couple of years. It’s like the world adapts and expectations change.

Kate: Right. As we’re coming to the close of this conversation, I’m curious about hearing you summarize what you would say are the greatest takeaways, both from an app user and a developer perspective. What are the main threads that we should keep in mind based on the data that you’ve shared with us?

Andrew: Yeah. I mean, I think there’s a couple of things. First, it’s worth really diving into what concerns users express versus what an industry standard or even a particular OS might enforce upon you. I think the burn on user expectations not being met versus not checking the box that Apple or Google might make you check… it’s longer, but it’s probably more punishing overall because if people don’t find that your app experience is great, eventually they’ll uninstall the app. It’s an area where there’s a lot of opportunity to really create better overall customer experiences. I think that turns into the second one, which is poor user experience has more lasting negative consequences for a business and brand. Because I can uninstall the app, I’ve essentially turned you off and I think a lot of developers realize this. This was reflected in the data we saw from software engineers who wanted to prioritize app performance. But I think decision makers where there’s a product manager or an executive team probably would be well served to recognize it as well. And then I think the third one is interesting, which is, we tend to get set in our ways and do a lot of things the same way we’ve always done them because, well, I’ve got nails and this hammer seems to work just fine, but it doesn’t mean it’s the best way to do it and it doesn’t mean it meets the needs long term of what you’re building. And so I think organizations are well served to think about if they’re really giving their engineering teams the tool sets that they need. I mean, we talked about this at the very beginning, to actually get at the answers as opposed to just kind of checking the boxes as they exist today.

Kate: All right. I want to thank Andrew and Virna for joining me today. If you would like to check out the report for yourself, you can find it on the Embrace.io Website and I will also link to it in the show notes.

Andrew: Thanks, Kate.

Virna: Yeah thanks, Kate.