James Governor's Monkchips

Twitter’s Incredible Chonky Boi Metrics. An amazing job, but not for the 99%.

Share via Twitter Share via Facebook Share via Linkedin Share via Reddit

fat cat

I had a delightful conversation the other day with Dr. Jean Yang, founder of Akita Software. She is just as smart as her excellent writing indicates, as well as being extremely personable. The VC company Andreessen Horowitz has published a couple of her essays.

The Case for ‘Developer Experience’

and

Building for the 99% Developers.

It is the latter post which I highlighted in my recent note What is Developer Experience? a roundup of links and goodness, and which I want to turn to again now.

As the founder of a developer tools startup, I’ve talked with hundreds, if not thousands, of software developers over the last few years in the course of routine user research. The common theme in these conversations, even bigger than the need for the product we were building, was an overarching need that is currently underserved: building for real developers, or what I like to call the 99% Developers.

These are developers who are getting work done outside of the hip companies and frameworks, who often get neglected in conversations about “what developers want.” There’s a huge gap between what “developer-influencers” are talking about, and the daily reality of most developers. When you look at what gets covered by the tech media, or the speakers at top tech conferences, it’s often people from high-growth darlings like Airbnb or Stripe, or established, highly profitable companies like the FAANGs.

In fact, there’s a longstanding assumption that companies, outside of a small number of Silicon Valley unicorns, should aspire to have the processes of a “baby FAANG.” But this is increasingly not true. Our users would tell us, often sheepishly, that their practices look nothing like what they are “supposed to.” But for these “dark matter developers,” as Microsoft’s Scott Hanselman calls them, the practices of a Facebook or a Pinterest don’t make sense. Their user needs are different and their team needs are different.

It matters to talk about the 99% Developers because these are the developers building the software that powers our lives — insurance, health care, retail, and banking, just to name a few.

This argument is not just philosophical – Yang is determined for Akita Software to make Observability tools easier to use, to make sense for a company in Des Moines, or Houston, or Cleveland. Or Lyon, or Manchester, or Lagos. Which is to say, for companies that aren’t based South of Market in San Francisco, stuffed with engineers earning half a million dollars a year. She wants Akita to be low touch, drop in and go, Observability.

Which brings us to Twitter. I saw a fascinating job in my feed this week.

Ismet Karahman said:

I am looking for an engineering manager with an experience of leading a group of high performing engineers in a platform organization. You will be leading the Metrics Backend team, responsible for powering the entire #Twitter’s monitoring system for 500+ teams and 40K+ services.

Sounds like an interesting job, solving tough problems at scale, right? So just what kind of scale are we talking about? That’s a lot of teams and services.

We are at 25B metrics per minute while with 70% YoY growth, solving bleeding edge scalability problems. If it excites you, hit me up.

25 billion metrics per minute… Well alrighty then. The data management here is off the charts. This is not the land of standard products, tools and infrastructure, but of custom builds, infrastructure optimisation and amazing hacks. The same would be true at Spotify, or Apple, or Facebook, or any truly cloud scale company.

But these companies are definitely in the 1%. They have a roll your own culture, because off the shelf tools invariably don’t scale in ways that will suit their needs. The fact is the industry needs both engineering cultures. One is about marginal gains at absurd scale, engineering optimisation as a way of life, everyone one call. The other is about normal enterprise scale, people with lives, with more legacy to deal with. But we shouldn’t confuse ourselves that what works at Twitter is needed by most other companies.

Yang is right, but it doesn’t make Twitter’s Chonky Boi metrics needs any less impressive or interesting.

Neither company mentioned is a client.

No Comments

Leave a Reply

Your email address will not be published.