AI agents are writing code faster than anyone can build, test, and ship it. James Governor talks with Helen Altshuler, CEO and co-founder of EngFlow, about why the bottlenecks that once happened at Google scale are now showing up everywhere. Helen traces the story back to Google’s 2008 push for five-minute builds, which produced Blaze and eventually Bazel, and explains why that system is finally having its moment a decade after it was open sourced. The numbers are hard to ignore: EngFlow’s customers went from 250 million build actions a week to nearly 2 billion in a single year. Companies like Snowflake, Databricks, and Tinder are using Bazel, managed by EngFlow to keep up. James and Helen also dig into agent-ready documentation, why verification is increasingly automated or even ignored, and what happens once people stop using pull request workflows. We’re about to hit a wall of AI created scale, and companies are going to fundamentally rethink architectures and approaches if the want to make it.

This RedMonk video is sponsored by EngFlow.

Links

• LinkedIn: Helen Altshuler

Transcript

James Governor (00:04)
Hey, it’s James Governor. We’re here for another MonkCast. Today I am joined by Helen Altshuler, CEO and co-founder of Engflow. And we’re going to be talking about the challenges of build at scale, build management at AI scale. So I think from my perspective, and I’ll jump in a little bit. Well, first I should welcome you Helen.

Helen Altshuler (00:24)
Thank you. Pleased to be here.

James Governor (00:27)
Right, so what we’re seeing in this industry at the moment is pretty clearly with the onslaught of software being driven by agents, being driven by frontier models and assistive coding, we’re just seeing more and more software being built. And there are bottlenecks being created in terms of our processes for getting that software into production. This is a very real problem. I’d say the industry is sort of creaking at the seams. We’ve certainly seen problems, for example,

outages at vendors like GitHub. And so all in all, this is a real industry problem. We’re creating so much software at such pace that, the bottleneck is, as I say, getting that into production. Now, I think what’s interesting is that historically, perhaps, we haven’t all had these sorts of scale challenges. We haven’t had engineering organizations that are large enough. We haven’t had sophisticated enough

are operations that we can be thinking about, even rolling out the applications that we’re building at this kind of speed. But it seems like now with AI, the world has changed. And I think that’s kind of the topic of the conversation today. So Helen, I think one of the key questions for me is, you came out of an environment of high scale. I’m gonna ask you a bit about your history, how you got here.

But do we all face the kinds of challenges that an organization, that a web scale organization faces? Is that really where we are with AI? So let’s hear a bit about your journey, why you are well, you’re the right person to ask about build management at scale.

Helen Altshuler (02:15)
Sounds great. So yes, our industry is going through a seismic shift. It’s quite clear. The code growth significantly accelerated with AI tooling. And as you mentioned, causing outages, really pushing the limits of every developer infrastructure on its way. And so it’s interesting to see this because it is familiar environment and it is something that I’ve witnessed the

successful results of while at Google. And that’s for that reason, we’re seeing the same kind of challenges now faced by the industry challenges that Google itself faced in 2006, 2008. And so I would like to kind of bring us back into what happened with code growth at Google and what did Google do? And now the rest of the industry, why are we all looking?

at Google’s build system called Bazel. And so in fact, what we see now is that Bazel is having a moment 10 years after it’s open sourcing. And so what happened? What happened to the rest of the

James Governor (03:27)
There’s a couple of things here. One is almost all great overnight success stories took 10 years. That is a common pattern. we’ve got the overnight success, Bazel having a moment. But yeah, also we love lessons from history at RedMonk. We do think that to truly understand what’s going on in technology, it always makes sense to understand the foundations of how we got here. So yeah, I’m super interested to hear about. the problems Google was facing and and Bazel and so yeah over to you again.

Helen Altshuler (03:59)
Sure. Google was facing the building of scale challenges in those years, in 2006, 2008. Now Google is doing something unique, which the rest of the industry caught onto as well, which is monorepo style development, which means all of the code base is in one place and everything is built from source and everything, all of the tests and builds across different projects are in one repository.

And so that allows Google to do a ton of efficiencies and automation, but it also means that Google’s builds are huge and Google’s tests are taking a long time. And so over that period of time, Google noticed that the Python scripts that they use to read it, to compile the language and wrap around the two million lines of code make file cause significant problems and

That’s when in 2008, Google decided to staff this project called Revolution Fix It, which one of our current employees at EngFlow, Mike Bland, was the lead for while he was leading the testing group at Google. And that Revolution Fix It was aimed to bring together the powerful tools that were in place and powerful tools that were being developed into one cohesive system.

that is changing how builds are working and changing how developer workflows are. And in that timeframe, Google staffed the team and did a lot of prep leading to it. Some of these projects were in their nascent form. And essentially, it led the effort to revolutionize its developer infrastructure with the goal to get to five-minute builds for any project.

that meant that builds could no longer be done sequentially. There had to be a system that does it in parallel. And so it led to the creation of Google’s internal build system called Blaze, which was the starting point for Bazel that we’ll talk about later today. And also building a parallel engine called Forge, which is the parallel execution of builds at scale, as well as the file system that

that really was meant to address the source scalability. And so what’s really interesting to me here is that around the same timeframe, Google noticed that it needs to do massive change in the developer built infrastructure. So like that’s where it kind of like thought about build management at scale without actually calling it build management at scale. At the same time, there were also efforts to do source at that scale. And so it’s also…

quite interesting for me to see that today we’ve seen lots of companies starting to re-envision GitHub, the new GitHub, the source management at Scale because they pushed to the limit. And back in 2008, this is also when Google migrated from one code management system into its own created one. So Scale is the driver, the… 10 years ago, 15 years ago, it really was about Google building better internally as it was growing as a company. And today we see in parallels that teams started adapting AI and now everyone is reaching Google level scale faster than ever before.

James Governor (07:37)
Yeah, some of the growth curves are, you know, honestly, frankly, absurd. know, in this industry, I guess, we keep seeing again and again, you know, the fastest growing ever thing. But that doesn’t seem to stop, you know, whether we get the fastest growing company ever to, you know, a billion ARR or just the sort of the scaling characteristics, you know, sort of a chat GPT is an amazing example.

fastest growing consumer product ever. We’re seeing sort of new markets. This sandbox market at the moment, I think is a funny one where there are like multiple vendors all doing almost geometric sort of growth because of the, just frankly the growth is, as they say, off the charts.

Helen Altshuler (08:29)
Yeah, and this just a few weeks ago, or maybe even just last week, Anthropic published what’s happening internally with respect to code contributed by person by quarter. And we see in Q1 of 2026, it jumped from previous quarters, which were around one to two X to almost six times.

And Q2 of 2026 is eight times contributions by person or I imagine they mean agent as well in this particular case. And so that level of growth really requires rethinking of the developer infrastructure. And the symptoms we’re hearing are first felt on the code side. And so you can hear companies complain about pull requests. So the volume of pull request reviews has

has increased significantly. We have some of our customers that are reporting three to 10 times more. But pull request is now the bottleneck, which is throttling the build problem. And so the first issue that we’ve seen, and GitHub famously created a lot of interesting charts about the growth of code and pressures that it put on GitHub. So source is the front of it. Source is under pressure.

The next one is review. So code reviews are the challenges and are becoming a little bit of this like throttling mechanism for the real problem that is starting to surface, which is build and test infrastructure. And as soon as we addressed issues with codes, issues with reviews, we will start seeing the challenges that basically we’ve been working for more than a decade on.

with builds. And so this is the moment. This is the moment. So Bazel has been adopted in its nascent C 15 years ago, 10 years ago by companies like SpaceX and Uber and Robinhood. And it was mostly because even in its beta state, was about the best built platform for scale. But now we’re seeing Bazel is having its moment really as the only solution for this level of scale. And I’m happy to talk more about just the basal adoption and what we’ve seen here as well.

James Governor (10:57)
I’d love to hear about that. I think that we are always, yeah, as I say, the history of the project, we’ve seen this in other areas. When we were thinking about container orchestration, an organization that had been doing that at scale and built something that was gonna benefit the industry in this case, Kubernetes, was Google.

Certainly Google meets scale challenges, well, before a lot of the rest of us. But yeah, no, I’d love to drill in a little bit then into what you’re currently seeing. As you say, there were some early adopters there. But I think also the balance between, you you’ve got an open source project. So let’s talk a bit about the actual scale challenges and then a little bit about why, you know, why Engflow is the right organization. to be standing this up.

Helen Altshuler (11:57)
Of course. So as far as the challenges, obviously, scale is the pressing one. But there’s also other benefits of using Bazel. It’s hermeticity and security and its ability to know that all of your inputs were accounted for and you’re using the exactly tested versions in creating a shippable artifact. And so.

Bazel is now being adopted primarily for scale, but also for its extensibility and ability to help organizations standardize on the common developer infrastructure. And so in 2017, when I actually created the first BazelCon while still working at Google together with my colleagues, we saw 200 people packed into the room.

in Sunnyvale and those were the early adopters of Bazel. So 200 engineers, maybe one or two per company, but behind each of them, there are thousands of engineers using built pipelines that these folks were creating. And this was the first kind of community mindshare coming together and deciding that this is now the time to create more

tooling to create more community governance and start exchanging ideas around making Bazel scale. And as the community grew, so I’ve been involved ever since in open source community creating the conference, giving talks.

James Governor (13:40)
By the way, mean, I guess a lot of our listeners will know about this, but that’s a very magic moment, that first conference for a project or product where, yeah, mean, having been involved in a few of those, and certainly there’s something, you know, it’s just around that Dunbar scale, it’s the first time the village gets together, and you’ve probably been working together, sort of, offline and coming together. Yeah, no, it’s a beautiful moment. So that must have felt great.

Helen Altshuler (14:17)
Yeah, it was actually interesting to see that at the time Google did zero marketing for Bazel. So Google really built it for itself and open sourced it only because other companies started creating similar solutions like Meta that’s created Buck and Twitter at the time created its own pants system. And so this was a wake up call for Google that the standards are going to start being…

really having a challenge if Google does not open source the build system. So while we’ve created that conference to bring together the mind share, we actually had lots of collaboration with engineers at Meta and at Twitter at the time and converging on the language that is used for Bazel definition, which is Starlark. And later build systems also started adopting that language. It is also something that Buck2 is using.

from Meta as well. And really this was a place to bring together folks working on the scale challenges at various companies and discuss how to make Bazel extensible and adaptable. And so that was the moment where rules ecosystem became more prevalent. so Bazel in itself is a powerful tool, but what makes it more valuable is that it has extensive rules ecosystem where

developers can have the same build platform for JavaScript or Java or C++ where traditional tools you need to have different build platforms building these components. So one of Bazel’s unique elements was that, and a lot of the talks in the early BazelCons were all about rules, creating rules, creating migrations, using rules to have more effective migrations.

using Gazelle, which is the open source tool that’s created by one of our EngFlow engineers while still at Google at the time. so community saw the value and community saw the need for strong investment. At the time, that investment came a lot from Google. And over the years, the balance was shifting with obviously with companies like EngFlow being created with Linux Foundation that we’ve just established to create governance around Bazel.

community tooling and ecosystem. We are now seeing the move and maturity of this ecosystem. And that to me is also a good signal why more companies now than ever before are adapting Bazel right from the start because the community matured, but also more importantly, things that were rough edges of Bazel.

AI is smoothing those over. So for example, one of Bazel’s rough edges is its learning curve and documentation. And my team actually is also investing in this and making Bazel documentation better, collaborating with engineers from Astranis and several other companies making it better. But really, I think we’re now making documentation better for agents to read. And it’s been more and more developers are interacting.

with their agents to learn how to use Bazel to do Bazel migration itself, which is also one of the rough edges. And then perhaps the most phenomenal one is Bazel never had like really strong IDE integration. And so now we are living in a world where IDEs are at least the visual ones or less of a dominant code creation engine.

James Governor (18:00)
100%. So the sort of the Claude code, I mean, Claude code is pretty good at handling those kinds of integrations.

Helen Altshuler (18:10)
Absolutely. What we’ve seen from our customers is that before they would put error messages from Bazel into Stack Overflow or just ask on Bazel Slack, overwhelmingly now, Claude is pretty good at answering those questions. And I hear good feedback from developers using Codex and other platforms as well.

James Governor (18:33)
Yeah, well, that’s, it’s a great story. You know, a thing with a few, you know, a few, as you say, rough edges can get smoothed out in this era of AI tooling. And to your argument about, know, Bazel’s time is now, yeah, definitely. I mean, it’s just interesting because so much of the developer experience that we’ve said is so important is now agent experience. So.

Let me ask you a quick question about that. I mean, what are some of the things you’re doing to make sure the docs are agent ready? I mean, there’s the obvious, you know, publishing and markdown and so on, but what are some of the specifics that are about making, I mean, well, it’s a very different definition of accessibility, but agent accessibility. What are you doing in that regard?

Helen Altshuler (19:22)
Yeah, let me, in that point, also reveal another cool feature of Bazel and use it as an example. So Bazel has tremendous amount of data that it outputs, which is really hard for humans to process. It’s got build event stream. It’s got Bazel query. It’s got a profile information, Bazel profile itself, the tracing information and all of that data. Today we are processing through

or at least for the past six years, we’ve been processing by building relational stores, by creating observability as part of our platform. But more and more, I’m now starting to see agents are using that information much better, but burning a lot of tokens on the way. And so what we implemented at EngFlow more recently with this customer insight that we’ve gotten is MCP and skills that are

using deterministic information from profiles to help agents come to the conclusions faster and complement the default to this discovery when deterministic approach is not working. so that’s the big part, like using what you can about your data and deterministic way to answer it.

James Governor (20:46)
Yeah, super important. mean, you know, look, it’s July of 2026. You’re not going to be talking to any organization that isn’t concerned about how many tokens they’re burning. So, yeah, that’s super important. we got some of the sort of the journey from Google, you know, to more general purpose. Maybe let’s cut over a bit to…

To these questions of customers and scale. yeah, it’s always, engineers love stories, numbers about high scale and so on. So yeah, have you got any examples about current state of the art and what you’re seeing in your customer base?

Helen Altshuler (21:36)
Indeed, indeed. We actually did that analysis recently for our internal team summit just to understand what is the scale that we are processing today across all of our customers and how that changed in just one year. And remarkably, what we saw was that in May of 2025, across all of our customer clusters, all of the companies combined that we’re managing were executing.

250 million build actions per week. And so this is your build and test workflows. So it’s a good metric for your developer velocity, but also for the complexity and scale you’re working with. And just within a year, by May of this year, we’re seeing close to 2 billion actions executed by the same kind of set of customers per week. And that, to absorb that level of scale,

like EngFlow had to do a lot of work over the past year. And the reason we did this internal study is because it didn’t feel like we were absorbing that scale on a day-to-day basis. From time to time, we have a customer request to look at some tuning, cost tuning, or some performance tuning. And so we’ve been investing in this along the way. But when we look at the results, we’re seeing that our customer applied more recently actually commented on that, that they’ve absorbed

three times the number of pull requests without any change in their cloud or or EngFlow license infrastructure. And so that’s a way that we can help companies scale sublinearly scale their costs and have a better efficiency in the overall cycle.

And this is where like, I’m going to bring it back to like, again, why Bazel, why Bazel is having its moment. And we do have customers that are not using Bazel, but Bazel is uniquely positioned here is because…

James Governor (23:40)
I just want to hear a little bit about that when you say customers that aren’t using Bazel because like, you know, there is a world out there that is not all Bazel. There are customers that have alternatives. Migration can be hard. What do you mean some of your customers aren’t using Bazel? I thought you were the Bazel company.

Helen Altshuler (23:56)
That’s right. It’s interesting. I let the cat out of the bag. So we are a Bazel company, but we actually our own thinking as the team that built Bazel at Google and like we are at this point, the largest concentration of Bazel experts anywhere in the world here at EngFlow. We actually weren’t bullish that Bazel will win the market in 2021 when we started implementing some of these systems. And we looked at the approach of

Well, developers will always want to have choices. So can we bring some of Bazel’s best features into the older generation build systems or into other build systems that may be compatible with Bazel level of thinking? And so actually Bazel has a ton of advantages, but the magic is not only Bazel, it’s remote execution API. And any build system that implements that remote execution API

means that they are able to parallelize their workload on large number of smaller machines as opposed to smaller number of large machines, which is where a lot of the performance and cost savings come from. And so we basically implemented a remote execution service that works with any built system that implements remote execution API. So Mathos Buck 2 is implementing that.

And Android platform is implementing that AOSP. And so we increased the number of customers that are using Bazel within, sorry, that are using remote execution within that stream. We also acquired a company, tipi.build a year ago, and they have been working on making C++ CMake build system.

more compatible with the modern concepts. And so we connected that through a remote execution API. And so we’re able also to get them the benefits of parallel execution. But with all of this, while remote execution API and parallelism is really, really essential for speed and performance, Bazel has other advantages. And this is why we’ve seen companies benefiting from having more

tooling that is built on Bazel that may not exist for other build systems. And that tooling is what’s enabling companies to scale. It’s the security that I mentioned, being able to create a software, a bill of material as part of the build process, which is becoming much more relevant. And it’s analyzing vast amount of data from Bazel to improve the underlying performance of their

their build processes. And so we call this build graph analysis. Bazel uniquely provides visibility into this that allows companies to parallelize further and perhaps not continue to scale through adding more hardware at the problem.

James Governor (27:03)
Is this something that organizations can run themselves? mean, it’s an open source project. mean, you’re making it, it sounds very cloud-like, but I mean, obviously there must be organizations that have reasons that they’d want to be running their own infrastructure.

Helen Altshuler (27:21)
Indeed. And what I’ve been also saying recently to my team is that I’m seeing signals that AI future is multi cloud and by multi cloud, means run everywhere where you can get good amount of resources. So Bazel itself is an open source project that you can install on any infrastructure, be it on-prem, your personal machine or cloud.

EngFlow provides the platform that distributes Bazel builds and parallelizes them with observability in a most efficient, cost efficient and performance way. In the cloud, we support multiple cloud environments such as AWS, GCP, and also building out the Kubernetes based solution that allows you to run it anywhere on-prem or…

or on any other flavor of Kubernetes like Azure Kubernetes I’m seeing also is on the rise right now. And then we’re hearing neoclouds like Nebius and CoreWeave. Oracle cloud has reached my radar as well recently. it’s actually quite remarkable that from Bazel’s perspective, it doesn’t really care where it runs, but being able to paralyze it effectively, you need to know a little bit more about

the auto scaling of a particular cloud, the storage system over a particular cloud on-prem infrastructure. And that’s where we focused the past two years at EngFlow specifically, and we built more efficient auto scalers than GCP and AWS native ones. So Snowflake is one of our customers that actually ran it themselves, that ran Bazel and Remote Execution open source project on their own AWS infrastructure.

And by simply switching to EngFlow, they cut their cloud costs in half because we are that much more efficient in utilizing the cloud infrastructure for the same performance benefit.

James Governor (29:27)
So Snowflake made the transition from running it itself to running it on EngFlow.

Helen Altshuler (29:32)
Yes, yes. And the case that is coming up.

James Governor (29:36)
One of your sales people definitely made their quota that quarter.

Helen Altshuler (29:40)
This was one of those deals where I got a call from Snowflake CFO, just trying to understand how they’re making money on the software that we are selling them. So this was a really interesting opportunity for us. And since then, we’ve seen more and more companies. And so you asked me to share some numbers. Last November at BazelCon, I actually gave a talk together with a staff engineer at Databricks, also our customer.

they reveal the numbers. So Databricks is at 3000 developers. Their code base grew to a hundred million lines of code in their monorepo. I expect that it’s close to double now at least. They have seen twice as much code growth in a year and they were like executing a billion actions a day. And so that is a really huge scale. And now when we’re talking to customers at this scale,

when they’re anticipating more and more load, mostly agent-based load, we are asking them questions. Have you looked at your dependencies? Did you look at the build graph? I actually published a blog post a few months ago about my week in a field, just like being on calls with customers and meeting customers at various industry events and dinners.

And hearing these signals and the signals are overwhelming that CI is under pressure. The company is trying to figure out how to paralyze their, their, their builds, or they’re already using the latest tooling like remote execution. And now they’re investing in their bill graph optimizations, which is what we can surface with our analytics for companies to scale smarter. And so the goal for me creating that blog post was how can we help companies

scale smartly because even if you can throw that much hardware at a problem, we’ve also seen our customers that are running 200,000 core clusters getting cloud provider stock out issues. So there simply isn’t enough computer available at the time that they need. Back to the point as well.

James Governor (31:58)
By the way, we’ll put a link to this blog post in the description. yeah, no, we are seeing these panelists. And also, mean, quite a nice position to be in to say, hey, we’re running builds for the two arguably the key players in data infrastructure of the current era.

Helen Altshuler (32:24)
Yeah, I would say that there are clearly leaders in seeing the volume from AI and investing in developer infrastructure early. And obviously, like Databricks and Snowflake are those leaders. MongoDB and other customer also has invested in

James Governor (32:43)
You’re racking up all the data.

Helen Altshuler (32:46)
We right, we own the large scale data space. But more importantly, at this point, there is no company that can handle the scale that we’re running at, which is like 200,000 core infrastructure, know, like two billion actions per week. And so we started by…

having these companies as early adopters because they are seeing signals before everybody else. early majority is coming. And like one of the examples I wanted to actually also highlight Tinder published a really, really great blog about how their investment in Bazel and in parallelization and hermetic tools positioned them so well that now with cursor and Bazel, they’re able to do their software upgrades much, much faster than ever before.

And so we’ve seen companies learn from each other. They read the blogs, they learn what their peers are using, which is by the way, one of the reasons why we run BazelCon is bring all these peers together to talk about latest tooling that they can take advantage of. And this is an example of fundamental AI infrastructure investment. So Tinder made that investment fairly early.

So did Databricks, so did Zooks, another company that we work very closely with. And Zooks has migrated from their DIY solution to EngFlow and again, like with cost savings, but really extensibility and supportability of the new level of scale that they need to be working with.

James Governor (34:26)
Let me ask you a question. So you’re I mean, look, if I if I if I cut you Helen, you bleed Bazel. I mean, pretty clearly. You know, this is the you know, right. Absolutely. You are. Absolutely. But you’re you’re sort of the way you described it. And if we go back to that twenty twenty one moment where you’re like, what if Bazel doesn’t win? And sort of I’m quite interested in I mean, that’s a really interesting position, but also This question about you don’t want to just be the basal company. You want to be the build management at AI scale company, right?

Helen Altshuler (35:05)
We are, we absolutely are at this point. And so we’re running that level of scale for major stock exchange with CMake, for major gaming company with Buck2, for most of the browsers out there with Chromium build system. we are definitely the company that owns build management at scale. And interestingly enough, Bazel is having its moment

for the first time, like since it was open sourced in like with much higher number of people, not just adopting Bazel, but also a sign of like job recs. can see every single company at this point that has large code bases is looking for people with Bazel expertise. So we see in the future is converging towards using Bazel for its core features.

We also seen these other build systems that I mentioned being essential for the audiences that they serve. So Chromium is very much in Chromium build space. And gaming is certainly adopting Buck 2. And definitely, Matter is having its moment with Buck 2 because of scale as well.

James Governor (36:22)
Yeah, well, I mean, you know, the bet on parallelization is generally a good bet.

And So!

Let me ask you that question about GitHub then. I think that when I look around, there are people sort of looking at actions, trying to work out what their strategy should be. mean, how do you embrace the GitHub customer?

Helen Altshuler (36:51)
GitHub customers, first of all, they come with a number of other tools that they use. And so as I mentioned, code is step one, reviews step two, then there’s build and CI. And so we see this as a challenge. Most of our customers are pushing GitHub to its limits. And at this point, really the solution is

parallelization of the build workflows, parallelization of the test workflows. So we natively integrate with GitHub actions, for example, making them more scalable and more parallelizable through the use of our CI runner infrastructure. This is a fairly busy space as well. There’s lots of companies now trying to plug the gaps that GitHub has created until GitHub itself fixes them. But more importantly, I would say it’s really the build

metadata that we are the experts in. So to be the leader in build management at scale for us, it’s combination of our team’s expertise and the data that we accumulated in our platform to make smart decisions around how to restructure dependencies, how to squeeze more from compute, how to parallelize data. And so to ask for any company that’s starting with their GitHub infrastructure,

looking at builds and looking at their CI, the next logical step for us is to work through the GitHub actions, self-hosted runners that we’re able to provide or manage. And ultimately, we’ve actually even seen the benefit of having this kind of forward deployed engineer impact where we started helping our customers to move more workload from

GitHub actions to core remote execution. And by doing that, they’re saving a ton on compute, but they’re also getting much faster performance improvements. And so we’ll also, be publishing the blog that we’ve done with companies like Apex and Iterable and their combination of CI runner system plus remote execution and really understanding what

can be parallelized in the code base and using agents to parallelize those things more effectively gets the company’s ultimate kind of performance and speed and scale and makes them really ready to absorb the scale that’s coming at them. And the example that I brought up in the beginning with Plaid absorbing three times more without any changes to their cost is really because

there’s so much parallelism in their system and we work together to implement more parallelism as we go. And I would also say another kind of data points to give you back to the anthropic example that I’ve given. It’s not surprise that AI companies are predominantly convergent and basal. And so,

OpenAI is investing heavily. There’s a lot of open source efforts with Codex driven by some of the prominent community leaders like Zbarsky and several others that are openly in the community or showing the impact of running this on Bazel. We’ve seen that traditional build systems are not scaling and we knew that.

that there’s like about a million lines of code in your monorepo when you really need to start looking at different solution for build management at scale. And so every company is getting closer to that metric than before. But also we’ve seen that AI companies, even AI companies, I would say they started not with Bazel and they are converging on Bazel and they’re migrating now that they have doubled or tripled their code base within short amount of time.

And so what I wanted to of like kind of leave the folks with who are watching this is how do you know that you need to take a different look at the build infrastructure, at the build management of scale? It’s first of all, you’re using variety of build systems for different languages. Like that’s a sign that you have complexity in your system. The other one is that

for languages that do not require a build system, interpreted languages like Python or JavaScript, you’re not using build system, but you are generating so many tests that you’re not sure how those tests will be operated together and how to parallelize them. And what we see happens at that point is that companies are starting to turn their CI system into these manual shards.

so that they can create parallelism by doing these shards of tests instead of letting the build system like Bazel take care of all of that for you.

James Governor (42:00)
development, you go. Because I have to say, summing up, the thing that has really surprised me recently, and you spoke about, know, development, verification, deployment, where build sits in that.

The thing that I have been… really surprised by is the rapidity with which organizations, even in regulated industries, are saying things like, we’re not going to do manual code review anymore, at least not for everything we’re building. And that is definitely, even a year ago, I would not have expected

I wouldn’t have expected going and be talking to a bank in the UK in the last few weeks, a manufacturing company, bank in the UK, a gaming company, where they’re like, actually, and one of these organizations in particular, and I don’t think I should name them, they’ve started saying, we don’t call them, it’s not about the branches, it’s about the leaves. Branches, we will…

do some human verification. But for a lot of what we do, they’re just leaves. And I think that if you think about the, know, cattle versus pets that we talked about with infrastructure, we’re seeing applications and they’re just much more disposable now. And that of course accelerates how much software is being built. Because if you’ve got a good process where you can do a rollback, if there is a problem, then you’ve got a much more much higher tolerance for risk and yeah you know as I say when you have this conversation with the bank it was it was really surprising

Helen Altshuler (44:04)
I heard the same conversation two weeks ago when I talked with some of our ride sharing and autonomous driving customers that they’re looking at risk-based pull request reviews. figuring out what goes automatically and what’s not. And as I mentioned in the beginning of this conversation, pull requests are the current lever that once you pull it, the throttling on your build infrastructure

will not be in place and companies that have not invested in build management at scale will suddenly start facing exactly the same issues that GitHub is facing when code bases blew up. And so this currently is a really cool moment to observe in our industry is that pull requests are still throttling how many builds are happening. And yet most of our customers have seen from two to

four times an increase in the number of builds that are coming happening on a weekly basis. And so, but once that lever is pulled, because you now doing a lot more automatic reviews, build infrastructure will show its cracks. And the test for, know, are you gonna have cracks or not is predominantly, are you using Bazel and are you using

James Governor (45:13)
Yes.

Helen Altshuler (45:26)
remote execution and caching system that Bazel is natively providing and with all of the acceleration that we can offer on top of it.

James Governor (45:37)
Okay, well, I think you’ve been even more bullish about Bazel than I had expected. But I did, you know, I was pleased when I this conversation. What’s important, as I say, as we sum up, is this, yeah, like, and it’s-

Again, every day in the AI era, of reassess your prior’s day. again, one year ago, I was like, well, actually the agent pull request, that makes sense, right? You have a human, get the agent to do some work, makes a pull request, look at the code, that’s just nothing like how we’re actually using agents. That’s just completely…

That’s broken down. I think that that fundamental primitive in terms of workflows that has propelled the industry for, you know, the last, well, 12, 15 years or so. Yeah, fundamentally breaking down. So, yeah, new laws of physics, basically.

Helen Altshuler (46:51)
Yeah, essentially what’s changed really isn’t Bazel, it’s everyone else. And the pressures that once defined a few large code bases, like the one at Meta, Google, Twitter, are now showing up everywhere. And languages, like there’s more languages in a code base. And all of this is putting CI pipelines under pressure. And I’m actually looking at this on a weekly basis with our customers and the more

the more investment that they’re able to make in understanding their own code bases and restructuring and using agent workflows to restructure their own dependencies, the better they will be ready for continued increase that is happening. It’s not even an increase in the code growth, it’s increase in velocity is that there’s more changes and more changes means that more changes need to be reviewed and built. And I do hope that with AI code bases are growing, hopefully there’s AI is also good at deleting code as much as it is creating code as

James Governor (47:57)
Well, we’re going to have to see a lot of that because as I say, there’s so much of it is fungible. Yeah, we’re to need to throw away an awful lot of code. Otherwise things are going to get very, very expensive. And on that happy note, yeah, that’s been amazing, Helen. I really appreciate your time. Thanks for joining us on the show. Like I said, I really enjoy a good sort of lesson from history.

you know, we recently, had a scary engine in neon from, Google’s cloud run team learned a bit about the foundations of that product. You know, I always think it’s interesting. Yeah. Of course things are interesting that happened today and over the last 18 months.

Helen Altshuler (48:44)
And of course, the fun fact is that Steren was my colleague working on the Bazel project in 2017.

James Governor (48:50)
There you go. this is where… Go to the RedMonk podcast if you want to hear these interesting, interesting stories about Google technology. Yes, Steren and Helen working on that project. So yeah, what I would say, everyone, I hope you’ve enjoyed the show. The classic things, please do share it if you think anyone’s hitting some build limits and they might need some help. But yeah, share it, like it, subscribe, do all of those good things.

Helen Altshuler (48:52)
goes a little closer.

James Governor (49:20)
But yeah, really Helen, thanks so much for joining us.

Helen Altshuler (49:25)
Sounds great. And join us on the Bazel world tour where we do bring together people that care about it. And we publishing our events at meetup.build for that as well.

James Governor (49:36)
Okay, so there you go everyone, that’s your call to action. Thanks very much Helen and thanks all for joining us.

Helen Altshuler (49:42)
Thank you.