In this MonkCast conversation, RedMonk’s James Governor talks with Paul Brookes, a senior AI engineer at TurinTech, about making AI inference faster and cheaper. TurinTech predates the generative AI boom, having spent years optimizing complex code with genetic algorithms, and now points those tools at the models themselves. Brookes walks through techniques like kernel fusion and model compilation that squeeze more tokens per second out of specific hardware, drawing on the company’s work with Intel on OpenVINO and vLLM. The two get into running capable open models such as Qwen on local machines, the spiraling cost of AI, and why judging engineers by tokens burned misses the point. Brookes also describes Artemis and its discovery harness, which lets agents learn from past results, and traces his own route from quantum physics into low-level performance engineering.
This RedMonk conversation is sponsored by TurinTech AI.
Links
Transcript
James Governor (00:04)
Hey, it’s James Governor, co-founder of RedMonk, and we’re here for another MonkCast talking about what else would we talk about in 2026, but of course, AI. I’m here today with Paul Brookes, a senior AI engineer at a company called TurinTech. TurinTech is it’s kind of a bit of an AI OG. It’s been around in in the UK scene, pre-LLM at least. You weren’t launched 18 months ago. You’re a company with a bit of history. So why don’t you tell us a bit about sort of of TurinTech and what you well I don’t tell us a bit about TurinTech.
Paul Brookes (00:40)
Sure. Well, indeed we have been around for a few years, in fact before generative AI, and we’ve been focused the whole time on really complex optimization problems. So whatever business that may be in, maybe it’s financial, maybe it’s logistics, we have a platform to deal with that.
And in particular, we have a platform for performance optimization of code, which is focused very much on applying genetic algorithms for those optimization tasks. And of course, before generative AI, you could do that with all sorts of techniques like pattern matching and recognition and so on. But basically, since the onset of LLMs, we have a huge new array of tools that we can use to optimize your source code.
James Governor (01:21)
Okay, and you have, as all companies have in the software industry, had to respond to this onslaught of code from LLMs, to the change in working practices. So tell me a bit about, or in fact, even the use of of models, because I understand that you know you’re kind of a bit of an inferencing nerd. Is that
Paul Brookes (01:42)
That that is fair to say. That is fair to say. So, you know, obviously everyone has this huge, huge desire for tokens and you know we’re we’re not like unlike anyone else, just you know, we we also are heavy consumers of AI. But
Being a performance optimization company, we do get to use our tools to actually look inside those models and figure out what exactly is it that we can tune up and boost in order to get the most tokens per second and the best cost basically out of all the all the models that we’re using.
James Governor (02:17)
And on the inferencing side, I mean, so you can you you know that’s your own models, but what about what what about happening what what’s happening sort of in in customers? Because I I think at the moment we’re we’re we’re optimizing inference is quite new as a as a as a field, certainly to you know, if we think about your customers, it might be financial services or telcos or something like that. Yeah, the state of the art and inferencing is it’s it’s early days.
Paul Brookes (02:40)
Yeah. Isn’t it? Well, I mean that there is a massive market out there for it. And in fact, we’re you know we’re very pleased to be working with some leading semiconductor companies on that. In particular, we get to work with Intel on their OpenVINO inferencing framework, their toolkit, and also we get to optimize vLLM for Intel hardware.
It’s it’s pretty in-depth. I mean it really goes all the way down the stack, you know, right from the end user, you know, how are they conceiving the model, all the way down to actually what are the particular instructions that are best for this specific piece of hardware to tune the model for for that particular chip. So
James Governor (03:21)
So semiconductor companies are are your sort of they’re your canaries. I mean canaries in the coal mine. They’re the early customers. You’re expecting other organizations are gonna be using their own models and optimizing themselves.
Paul Brookes (03:36)
Absolutely. I mean, foundationally, obviously the semiconductor companies want to make sure that people are getting good experience on their hardware, that they’re able to get inference on their hardware, you know, at the at the kind of optimum level. And so, you know, they sit at the bottom of the stack there. But everyone else really wants to get in on this as well because they are concerned about cost and security. They want to deploy their own models.
They may be a bit more hardware agnostic, but they may have other kind of metrics that they’re trying to optimize for. And they really want to get the best out of their particular use case on whatever hardware it is.
James Governor (04:18)
And and so you mentioned those examples with Intel. What are some of the the you know let let’s let’s drill into that, of where you are, are there are there any I I’m always examples. Like yeah, sort of w what what is it that you most enjoyed working on or some of the interesting challenges?
Paul Brookes (04:38)
Well, you know, I I think that that when you when you’re carrying out optimization on on LMs, you as I mentioned, you do go all the way down into the stack and you get to see a little bit about how those models are kind of represented inside those those inference engines, right? And how do you optimize? Well, there’s so many different ways. one way is through kind of model compilation, where you know you you kind of search through this huge graph of the model in order to figure out which operations you can fuse together to make the the
best use of your of your of your memory bandwidth or reduce you know something called kernel dispatch overhead which is a very very nerdy topic but also right well for for example you know obviously many people
When they’re running models, when they’re running models, they are running them on GPUs. And if you are running on, say, a laptop with a CPU and an integrated GPU, then when you run inference of the model, as you move through the model, every single operation will at some point be dispatched onto that hardware. So from the host sort of CPU of the laptop to the device, which is that integrated GPU. The thing is,
There’s a lot of bookkeeping when you do that, right? So every single thing has to be sent off, you know, almost like I don’t know, I mean, wha what’s the best analogy? You might think of well, I don’t know, maybe it maybe it’s like sending a parcel or something like that. Who knows? Anyway, you
You basically have to send off all these kernels onto that hardware. And if you’re running on a workload where you’re having like a single conversation, actually there’s not too much work to dispatch on each token of that conversation. But if you’re trying to get the very, very best performance for a single user, you know, on their local device, then this is a big problem. You’re completely underutilizing your GPU. So what you can do really, and what is is is widely done is is kernel fusion, which is Fusing together the kind of the operations that are being dispatched onto that onto that GPU into bigger and bigger jobs. So
All that being said, you know, when you have a framework that has some kind of compilation layer layer that that kind of scans through your your graph and figures out what are the best things to fuse, you can basically do a lot of different heuristics to kind of tune exactly which things to fuse up and how exactly they’re going to be put together into these bigger packages of jobs that are dispatched onto the device, which is perhaps a bit of a mouthful, but you know, it’s way down there, and and I think it’s never been a better time to to know about how these things really work.
James Governor (07:25)
So did you I mean I I I don’t know how closely you follow well I mean partly it’s it’s it It it’s partly the the Microsoft world, so I’m not sure but but so NVIDIA’s Spark RTX chips were announced last week. they’re basically I don’t I don’t know whether this is so I’m yeah, I just given you’re nerding out about hardware and possibly running models locally, there’s there’s gonna have to be a lot of work in in that space. Basically, NVIDIA is doing something i that that’s a bit closer to what we’ve seen in terms of of Apple Silicon. Yeah. So
load of ARM cores, GPUs, 128 gig of unified memory. Right. A an architecture that’s a lot more like what we’ve seen, as I say, in in modern Apple laptops. And so I think for them, obviously, and and for Microsoft, well I it’s two there’s two things. A, it’s gonna make games f really awesome. Yeah. as a gaming platform it’s gonna be very sweet. But obviously for AI workloads, I think the hope from Microsoft
And you know, this is definitely something that Apple’s gonna benefit from as well as as we begin thinking, hang on a minute, do do we even need to be running or could we be running more of our models locally?
Paul Brookes (08:46)
Yeah, I mean I think I think it is a very important thing to be thinking about because costs really are pretty much exploding. I mean and y you see that in a lot of a lot of reports around. I mean I think COO of of Uber was was pretty was was pretty shocked by the the cost of AI recently and I think they’ve they’ve burned through their entire, you know, yearly AI budget in about four months.
James Governor (09:10)
be fair, Uber as an organization, I thought it was interesting looking at this because of course they have a history of burning through money. And and you know, you have that near-death experience of those those sort of large, extremely tech-savvy organizations. It was no surprise to me that Uber was the one that went, hang on a minute, yeah. We are a company that has to be about managing cost, and tokens is one of the costs to manage.
Paul Brookes (09:37)
I mean AI is is so useful in so many jobs, but you’re not gonna necessarily solve all your problems by just, you know, setting fire to piles of money. Right. So so you know, I think it really is important to kind of
James Governor (09:48)
I’ve been saying that the the real bitter lesson is your anthropic bill at the end of the month.
Paul Brookes (09:53)
Yeah, you know, it’s you you can have a wild ride and and of course, you know, you can prototype all kinds of things and you can learn so much. Yep. but I think it is very, very important to to direct that towards some kind of real evaluatable, kind of validated criteria, right? So whether that’s performance optimization, you have benchmarks, some harness that’s telling you if you’re getting the most out of your AI, or or well, some some other metric, some other kind of business KPI.
And I think, but I think, you know, what we’re also seeing is that basically the the quality of the hardware that’s coming out, and also the quality of the the open source models, whether that’s Kimi K2 or Qwen, the latest Qwen versions, it’s it’s pretty much reaching that point where you can actually start using local coding agents on this hardware that are really delivering value to you.
James Governor (10:47)
Yeah, a hundred percent. I mean the amount of of well, and the amount of of investment that’s gone in, but the quality of, you know, certainly Google’s Gemma models, even running on on like phones and stuff. Yeah. again, I mentioned Microsoft, they announced a whole new suite of local models called Aion last week. they already had some small models. I do think the small model thing is I mean it’s funny, what is small?
Do you know what I mean? It’s like, 30 billion parameters is a small model now. So but but yeah, what do you you know, as a nerd, are you like what do you run locally? What what what are you found interesting in terms of some of the open weights models that you’ve just mentioned?
Paul Brookes (11:29)
Yeah, I mean locally we run Qwen 3.6, I think it’s 35 billion with three billion active parameters. Okay, so this is a mixture of experts model. So it really does pack a pretty good punch without having to, you know, use all parameters in the network simultaneously. and what you’re seeing is that this is able now to perform like more complex tasks, you know, algorithmic challenges or or perhaps tuning of the LLM itself, as I spoke about, you know, when you’re when you’re looking at the
the kernels or or or or the or the compilation passes. I think it’s a very, very effective model, but of course it’s just one of many. I mean Nemotron is there and and Kimi K2 2.5 I think is is also a a very powerful one. But that ends up being actually not a really small model at all because I think that’s almost that’s several hundred billion parameters. So not something that you’ll you’ll run locally. But yeah I think certainly certainly out of the the Qwen models we we get a lot of good performance.
James Governor (12:28)
you I mean what is your what is your infrastructure I mean what does your infrastructure for testing look like? So you’ve obviously got you must have I mean an absurd I mean yeah what what do you yeah how do you how do you test at at TurinTech?
Paul Brookes (12:41)
Sure, well, okay, so you know, we do have access to some of our own hardware. You know, we have servers with with various NVIDIA GPUs or Intel CPUs. but we’re also out there, you know, using using GCP instances.
We have basically Intel Granite Rapids bare metal hardware on which we can we can actually profile these models that we’re running. So I think actually that’s one of the really important points is that is that if you are doing this kind of performance tuning, you need to not only be able to access just some some cloud instance, but you actually need to be able to access it without having some kind of hypervisor in the way that’s blocking you from really working with with the hardware counters on the hardware itself. Because you want to bring to bear all the
Profiling tools possible that you can then kind of aggregate this information for your for your coding agents to perform that tuning. Okay. And so that’s one side of the infrastructure, is really the hardware itself. But on the other side, we have our platform, our own performance tuning platform.
which is Artemis, which basically combines together kind of a knowledge graph of of past results so you can learn from the past and and perform some iterative self-improvement. And we use that to really, you know, power the the dispatch of of coding agents across these code bases in order to find the optimal tuning. And then on top of that, of course you have the benchmarks that really sift the signal from the noise and figure out what it is that should be presented to the user for For for review. Mm-hmm. So it’s a combination there from hardware to software.
James Governor (14:13)
So can we go back to Intel a little bit? Yeah. so you mentioned two projects that you the you’ve worked on with Intel. Like to what level how what does that partnership look like? And you know, what does it feel like when you get a breakthrough that like you where where you’re seeing a significant performance improvement? So I’d love to sort of drill into a bit about maybe, you know, some of the techniques that you that are showing value in terms of the optimizations you’ve done at Intel.
Paul Brookes (14:47)
Yeah, I mean I think the partnership, it feels it feels very, very good to be working with basically some of the you know the world’s best semiconductor companies, right? But all the world’s best hardware companies. And it’s really great validation when you when you reach a point that that you’re you’re you’re able to surf something interesting.
Of course, it can take a lot of work. And I don’t think AI can ever completely, you know, replace that that kind of kind of human expertise of of knowing exactly the system and how it works really, really well. But I think in partnership with a you know a bit of a bit of elbow grease, a bit of validation and a bit of a bit of you know iterative improvement, you can really make some breakthroughs in performance tuning. And I mean when you see them.
You always have to like take a bit of a step back because there are I think there are a lot of a lot of a lot of times when you know random fluctuations or or noise will will will make a particular thing appear better than it is. But if you take a deep breath, you know, and you measure and you re-measure and you A B test, then you can, you know, then you can start to celebrate. So I’d say one of the biggest things I I’ve learned there is really to to not celebrate too early. Just to just take a breath. Yeah. Work through it. Yeah. Yeah. But take another look and then and then you’ll see. Okay. And a lot of the time you’ll get good results.
James Governor (16:11)
So in terms of technique and and just you know sharing a bit of your knowledge, like what are what what are some of the the the key areas that like if I was you know if I was working in an organization and I knew that we were gonna be doing more more work around optimizing inference, like what What should I be focusing on, learning? What tool I mean, other than Artemis occur of course, what what sort of tool sets and approaches should I be learning and focusing on?
Paul Brookes (16:54)
I mean, I I think the most important thing is to have some clear kind of value criteria for what you’re trying to optimize. And that actually I think is very tough. It’s not like everyone approaches a problem and they’ve immediately gotten the most sophisticated benchmark and unit testing suite that tells them exactly whether they solved their problem. That can come down, you know, a little further down the line. But I think really setting setting your benchmarks close to what you’re really trying.
Trying to solve. is very important. You know, you wanna you don’t want to develop benchmarks that are only only there because they’re easy, because you have all the necessary data. You might need to go out and collect some data, collect some examples. And that’s where the work comes in is to actually figure out what your target is. That’s what I would say really about about the high level. And then if you’re a you know if if if you want to you know nerd out then and then you learn about exactly what’s going on in the kind of different sorts of performance bottlenecks you can have on that piece. of hardware if you want. But yeah I think connecting the whole stack together is is quite important.
James Governor (18:02)
Yeah. So like I guess I’d be what’s your what’s what was your journey to being someone that actually worked at that low level? Like sort of yeah, so I talked to one of your colleagues, Mina, she’s her back room’s neuroscience. Yeah. like how did you what was your route into to this this world?
Paul Brookes (18:27)
Right, well, you know, I I actually have a background in physics. Yep. So I guess I guess I’ve always had you know an interest in the in the fundamental. but you know, I did take some kind of detours from that. So, you know
A few years ago I finished my my PhD, which was on topics of quantum computation and superconducting circuits. From that, I actually moved into data science. So at a much higher level, you know, dealing with software and data and trying to solve, you know, much more, much more sort of much more higher level problems, you know, whether you would express essentially as as business problems. But I think I always had that kind of pull to delve a little bit deeper again. And and I think, you know, this is like really the perfect opportunity for that. Yep.
James Governor (19:16)
So working with the semiconductor companies is a lot of fun.
Paul Brookes (19:19)
Yeah,
basic basically that’s it. Okay. No. Okay. That’s the draw.
James Governor (19:22)
So AI engineering, so I mean again, that’s another reasonably new discipline. So you’re you’re a a senior AI engineer at the company. Tell me a bit about like w what what what is what what do you think an AI engineer is and what does your team do?
Paul Brookes (19:39)
Well yeah, right. AI engineer, obviously it’s it’s a fairly fairly new term. and yeah, I think best described by by what those people people do. But the problem is that actually it can be a wide variety of things. It can be pretty, pretty cross-functional. So of course, you know, there’s ML engineering, which can cover, you know, the the building and and deployment at scale of of of
ML solutions, ML models. that’s the one side of it. But then there’s also now you know, now that we’re empowered with with different AI coding tools, you actually get to deliver, you know, much more across the stack. So you can even you can demo your projects much more easily. You can basically show off what your kind of prototype solution is. and you know you can take that all the way to the customer, almost like a solutions engineer or forward deployed engineer. So I think one of the things really is that basically now that we have access to so much tooling, we can you know bring together
a lot of different technical and commercial and kind of customer-facing elements into that overall role of AI engineer. Although some people might be you know much more specialized within that.
James Governor (20:54)
Do you think I mean, how important do you think that FDE, the front forward deployed engineer, will be in terms of like your work going forward and your ability to engage, engage with customers? I mean, how much domain experience do they have? They sort of know they want to improve something. Yeah. But but they do you follow in a lot of your customers like, yeah, we love the software, but could you just send us a bunch of consultants? Like how how does what’s the balance? How do you sort of manage that challenge?
Paul Brookes (21:21)
I mean that certainly is a challenge. And the thing is that, you know, when you want to do, when you want to to provide a product that does performance engineering, it’s it is a tool, but it you know it’s only one half of the equation, and you need to have a lot of expertise in order to use it correctly. So I think you do need some some work there to kind of bridge that gap. Explain, you know, how is this delivering value to you? And really I would phrase that in terms of you need to basically
To demonstrate some hook of this is one of the small things that can be done with this tool that adds value to your workflow and now hand it over for you to take it forward. So you need to really align those two things and kind of bridge that gap. But I think I think it’s certainly like a quite a fun place to be and lets you learn about all sorts of new problems.
James Governor (22:16)
Yeah, a hundred percent. Well, we love that. solving problems, lots of optimization. I say, your colleague did a nice sort of judo move and token maxxing is now not burning as many tokens as possible, but using them more efficiently. So it should be, shouldn’t it? Yeah. I I certainly think that measuring people’s performance by how many tokens they burn is kind of daft. One of the silliest things I’ve heard.
Paul Brookes (22:29)
Right.
Yeah,
I would I would not advise this in general. but if you you know, if you have a harness that tells you if you spend your tokens well and if you’re learning from what they’ve done via, you know, basically you know a knowledge graph of of your past results, then I think, you know, burn bar away. It’s as long as it’s in helping you, you know, achieve your goal.
James Governor (23:07)
tell me a bit about the harness question. So you know, is are you building a harness now? Is that would that be a a a reasonable way of of talking about what you’re building or do you you more integrating with with harnesses?
Paul Brookes (23:21)
Well, I think so. We have the discovery feature inside of our platform. The discovery feature is essentially a way of tying together agents with the tools that they need to record and learn from past results and benchmark what the performance was. So in that way, I really see that as as a harness. But not just a harness, it’s really something that kind of it doesn’t just keep you on the rails, it really
helps you improve over time. And I think that applies not only to the the the the model itself that’s performing those changes, performing those code edits, but the user who can now look through a review of all the experiments that were done, all the ideas and hypotheses that were were were evaluated and see what worked well and and what didn’t. And I think that really that’s that’s probably the best way of using AI is something that can kind of feed back into you as a as a user and that you can then learn from as you go forward.
James Governor (24:18)
Okay. Well yeah, that’s a little bit of nerdery around optimization for model inferencing. This is Paul, James, this was another MonkCast, RedMonk conversation. you know, if you enjoyed this, want more content like this, please smash like, hit subscribe, share with your friends, all that good stuff. but yeah, I’d just like to say Paul, thanks very much for joining us today.
Paul Brookes (24:46)
Thank you
