Most conversations about AI agents stay on what agents can do. Less attention goes to what they leave behind: logs, traces, and tool-call chains at a volume nobody’s pipeline was built for. Nikhil Mungel, Head of AI R&D at Cribl, talks with Kate Holterhoff about why that is becoming observability’s cost and noise problem. They discuss why token prices keep falling while total spend climbs, how employee spend differs from the tokens a product burns serving customers, credit-based versus outcome-based billing, why users always reach for the priciest model, and how the Open Cybersecurity Schema Framework (OCSF) standardizes what agents emit. Underneath all of it sits the question the industry keeps dodging, which is whether to keep scaling the backend or get stricter about what gets collected at all.
This RedMonk conversation is sponsored by Cribl.
Links
Transcript
Kate Holterhoff (00:04)
Hello and welcome to this RedMonk Conversation. My name is Kate Holterhoff. I’m a senior analyst at RedMonk, and with me today is Nikhil Mungel, the head of AI R&D at Cribl. Nikhil, thank you so much for joining me here on the MonkCast.
Nikhil Mungel (00:17)
Excited to be here. Thanks for having me, Kate.
Kate Holterhoff (00:18)
Awesome. All right. So let’s just start off with some introductions here. Tell me about yourself and what you do with Cribl.
Nikhil Mungel (00:25)
Sure. So at Cribl I have the very fun job of leading all AI research and development. So I I lead engineering management as well as product management. when it comes to AI research at Cribl. So I have a few teams that I support which are building agentic products that simplify the lives of IT practitioners and security practitioners. And then I also lead a research lab team where we
train our own models that can work on really high throughput streaming telemetry data. I that’s what Triple does as a company. that’s what we specialize in. So yeah, we’d love to, share more about what we do and, you know, where where the industry is going with AI as it applies to telemetry and observability and security use cases today.
Kate Holterhoff (01:13)
Okay, so we are definitely gonna be digging into that. I I I figured we kind of start this conversation around some of the things that you’ve written. So I am, specifically excited to have someone who’s an expert on like FinOps in agent space on the show because I mean I I I think token spend we can all agree is top of mind. It’s a become a real pain point for a lot of folks. and you’ve written about this, in the in the context of the first, agent that your team has shipped.
So can you walk us through what happened there and and what you actually saw in terms of that sort of larger question for our conversation of of telemetry?
Nikhil Mungel (01:46)
Sure, yeah. So it’s I mean, yes, tokens and token costs are a pretty hot button topic, you know, right now in August of twenty twenty six. it’s token costs also like, you know, keep dropping compared to a few years ago. But then token use is going up quite a bit. I mean, models today and use cases that are powered by, you know, LLMs today are much more complex. They’re much more longer running, and they tend to consume a lot more tokens than
Some of the some of the use cases that we used to have, you know, back in 2022 and 2023. So as an aggregate, companies are spending a lot more on tokens today than they were a couple of years ago. and I think it’s pretty valuable. I mean, I feel like tokens are adding tangible, tangible benefits. we, you know, at at Cribl, we lean a lot on so there are there are two ways in which we where we use agents. One is
baking agents into the product itself that helps our customers and our users a a much more smoother experience when it’s using the product and when it’s you know taking taking the value out of wh whatever Cribl deployment is configured to do. And then on the other side we also use agentic software development and agentic research for the stuff that we do internally as well. So when we are training new models, we have a research agent that helps us
And when we are writing code, obviously we use agentic coding tools like Claude Code and Cursor and OpenAI Codex as well. So I would love to would love to share a a personal experience where we had this runaway agent that accrued a massive token bill. and I’m pretty sure a lot of listeners are gonna have to. they’re they’re they’re they’ve they’ve been in similar situations, I feel like.
Kate Holterhoff (03:37)
Yeah, let’s
let’s hear about it.
Nikhil Mungel (03:39)
Yeah, yeah. So one of one of one of the teams that I support is our AI research team where we, you know, train we have a little GPU setup here in San Francisco where I’m talking to you today from, and we run experimentation and we run research experimentation where we train new models that can operate on high throughput telemetry data. Now these models are a little different from LLMs, since LLMs can or you know, AI models, multimodal AI models can intelligently
Come up with answers to the questions you’re asking, et cetera. In the world of telemetry, it’s a little bit different because you’re dealing with a lot of really high throughput streaming logs. So you have terabytes of logs that are being processed by Cribl every single hour for every customer. Sometimes it’s tens of terabytes every single hour. So using an LLM like Claude or, you know, using a model like Claude or using a model like GPT five-six is not really feasible for those sort of
volumes of data that we deal with. So we design our own models to do that. And we have a research agent that runs in a loop overnight. So we run, you know, we ask it to run several experiments and then you know it’s able to go and it’s able to run these models, you know, run run based on an objective function and you know gauge the loss overnight. And then in the morning when the researchers come in, they’re able to see the progress that was made overnight. So
The objective function of an agent is really, or when it’s running in a loop, is a goal that you give it, and then it optimizes for that goal and you know uses that goal to really train one of these models that we have. So a few months ago or a couple of months ago, we had this agent that was going off and you know training a model, and its objective function was to b to keep thinking more, right? I mean the way somebody had written that experiment was it’s like the more you think
you know, the better the output is gonna be. So the agent just every call it made to an LLM, was like, cool, like this is this is great. You know, I’m gonna do more of this. And in the morning when the researcher logged on, we found out that it had accrued an enormous token bill that I’m not gonna share the exact number, but it was quite a bit. And that was that was a big shock. And ever since then we’ve been a little bit more thoughtful about, you know
how we structure these experiments and also put in some guardrails in terms of how how many tokens a single experiment can consume before it raises or you know taps a human researcher on the shoulder and asks it whether it should continue.
Kate Holterhoff (06:19)
Wow. Okay. Such a good example of why guardrails are becoming more and more important today. all
Nikhil Mungel (06:25)
Yeah.
Kate Holterhoff (06:25)
right. And so, you know, y you’re this is just one example I suspect of many. I I’d love to hear, some of the trends that you’re seeing around this, I guess. So, like, okay, are companies that you’re speaking with, using AI and agents to address observability in in practice? You know, like how how is it that AI has shifted the way that you’ve approached this more recently in in our sort of agentic present
Nikhil Mungel (06:54)
Yeah, I think so the way I think about this is in two parts. First, there is a lot more software today than there was like two years ago, right? Because of agentic coding and because how approachable and easy Agentic coding tools have made for teams to produce software. Teams have been producing a lot more software, you know, in the last in the last few years than they had historically done. I mean a a team of five can now build extremely complex systems, whereas
before before agentic coding took off, it would probably take you, you know, 50 engineers or 50 developers to do the same thing. And because of that, because of the growth in the amount of software that has been put out there, the operational footprint has also grown sufficiently complex. So in a way, you don’t have an you don’t have an option but to rely on agentic SRE tools, agentic observability tools.
to really understand you know how your application and how your software is behaving in production. So one one one example that comes to mind is how so if you if you have any software system that has agents within it to you know deliver user functionality and user value, these agents are running many times in a in in in a window of time compared to a human operator or a human SRE would have done.
And they’re producing a lot more, they’re producing a lot more data. They’re hitting the microservices or any any any server-side compute that you may have a lot more. And it’s just creating a massive increase in telemetry that we have been seeing across the industry. And in order to make sense of all this telemetry, you need to ingest it, you need to store it, you know, you need to do all the best practices that you have have been doing so far. And in order to reduce the burden on your operational teams, your SRE teams, your IT ops teams, and your DevOps team.
DevSecOps teams, you have to lean on agentic tools to, you know, take away some of that burden from the other side of the house as well.
Kate Holterhoff (09:03)
yeah, we’ve been tracking the amount of code that has been published recently and it’s staggering. okay. so you you’ve also written a little bit about agentic saas, specifically and arguing around cost being a reliability metric. So we’ve seen more attention on token spend. I’m I’m thinking about the Linux Foundation’s, They just launched their tokenomics foundation, for instance. I am not hearing panic yet, but from your perspective, is this becoming a bigger deal for folks like CFOs?
Nikhil Mungel (09:37)
Yeah, I I think I think it is. And if you if you again split split split token expenses between what your internal employee workforce is using to drive productivity, and you the first thing you want to do is separate that out from the token spend that your SaaS is driving to deliver value to your customers, right? So if you take an average software engineering company or a SaaS company, you have the SaaS that it produces, which is sold to customers and you it’s a
usually a high margin business. And then you also have developers and product managers and employees and staff who are using tools like Claude Code to deliver and build that SaaS. There’s been a lot of focus on like the internal token usage right now because that’s that’s something that you have there’s typically no bound on it or you know there’s a very high bound on how many tokens a an individual employee can use and you know to deliver value. So that’s that’s one that’s one part of the equation that we can get into.
but the second part is your COGS or you know cost of goods sold that your SaaS is accumulating when you’re delivering value to customers. What’s one one unique thing that you know we have, maybe it’s not even that unique, but one thing that we have seen happen in the last couple of years is customers have demanded that vendors support agentic flows within their software. Nobody wants to use
Software 1.0 where you have to manually configure, the software and like go through like six or seven different configuration screens and like look at docs and stuff like that, right? Everyone wants an easy button to set up whatever it is that they’re doing, right? Maybe it’s creating a ticket on Jira, maybe it’s, you know, setting up an observability pipeline in Cribl whatever it is that you want to do, everyone wants an easy button and agentic flow. Have are you are you have you have you seen that as well?
Kate Holterhoff (11:25)
I it’s funny, I was just talking about ClickOps today and and sort of how it is that agents can best use these infrastructure tools. is it that they are it’s just through like a Terraform something, or is is it that everything’s gonna be through API keys now? Or you know, I think it was like nine months ago everybody was talking about like, I need an agent to buy me a plane ticket. I’m gonna give it access to my browser.
So I I do feel like there is this, it’s kind of an open question right
Nikhil Mungel (11:56)
Right.
Kate Holterhoff (11:56)
now, but I I absolutely am am seeing it as like, something that we’re we’re grappling with. I I I guess I just I
Nikhil Mungel (12:01)
Yeah.
Kate Holterhoff (12:02)
am not sure where it’s landing at this point.
Nikhil Mungel (12:05)
Hundred percent, right? And when you when you really try to unbundle, you know, how much tokens your SaaS is consuming to deliver value to your customers, that’s when things start getting really really tricky, right? And that’s the stuff that CFOs care a lot about. It’s like, hey, are you running a high margin business or are you running a low margin business? Really that’s where it comes down to. And when you when you look at when you look at tools out in the or SaaS products or you know, just any products out there in the market today, I’ve seen like two two
types of billing models that seem to have been catching on. One is some sort of a credit-based model. So if you look at something like Notion, you know, you purchase AI credits that then you can use for generating, you know, documents, summarizing documents. So every agentic flow that you use within the product consumes some sort of credits. And that’s one way to pass on some of those costs back to your customers and not end up in a low margin business.
And then on the second hand, I’ve also seen tools offering a more outcome based billing model. So, you know, if you if you have an agentic SRE tool, you may charge customers based on the number of incidents it closes or you know the the how quickly it is able to close an incident or an investigation, regardless of the token use, right? So in that case, companies or or like SaaS vendors are now incentivized to drive the same outcome using as few tokens as possible.
or using a smaller model instead of a frontier model for you know some of those tasks. So there’s a bunch of a lot of a lot of R and D these days is happening around how do you drive down marginal costs for delivering value to customers. Smaller models have also gotten really better at a lot of tasks. So you don’t really need the beefiest model for every task.
Right, and there’s so many different model architectures and you’re you you can have like an Oracle architecture where you have a smaller model that’s running a task and then it consults with a much smarter model as it as it needs. And you can have a pretty tangible effect on token costs if you use one of these patterns to drive down costs when you’re operating a SaaS business.
Kate Holterhoff (14:21)
Okay, that makes sense. I guess I’d be interested in hearing how you’re pivoting these observations and and I guess just the way that you are handling research right now, toward governance and and some of the things that you know a lot of the leadership are are concerned
Nikhil Mungel (14:37)
Yeah.
Kate Holterhoff (14:37)
around around these these subjects. So yeah, what is what’s your general stance towards governance when it comes to agents?
Nikhil Mungel (14:46)
I think it’s we are so early in the game right now and government’s just about to take off in a really big way. I mean it kinda reminds me of like the early cloud days, you know, like fifteen years ago or something where companies were trying to rein in, you know, cloud usage and I I don’t I don’t remember which CFO it was, but somebody interviewed them and they’re like, Hey, how do you use any cloud? Do you use do you have any cloud exposure? It’s like, No, we don’t use cloud at company and then they survey the
the the employee population and they had like a hundred cloud accounts, right? So there’s just a lot of shadow use. There’s a lot of shadow agent use, there’s a lot of shadow MCP use, which is fine because a lot of businesses right now are trying to turn around and deliver an ROI on all this token spend that we just talked about earlier. So people are
Kate Holterhoff (15:36)
Mm-hmm.
Nikhil Mungel (15:37)
okay, enterprises have been okay to move a little fast and you know, maybe bend some certain rules somewhere to, you know, just
lock in those productivity gains. But we’ve you’ve slowly started to see the shift happening between let’s move fast and drive value. Well the value is there, you know, the values are for everyone to see. If you use AI you’re you’re more productive than if you’re not using AI. So it’s fast, it’s quickly moving to a place where governance is becoming super important. And the the the successes that I’ve seen with good governance, it just starts with creating a good policy, right? I mean, you can’t govern if you don’t have policy
So successful enterprises or enterprises have been successful in governance have been successful in establishing a policy that makes sense for for for for the employee workforce. And it’s best done at the department level. You know, like the the governance policy for an IT team is going to be different from that for an engineering team, it’s gonna be different for a marketing team. But having that policy is like the first step. And once you have that policy, then you know, there are there are
There are a few SaaS tools available for you to establish governance where you can, you know, work with a variety of vendors to get that get that out there. And there are many strategies to, govern as well. I’m happy to go into those. But that’s where I’m seeing the shift right now. You know, there’s it’s governance not only from a and so w what what what do most of these policies include? You know, there’s a cost component, so like you know how much are we allowed to spend? There’s an access component, like what sort of data are you allowed to access. And then there is a
a what I like to call a blast radius where you know if you’re using an agentic software, an agentic tool in a production system that has a much larger blast radius, right? If something goes wrong, it might take down a lot of your production instances, like it might delete data from databases and, you know, might suffer data loss, etcetera. Whereas if you’re running an agentic coding assistant locally on your computer, the blast radius is limited, right? Because your local machine.
Right, you’re not gonna be deleting data for that you have stored for your customers. So different policy makes sense in both those different settings.
Kate Holterhoff (17:49)
Right. I’m glad you brought up the locally running models too, because you had mentioned small models, but I I I feel like what we’re hearing a lot of is this interest in having these models, yeah, just running off of you know, private machines. I mean, this is this has always been a a a big deal for especially highly regulated industries, ones that, have historically needed air gapping and things like that. So, and
Nikhil Mungel (18:11)
Yeah.
Kate Holterhoff (18:11)
you know, frankly the security issues right now are you know, deeply unsettled and unsettling.
So yeah, so th this is you know certainly aligning really well with with some of the things that we’ve been following. and so I I guess what I’d want to hear more about then is is how observability and telemetry come into that. Like what is I guess you know, what is your stance right now in in terms of like how you’re handling this at Cribl?
Nikhil Mungel (18:40)
Sure. We so we we help customers capture, you know, your LLM traces, your your your token routing traces, you know, your how how your endpoints are engaging if there are like shadow MCBs running on your computer, how your cloud security posture exists, right? If you have like configured a bunch of resources on a hyperscaler like AWS or G C P. So we do all those sort of things and you know collect all of that data for you and then
The beauty is that we are able to route that data to whatever destination you choose. So it could be a SIM that you already have, it could be a a a just an S3 bucket that you just want to store data in and you know, look at it for audit auditability purposes or investigative purposes a few years down the line. that’s where that’s where we’ve really seen a lot of observability and telemetry use cases come to light. And one of the one of the products that my team
launched last year, end of last year is called Cribl Guard, which enables you to redact, scan and redact sensitive information from your telemetry data, which is kind of pretty relevant these days, right? I mean, like so do you remember when back when, you know, we had like the first Chat GPT moment three or four years ago at this point, everyone was very hesitant about the sort of information you should be putting into Chat GPT, or Claude. And now everyone is pouring their innermost
thoughts into
chat bots. and
Kate Holterhoff (20:09)
sure.
Nikhil Mungel (20:09)
stuff gets logged, you know, like it it gets logged, it gets exposed in compliance APIs, especially if you’re using a chat subscription that’s provided by your employer, then it’s hundred percent getting logged. If you’re directly purchasing it yourself, then maybe it’s not getting logged. but one of the cool things is that when it does get logged, you don’t want to you don’t want that personal private information to be stored anywhere.
So you can use a product like Cribl Guard to scrub it out and remove sensitive information from your telemetry logs, right? So if somebody puts in their social security number or, you know, some private health information in there, guard can be used to scrub it out. I will say though, it’s not gonna scrub out your innermost thoughts. So those will still be there in text. So, you know, you should be careful about what you put in when it when you’re using an employee sponsored chatbot product.
But that’s those are some of the use cases that we have seen.
Kate Holterhoff (21:06)
Okay. And some of the ways that I have heard you frame this in in your writing around the subject has been with guardrails. and I feel like we’ve kind of alluded to this already here. So talk to me about the
Nikhil Mungel (21:18)
Yeah.
Kate Holterhoff (21:20)
role of guardrails in all of this. Like what what does that look like?
Nikhil Mungel (21:24)
Yeah, so guardrails really help you enforce policy, right? So you want to enforce somewhere in the chain. And there’s multiple places where you can do it, right? You can do it at like the token gateway routing. So and just to just for just for clarity, a token a token gateway or an AI gateway is a single choke point that can be deployed into an enterprise or you can use an open product like open router. And what do you want to do is you want to route all AI usage through that gateway. So that gateway is responsible.
or the right model, you know, of the right intelligence based on the task. It can fall back between multiple model providers. So if one model provider is not available, it can go to the next model provider. And most importantly, it can enforce policy. It can enforce these guardrails. So if a particular agent or a system or a person is using more tokens than they should be using, the gateway can enforce that policy and start rate-limiting them, or maybe
degrade them to a lesser capable model, you know, if they’ve used their quota for a more intelligent frontier model. So it can do all kinds of interesting things, right? Diff depend again, based on the policy that you want to enforce governance around and guardrails around. and then when it comes to privacy and you know deep sensitive data, which is something something that I spend a lot of time thinking about, guardrails enable you to prevent sensitive information from getting into places where you don’t want it to get into.
Because if it does escape something, or if it does escape into or g leak into, you know, destinations or, you know, surfaces where it’s not supposed to, then you risk being out of compliance. You might be, you know, moving out of or you might be in violation of a SOC 2 compliance or, you know, a an ISO compliance or a FedRAMP compliance control, right? So you have these compliance controls and you wanna make sure that you’re adhering to those compliance controls and that’s where guardrails are really, really important. Something else I’ll add to that is
Software 1.0, which is you what I like to call software that was all the systems and enterprise tools that people used before the advent of modern agentic coding and agent development, used to just have a bunch of buttons and you know, radio buttons and check boxes and you know, a few fields, et cetera, where it was a very very much an on rails experience. Software today has a lot of free form text inputs, right? Like a prompt box.
Where you can just go in and you can just describe, I’m looking for so and so information and I’m looking for y you might go into your business analysis system and you can ask questions like, Hey, show me why did this particular customer start using more of my product or you know, why did this customer churn and those sort of things. And even that text that you input into these boxes tends to have extremely sensitive information, which now
Kate Holterhoff (24:13)
Mm.
Nikhil Mungel (24:14)
you you have to protect, right? Because it all gets logged, it you know, gets it gets into telemetry data, right? It it it ends up in anything that your users type into a prompt box in your product is going to end up in a telemetry pipeline somewhere. And if it goes in the wrong destination, then you’re gonna be in violation of a compliance control. And that’s why guardrails have become even more important in agentic w in the world of agents.
Kate Holterhoff (24:40)
And in terms of cost, we’ve talked about tokenomics here, but it also feels like this could be a solution around telemetry cost as well. So is there a way that gateways are also serving a similar function there?
Nikhil Mungel (24:56)
Yeah, a hundred percent. I mean gateways do, you know, all kinds of really clever things these days. you can one of the one of the cool things that I’ve seen gateways do is obviously enforce rate limits. So you can only consume you everyone has a quota. And that quota can be on an individual basis, it can be on a team basis, it can be on department basis, right? And that’s something that you know, usually the CF or the CIO will determine what what makes the most sense in terms of an investment standpoint. And then token
There’s there’s a there’s an interesting phenomenon when you give your users the ability to pick a model, your user always gonna pick the most expensive model, right? Regardless of the task that you’re doing. Because you everyone feels that the thing that they’re working on is, you know, the most important thing in the world. And who wants to use a less capable model, right? So you’re gonna pick the the the best one. And I like to call it model anxiety, right? Where you’re you’re you you’re just like always going with the most expensive model and
Token costs. And really good token gateway products out there are able to abstract that away from your users. So the token gateway will be responsible for gauging how complex a query is and routing it to the model that’s the smallest model that’s capable of addressing that query. Removing that burden, that decision burden from users to select a model.
Right. So you just you just you just fire off a query and Cursor, for example, does this really well when you use cursor for writing code. They launched a feature recently which is called Auto Router, where you don’t you don’t have to pick a model, right? It’ll do it for you and you can just focus on the task that you have in front of you. And that is something that I’ve seen drive down costs quite a bit on the consuming side.
Kate Holterhoff (26:47)
Yeah, I agree that that’s where the future is heading. I think having the option of choosing your own model can be valuable, but ultimately who wants to make that decision for every little thing? and then I kick myself when I’m using Claude and I accidentally use Fable or something for like a a dumb little query, like ’cause it was just already that’s what I had it set up for.
Nikhil Mungel (27:08)
That’s right. Yeah.
Kate Holterhoff (27:08)
all right, so I
I wanna get back in this security issue, which I I you I feel like we’ve kind of been circling around anyways. I mean so much rest on that. and so you’ve been involved with this organization called the Open Cybersecurity Schema Framework. Can you talk about that? I you know, I I I learned about it from some of your writing on the subject and I I thought it was fascinating. So yeah, what is that?
Nikhil Mungel (27:29)
Yeah.
Yeah. OCSF is a or you know, it’s it’s called OCSF, and it’s a format for capturing and storing telemetry data so that it can be used by security products like a sim product or you know, like a number of security products. That it was it was a it was a standard that was born out of a few companies like Amazon and Splunk. it’s only like three or four years old and
The adoption of OCSF has been phenomenal because it’s such a robust such a robust format. So yeah, I’ve been I’ve been lucky to be involved with the OCSF community. I’ve you know, I’ve we had a a dinner at RSA just a few months ago and you know there was a
Kate Holterhoff (28:12)
Awesome.
Nikhil Mungel (28:13)
very welcoming community, you know, if people are interested in contributing to OCSF or, you know, being a part of the
Being a part of the group that you know develops that standard. You know, there’s a call Tuesday mornings that you know I occasionally will dial into. But yeah, it’s a it’s a it’s a format that standardizes logs, right? Or telemetry information, if I had to put it put it in a simple way. So you have logs and telemetry information that’s coming in from microservices, you have it coming in from endpoints, like my computer, for example, you have it coming from networking equipment, and every
Appliance, every equipment, every service, every application has a slightly different way of emitting telemetry data, right? And OCSF standardizes that. It standardizes that into like dozens of different types. And it there’s a a structure that it brings to the process. So when you store data or when you convert your logs and your telemetry data to OCSF, it’s it all looks the same. Like when you’re looking at
Firewall logs, doesn’t matter if they came from a Palo Alto appliance or if they came from a Cisco appliance, they’re going to be structured in the exact same way. Which has a bunch of benefits downstream, right? I mean, if data is structured in a particular way, it can be stored in a way that’s really fast to retrieve it. You know, you can index it, you can optimize storage, you can reduce duplication. There’s a lot of interesting things that you can do from an engineering standpoint when you’re when you’re handed structured data. And again, the the reason why this is really important is
the sheer volume of data that we deal with when it comes to telemetry, right? I mean, enterprises are creating petabytes of telemetry every single day. And then when there’s an incident, when there’s an investigation, you want to be able to pull insights from petabytes of telemetry pretty much instantaneously, right? Because there’s an operator, you know, you’re you’re you’re you you got paged, maybe there’s a security incident and you’re going into whatever tool you’re using and you’re like, well show me what happened.
And the system now has to comb through petabytes of information to give you that answer, right? I mean, if you think about it, one way that I like to explain observability and telemetry to people is it’s an information compression problem, right? You’re trying to go from petabytes of data to like a kilobyte of data that can be presented to a human being, right? You’re you’re trying to go from all these logs into like a sentence. Like, tell me what happened. Like, you know, which service is failing right now, or you know, where’s the intrusion?
and in order to do that, you need all these optimizations that OCSF brings you or OCSF brings to the game. And you know, storage systems can then take advantage of the structure to save that data and store that data and index it and you know pre-process that data so that it’s fast to retrieve that data. I’ll I’ll put in the plug by saying one of my teams has created this awesome product called Copilot Editor, which can be used to transform data into OCSF. So you can create
if you have high throughput streaming data that you want to convert into an OCSF format, you know, be the latest version or, you know, even we support like several versions, it will automatically convert that data into OCSF for you. And it’s an open format, right? It’s an open cybersecurity format. So you can convert it to OCSF and then you can send it to whatever destination of your choice and then you can take advantage of what OCSF the OCSF ecosystem brings to the game.
Kate Holterhoff (31:41)
Yeah. I mean it just makes so much s yeah.
Nikhil Mungel (31:41)
Those are really long answers. I’m really passionate
about OCSF so I can’t stop talking about it.
Kate Holterhoff (31:46)
No,
I I w I actually want to hear a little bit more about it. so am I right in assuming then that because of AI and because we have just all of these systems that the the the need has always been there, but it is so important now that the community is finally like, Okay, we need to do this because yeah, yeah, what, three years ago that was really when these things kicked off. is that accurate or has it just been brewing for for so long it was gonna happen eventually?
Nikhil Mungel (32:13)
Yeah, it was it’s been brewing for a while. I mean OCSF and
Kate Holterhoff (32:15)
Okay.
Nikhil Mungel (32:16)
the the forces that led to the creation of OCSF have been around even before agentic AI took over. it it supports agentic traces, you it supports LLM traces, it supports agentic observability. So it’s I think there’s a new OCSF version that comes out every few months. And the team does a fantastic job of keeping up that standard and the vendors that support OCSF do a fantastic job of adhering to that standard.
So if you’re, you know, sending your data to Amazon Security Lake or, you know, whatever sim that you’re sending to, they’re they’re usually pretty up to date on stuff.
Kate Holterhoff (32:50)
Huh, my goodness. How many people are are working on it right now? I mean, this sounds like it must be a very robust community.
Nikhil Mungel (32:57)
Yeah, I I feel like the community is it’s a handful of people. I’m like six or seven core committee members. I’m not a core committee member, but I know several of the core committee members and they work you know, all some of them work at di all of them work at different vendors, which is really what makes it a pretty well adopted standard.
Kate Holterhoff (33:20)
Hm. Mm hmm Okay, cool. Well I appreciate you talking about that. We’ll be sure to put a link in the show notes for folks who w who wanna learn more about that particular what community here ’cause I you know, I learned
Nikhil Mungel (33:32)
Yeah.
Kate Holterhoff (33:32)
something. So yeah, I l I I feel like we could still dig in a little bit deeper to this like agent telemetry question here. So how about the issue of retention? Like in regulated industries, that that’s a huge issue. I the the cost
Nikhil Mungel (33:48)
Yeah.
Kate Holterhoff (33:48)
of it can be pretty daunting here. So I guess one one aspect of that is like, should an agent be fed its own cost as a as like a planning input? I i you there’s there’s a couple of ways to go with this, but yeah, what’s what’s your sense around that?
Nikhil Mungel (34:00)
I think so when it comes to data retention, one of the hottest topics has been ZDR or zero data retention, right? Which is which which really means that whoever is giving you or whoever is fulfilling your inference is not should not be storing a copy you know of the inputs and the outputs that came from the model. Which in theory makes a lot of sense. You know, if you’re if you’re an enterprise, you don’t want to
You don’t want your LLM vendor to store like a copy of your data. But then there’s also the issue of safety as it pertains to LLMs. And this is something that I’m actually pretty you know, I have I have a lot of opinions on this, which is, well, you you can, I mean, these are these are really exceptionally intelligent models. And they have you have to have safeguards around these models. And in order to enforce those safeguards, even as a model provider, if you if you want to enforce those safeguards, you have to store a copy of that.
you know, the inputs and the outputs for a period of time to scan those things. That’s one of the reasons. So I understand, you know, why AI labs and inference providers have a policy where they are required in many cases to, you know, store a copy of that data and you know, use it for scanning and make sure that nobody’s abusing the system or nobody’s using it for purposes that they should not be using it, like you know, creating new vulnerabilities in software.
or cyber offensive work or, you know, whatever. You’re just red teaming the model and you’re trying to subvert the safeguards that have been encoded into the model weights. And on the other hand, enterprises also have a fair point. Like they don’t want their their corporate information being stored, even if it’s just for a day or two by an Amazon or by you know an AI lab or you know anyone else. So it’s a pretty I think both both both the sides have a good a good balanced argument on that front.
But your your your you also mentioned, you know, agent costs and providing a budget to agents in the planning phase itself. And that’s a pretty, you know, popular pattern. Like, you know, agents have a tool budget or, you know, a some sort of a budget to that they have to remain or they have to follow in order to accomplish their objective. And if you do follow that pattern, that does
result in slightly more optimal performance. So yeah, I think it’s a pretty common pattern that that we’re seeing across the industry, or across across, you know, most of the H gen T frameworks or frameworks that are used to create agents.
Kate Holterhoff (36:35)
Well I feel like we have got a lot to to chew on here. You’ve given us I yeah, a lot of interesting ways of of thinking through this sort of broader idea around how telemetry is changing, how we should approach this issue of what observability writ large in in this agentic era. So so thank you for that.
Before I I wrap us up though entirely, where are you directing folks for more information, both around how Cribl is helping with this and for your own writing about this, which of course I’ve cited a few times.
Nikhil Mungel (37:07)
Yeah. So we have a we have a new research portal that’s gonna be coming up on cribl.io. So we should post a link on that in the show notes. And then that would
Kate Holterhoff (37:15)
Can do.
Nikhil Mungel (37:16)
that would be the best place to, you know, get people plugged into what’s happening as it, you know, as it relates to research and development in AI, at the intersection of AI and telemetry. That’s how I frame it. Right. So if you want to keep up, like I would I would direct people there and then I obviously have a LinkedIn and a Twitter profile as well.
Kate Holterhoff (37:35)
Well, on that note then let’s go ahead and wrap up. I have really enjoyed speaking with you today. My guest has been Nikhil. again, my name is Kate Holterhoff. I’m a senior analyst at RedMonk. If you enjoyed today’s conversation, please like, subscribe, and engage with in the comments.
