Most of an enterprise application stack is open source that nobody is paid to maintain. Gunnar Hellekson of Red Hat puts it at 60 to 70 percent, and that gap is what IBM’s Lightwell is built for. He walks James Governor through the mechanics: members scan and triage their own dependencies, submit what they find with the source identity stripped out, and get fixes back after an embargo period before those fixes reach a wider network and, where a maintainer still exists, the upstream project. Financial services goes first. Hellekson is candid that plenty of customers cannot use the service yet, because joining would only lengthen a patch backlog already measured in months. Mythos, the two agree, mostly revealed how bad things already were. They also work through token economics, why nobody needs to scan OpenSSL separately, and why a fix should end up as deterministic code rather than something an AI keeps rediscovering.
Red Hat is a RedMonk client, but this is free and independent content.
Links
Transcript
James Governor (00:04)
So thanks everyone for joining us. thank you very much, Gunnar Hellekson from Red Hat for coming and explaining Project Lightwing, what it means, what it’s gonna mean potentially for the industry and certainly for IBM’s customers.
Yeah, I mean, you know, I I think it it’s kind of funny. I was actually I was having my lunch a little while ago and I was like, Huh.
This used to be something and and we’ll we’ll we’ll get to the specifics in a minute, but this used to be something that IBM was pretty good at. They would put out these, you know, press releases statements like, we’re gonna invest X billion in X. And, you know, it it’s sort of a way of putting a marker down, getting the market to understand from a strategic standpoint what’s important, you know, whether it’s like we’re gonna invest and I I’m actually gonna go back
You know, but I I’m old enough for me like IBM would be investing in this much in right to the olden days in Linux or in eBusiness or and so for me I was like, huh, Arvind is using a bit of the old playbook, but has taken this very, very important issue and highlighted it. And apparently it works because the share price got a boost based on the news, so that is always welcome. when you’re the
the you know, running a company, you want a bit of that. But yeah, I just thought let’s get into some of you know the the the practicalities. and and and yeah like you know it it press releases are always at a somewhat high level but it this is such an important area and I I think you know particularly in this era where we are
There are a lot of questions being asked about, you know, open source sustainability, and you know, what it means and what it should look like. so yeah, and frankly, obviously, it’s one thing for, you know, vendors to be like, Yeah, we’re gonna fix our you know, we’re gonna fix our vulnerabilities, which is extremely fricking hard in the current environment, you know.
you know, post Mythos and and you know, there was a reason why Glasswing happened. So yeah, it it it just particularly when there’s customers with like on-prem software, they you know are still running you know older versions of things. We know that that you know there are definitely vulnerabilities. Well there are vulnerabilities in new open source software, there’s definitely vulnerabilities in older versions of open source software. So I was just like, let’s drill into this a little bit.
I’m I I think I think it is, you know, such an important issue. And the truth is is that, you know, a little bit of consulting is gonna be necessary. And this does seem to play into some sort of IBM strengths. So taking responsibility, making money for doing so, yeah, that that seemed like something that I wanted to follow up on.
Gunnar Hellekson (03:18)
Yeah. Yeah. So, you know, one of the usually I don’t start with this, but the way that you framed it made me think I one of the things I really I’m genuinely excited about this part of the story is that, like you just said, we figured out a way to help to solve the problem and do it for money and also make the open source community a little bit safer, right? It seems like it actually seems like a win win in kind of every direction. and
So, and I think if you were to talk to Arvin or if you were to talk to Hicks, they would tell you that this is this is important for the future of open source as a going concern. because if we don’t as a as a as an ecosystem, if we don’t create mechanisms, norms, processes that can that are sustainable, that can actually make this stuff safer, then we’re putting the whole kind of put the whole model at risk, right? So absolutely. so that’s so that’s that that’s I’ve been doing like eighteen hour days on it for for the last several weeks. and that’s what gets me up in the morning is is actually is it’s a it’s a very exciting mission. I’ll put it that way. Yeah.
James Governor (04:38)
Yeah, I mean I think yeah, if you’re not excited by trying to solve hard problems. I mean so yeah, no. but make sure you get some rest. I hope you have some vacation planned this summer.
Gunnar Hellekson (04:50)
I I did.
James Governor (04:56)
amazing. So tell me a bit about the mechanics of of sort of marshalling this and you know particularly you know obviously we’re on a multi-je multi-year journey of sort of integrating Red Hat and IBM. we we you know, you you kind of you’re you’re you’re you know, IBM wants to hug Red Hat, but not too hard. maintain that independence. But something like this does feel like a cross-cutting sort of activity. Yeah, what is it? How does it work in practice? How’s this gonna how’s this actually gonna
Gunnar Hellekson (05:27)
Yeah. So this is s okay, so maybe maybe let me start by like
just spend a couple seconds kind of framing exactly which problem we’re solving and how we’re gonna solve it and then that’ll help me tell that’ll help me answer the question. So you know both Red Hat and also even IBM have been solving this problem for the products for some time. And you know that’s kind of one of the premise of RHEL for example. Yeah. And we had already started working with like putting some
James Governor (05:58)
Absolutely.
Gunnar Hellekson (06:05)
harnesses in place to actually do automated scanning. This is this even predates Mythos, which is like doing automated scanning and then figuring out how to quickly kind of move that through the system.
This, there’s a the problem specifically that Lightwell is solving is not actually the product problem for which we already have a solution. Like we already know how to staff that we already know how the SLAs and all the other stuff. For the I’m gonna fudge the numbers here, but it’s something like 60 to 70 percent of the stack is completely unsupported open source. Yep. With with varying levels of Of kind of maintainer availability and responsibility.
James Governor (06:49)
Absolutely no doubt. Yeah. There’s all sorts of of of projects that are yeah that we rely on that are, you know, some of them are orphaned. you know, some some are individual maintainers who, you know, are swamped. Yeah. yeah. But there there are there are plenty of orphan projects out there that organizations rely on.
Gunnar Hellekson (06:58)
Yeah.
Yeah, that’s right. And so it’s you know, the old XKCD with the one
James Governor (07:12)
But absolutely XKCD that is exactly what we’re living.
Gunnar Hellekson (07:15)
Yeah, yeah, yeah. That’s right. So there needs to so the so if you are a bank, you’re you have two bad options. The first is either you can go solve it yourself, which you could, you could go burn the tokens and spend the calories to to go fix whatever it is, knowing that you now own that fix for the the rest of the life of the application. Right. And now you you’re basically you’re you’re carrying your own fork. It’s fine. some people encourage this.
Like I know Mitchell Hashimoto is a big fan of like, nope, just vendor all your own stuff and just you know, just it’s it’s now your problem. It’s like okay. Kind of inexpensive way and kind of creates its own creates its own risks. So not a great solution. The second option is well go file the ticket with upstream and hope that it gets fixed and hope that their embargo, if they have an embargo system at all, hope they have one. and if they do, hope it holds, etc.
a lot of people have been relying on volunteer efforts for a very long time. Absolutely it’s kind of amazing that it has worked as long as it has. And it’s for good reasons, right? We benefit from it, everybody benefits benefits from it. But in a world where the number of security vulnerabilities is inexorably rising, this is even pre Mythos, it’s inexorably rising. The ability of if the if if we believe the number that the average
time to exploit is now negative seven days. And we believe that the average enterprise it takes half of critical vulnerabilities aren’t even patched within a year. That means that there is a lot broken in the system and it is not even it’s not even just the upstream that’s broken. Yes, the upstream is broken. And then also the remediation process is broken and anyway so lots of opportunity to to fix things there.
James Governor (09:17)
Could start all new development from right now using hardened images, and you would still have an enormous problem in most enterprises.
Gunnar Hellekson (09:25)
Yeah, yeah, that’s right. Yeah, that’s right. so and in fact that’s when we talk to customers about it. This the the first thing the first wicket they have to run through is like, did you eat your vegetables? Right? Like are you like did you go to Hummingbird? Are you using hardened images? Like are you know, do you actually have like, you know, paid are the platforms actually paid for stuff?
James Governor (09:43)
Okay. Have you brushed your teeth? Did you f did you spit the toothpaste out?
Gunnar Hellekson (09:45)
Yeah.
Yeah, yeah, yeah. That’s right. so then what then we get to what Lightwell actually does. So in exchange for a in in exchange for an embargo period, if you are a member of the clearinghouse, you can right now we’re for these early members, we’re requiring them all to b have the ability to do their own scans and do their own triaging. So they’re doing their own scans, they’ve got spreadsheets of
you know, spreadsheets of vulnerabilities that don’t even have CVE numbers yet. Some of them are singular, some of them are chained. And they bring them into the clearinghouse. It’s all deduped. we have a process for triaging them and then set all the robots and people to task on on resolving it. When it’s resolved, it is then shared first with the clearinghouse after a respectful embargo period. And then it is released
out to the network, which is so there’s a two it’s a two tier system. So you’ve got the members of the clearinghouse and then you have the light the Lightwell member network. The Lightwell member network is a read only kind of recipient of all the fixes. And then you have a even broader circle which is the upstreams. And so anybody can do AI scanning and remediation
Right. If you have enough money, like you can do you can do that as as much as you want. what we’re doing though is not just remediating it, but we are also ensuring that part of what that 20,000 number is, is actually having the bodies and the people to go have the relationships and with the upstream so that we can actually get these patches. If there is somebody to catch them, we want to give them an opportunity to catch them and then run their own embargo process or whatever and and upstream it. If there is, if they don’t, if they can’t.
or won’t catch the fix, then we would then we carry that ourselves. So in other words, protecting the number members of the network. Does that make sense?
James Governor (11:55)
Yep. Yep. So
Gunnar Hellekson (11:57)
And the and I should say this too. The scope of this is specifically around the 60 to 70 percent. So we’re very specifically focusing on application dependencies. And so like people ask us like, well, are you gonna do it for the kernel? Or are you gonna do it for Kubernetes? I was like, well, no, no, no. Those are all we already do that for and we have products for that. this is for all of the otherwise unsupportable stuff.
James Governor (12:22)
Yeah. Yeah. And, you know, over time, well there’s a couple of things. A, hopefully over time, there are fewer and fewer problems. But as a data thing, it’s quite interesting. You’re gonna get a lot of insight into you know, I think w the that’s one of the things is like when we’re in a cloud environment, we can sort of know what everyone is running. Mm-hmm. But that’s not the case on prem. we have some idea. but yeah, just getting a sense of Yeah, you’re definitely gonna be running into projects where you’re like,
Gunnar Hellekson (12:57)
Yeah. Yeah, that’s right. And so there there is a there’s definitely a network effect here. There’s kind of a Metcalf’s law, right? Where like just the more people get in, then everybody kind of benefits from the and getting kind of a clear view. So that’s another part of it, is that the the initially the clearing the first clearinghouse we’re doing is specifically around financial services, but you can imagine kind of a vertically oriented clearinghouses elsewhere because they all have their own norms.
Their own regulations, their own priorities. And so we’re going to get it’ll be interesting to see how that evolves. Like the, you know, so there’ll be a telco one or maybe a government one and so on. But yeah, sorry, and I should mention too, there is a the lawyers told me yesterday I’m not allowed to use the word anonymous, so let’s call it confidential. But there is a there’s an identity washing that goes on because once the CVE enters the system, we don’t particularly care where it came from.
Right, it’s just it’s part of the it’s part of the soup. so that gives some assurance to the cl to the to the customers of Lightwell that helps them manage their risk.
James Governor (14:09)
Yeah, absolutely. The last thing you want to be like, we have this problem. because if you have the problem, then Yeah, we should have a fix sometime soon. Okay. Tell me a bit about the balance there that I mean because the second order effect is, you know, a lot of organizations are
Gunnar Hellekson (14:13)
Right.
Yeah, that’s right.
James Governor (14:34)
As they go through this process, they probably are gonna start saying things like, Hang on a minute, I’ve I’ve really got to modernize this you know, it doesn’t to your point. You you if you find something that where you’re like, wait, IBM or you know, the project is gonna have to support this thing that no longer has a real maintainer, yeah, then really you probably wanna be getting off that thing. I mean, you you want something that has somebody that is looking after it now.
Gunnar Hellekson (15:04)
Yeah, well it’s gonna be absolutely and
James Governor (15:07)
I mean that that’s why I sort of think the the I mean I I you know I’m I definitely hesitate to I would definitely hesitate to put words in Mitchell Hashimoto’s mouth because I think that would be bad. He has very strong opinions and I will allow him to express his own opinions. but what I would say is if you know, it would be an intri if if people went through the process and they’re like, you know what?
I wanna have to manage all of my vulnerabilities across all of these new things that I’m gonna build because if I build it now it will be secure. And I mean there there’s that would be an interesting I don’t imagine that’s I mean certainly the kind of people that are engaged in this process. I don’t think they’re gonna suddenly go, I know what I’m gonna do. You know, I’m not gonna use any of this, you know, vendor or project supported stuff. I’m just gonna build my own and maintain my own.
Gunnar Hellekson (16:04)
Yeah, that’s right. Yeah, not well, and especially the for the kinds of people who are gonna be interested in this, these are the same folks who measure their system update. Like they are lucky if it takes less than a month to deploy a single fix to the entire state. It takes in some cases months, right? Often more. And so there’s another aspect of this, which you know you mentioned consulting earlier, and we’re already We’re definitely already talking about this. Is the Mythos, if it didn’t do anything else, it made everybody realize how dangerous things already were. Now that it’s which is only now accelerating because of the AS. But everything was bad even before Mythos.
James Governor (16:48)
Everything was a complete shit show. No doubt whatsoever. Yeah.
Gunnar Hellekson (16:52)
And so this is now everybody’s but everybody woke up now because now all their boards are asking about Mythos. And now they’re trying now they have now there there is an enormous pressure to improve the ability to remediate and and and not even security fixes, just patch in general. It becomes much more difficult to explain why it’s taking nine months to go close a critical vulnerability. The risk board wants that to be measured in days, hours, that kind of thing. So there’s a There’s a lot of hydraulic pressure on the entire IT operations.
James Governor (17:26)
Well, and it’s almost like IBM has some automation technology that might
Gunnar Hellekson (17:31)
You know, I’m I do yeah, that’s I never thought about that, but you’re right, yeah, that’s right.
James Governor (17:35)
Also I’m glad I’m here to tell you that that yeah, IBM actually does have some automation technology that can be quite relevant when you have of of of of of clusters. Yeah, no, and you know, VMs and containers. Yeah, I mean, you know, you you oughta think about this. You know, it could be an opportunity for you.
Gunnar Hellekson (17:56)
I’ll take that back. no, it’s true. That’s that and and in fact, what we’re discovering is that you know, we’ll talk to some customers who are not even able to consume the lightwell service. It’s just not even practical. Because all if they pr if they participated in Lightwell all it would do is make their existing backlog that much worse.
James Governor (18:19)
Right.
I I think I think AI has that effect actually just in software, just in how we build and manage software. So, you know, and we you know, we’ve had this with, you know, various t technologies before, but yeah, it’s like, well, you need a certain level of Yeah, if you do not have a certain level of maturity and rigor and operational excellence, then AI is just gonna find you out. Like you everything is in people’s heads. Funnily enough.
It’s gonna be difficult for AI to help you build software. have really good documentation, really good defined standards, really good internal communications mechanisms that capture things and conversations, then you can benefit. If you haven’t done all of that, then AI is gonna be very much a problem in terms of building software. And I think
Gunnar Hellekson (19:13)
Deming taught us, right? There’s you can’t you can’t automate what you don’t understand. Yeah. Right. Right.
James Governor (19:20)
So th you know, you and and this is slightly scary, but but makes sense, you need a certain level of maturity to even benefit. But there’s always opportunities to go further. What’s the how how are you identify like what are the cutoffs? Like are it do I need to be is there a level of currency I need? Is there a level of automation I need? do I need to be running containers? Do I need to be running like what is that?
Gunnar Hellekson (19:51)
No, so that yeah, so the the the actual artifacts that we’re shipping or the way that we are shipping artifacts now is gonna look like it’s gonna look like a Maven, a PyPy an NPM you know, kind of layer in front of the in front of the actual repos, right? And so could be containers, could be not, doesn’t really matter too much. But the operationally, it’s really just the ability to
James Governor (20:20)
I don’t think making rapid changes to your application or to your infrastructure estate, that’s I mean, put it this way, like if you’re not virtualized, then
Yeah.
Gunnar Hellekson (20:33)
Struggle here. Yeah. To the extent your application is is tied to the metal you are in for a bad time. Like it’s like the the more abstracted you are. And actually, but like if you do not have a CI process, you’re you know, you’re probably in trouble. or you need to find other ways of managing the risk. Uh-huh. there is a it’s and this is kind of an this is now now I’m off script.
But I think this is true, is there’s also this I don’t know if you call it a virtuous cycle, but there’s this kind of inexorable acceleration that comes from this, right? Because if the attackers are moving at machine speed and you’re moving at human speed, it’s you’re in trouble, right? That’s but now if you’re both moving at machine speed, now it’s a race of like what, who burns more tokens before you get in other words, like I am not persuaded yet that there is a stasis.
that can be reached or some balancing point that can be reached between the attacker and the defender. it seems like the the imbalance between the two, it just seems permanent, d I think.
James Governor (21:44)
Yeah.
Well, I mean, there was quite an interesting framing yesterday at an event I was with that that Fastly was running and they were talking about their web application firewall. And you the the thing there is the balance between, you know, basically burning tokens and then having something that’s predictable and reliable. So you wanna first pass where you’re finding things with AI, but then you wanna f you wanna generate code that fixes the thing.
Rather than using AI y all the time. I mean the idea that you’re just endlessly b burning tokens, I mean, fine for Sam Altman, but like maybe not so good. So you yeah, you want to build a predictable, deterministic, reliable, code driven fix, I think, rather than being constantly burning tokens.
Gunnar Hellekson (22:24)
That’s right.
Yeah, and not just yeah, it’s not just efficiency, as you say, it’s for it’s for the determinism, right? I mean that’s the absolutely yeah. That’s the that’s the I mean that’s the story that our Ansible folks say, right? It’s like, are we going to manage systems agentically? Yes, we are going to do it. Are we going to let the agent decide how to reconfigure the MCP server? No, we are absolutely not. Like there’s there’s a right there are specific ways that that’s supposed to happen and we’re gonna describe them, right? yeah.
James Governor (23:06)
So I mean I that’s actually I mean that’s that’s not no that’s not an angle that I so I you know I’ve been saying you know that the best token is the one that you don’t burn. the the second best one is probably the one that you burn locally, you know, with with an open weights model or something like that. And and then the third one is, you know, the one that you’re you know, there and and look, don’t get me wrong, there are definitely reasons to use the latest models, specific tasks.
But but in this case there is there is a there is a token economics token efficiency argument because you’re by everyone collaborating, you’re not instead of everybody burning their own tokens to fix their own problems, they’re centralizing the token burning that solves the problems. And that seems to me yeah. so yeah, there’s definitely a a token efficiency angle to like well.
Gunnar Hellekson (24:05)
Yeah, for sure. Yeah. Both on the scanning and on the remediation side, right? And and in fact, what we’re seeing is, you know, clear you know, by the end of the year we’re gonna have what a half a hundred clearing houses of various sizes and shapes and whatever. But one of the first things they do is like, we don’t need everybody scanning open SSL. Right? Like it’s like we just need one and that’s and everybody else can can share it. So peop you know, the market is already discovering this pattern anyway, right? it’s
James Governor (24:08)
Absolutely. Yeah, yeah, yeah.
Gunnar Hellekson (24:34)
especially with open source, it actually makes that possible in a way that it isn’t. Yeah, that’s right. and it’s funny you mentioned the best token when you don’t burn. What you made me think of is now that everybody’s figuring out exactly what AI is good for and how to deploy it, it’s starting it’s starting to me to feel a lot like machine tools, right? Like tools that build other tools as opposed to it seems
James Governor (24:36)
Absolutely.
Gunnar Hellekson (25:02)
Exactly for the token efficiency reasons and the determinism and creating predictability. Yeah. I agree.
James Governor (25:08)
But that doesn’t mean that you want complete goo and everyone using everything that they I mean I’m I yeah, I mean I yeah, I actually spoke to Ed Anuff and Anant Jhingran both, you know, IBM data people about this the other day. This this sort of question about mass customization. You know, is is that what we’re gonna have? Just everybody building their own thing for every problem. Right. And I don’t know. I I I don’t think we are.
Gunnar Hellekson (25:42)
think so either. I I mean my own experience was like I got three weeks into like building my own, you know, my own little hobby agent harness. And then like literally accidentally stumbled into Hermes and I was like, somebody else already did this. Okay, great. Like why would I why would I go rebuild this when somebody has already solved the problem? Like so I think people unless they’ve like unless they have like a Once they like a real fetish about the work, they’re probably gonna just sheer laziness will drive them back into
James Governor (26:17)
That’s the thing, I think humans are fundamentally pretty lazy. Like if you can if if there’s a thing that you can use, yeah, sure, you might want to no, don’t get me wrong. Like it you know, people love doing you know woodwork. Yeah. People love doing art. There are there are creative drives here, so technologists like to build technology.
Gunnar Hellekson (26:22)
It’s more involved.
James Governor (26:45)
And we’re seeing a lot of that activity. I mean, you know. the the you know, a lot of the token maxes are people that, you know, just constantly I mean, I’ve like three friends, they’re all building they’re all building their own database. It’s like I’ve never found the database with exactly the optimizer I wanted. So I’m gonna build my own database. but but whether
I I find it un unlikely the idea that every enterprise will invent its own database and then have to maintain that database in some sort of I mean I I mean what do you do? I mean, I guess y if someone leaves the organization then you just invent a new database. I mean I think
Gunnar Hellekson (27:32)
Well and to connect it to the previous discussion, inventing your own database also means basically creating your own CVE factory, right? Like why would you do something?
James Governor (27:38)
On your own. So I mean i I I think I think there’s there’s there’s life in the old in the old you know software business yet. Yeah, yeah. Okay. I mean, you know, honestly, I think I’ve got most of what I needed. I did want to catch up. but I I guess I should then finish by saying that was an accidental MonkCast. a RedMonk conversation yeah, but at the end of the day, you know, if we think about large banks, telcos, you know, any financial services company manufacturing, they’ve all got a huge estate out there that that that needs fixing. they’re gonna need help with that. And yeah, it absolutely makes sense for IBM to be jumping in. So thank you very much again, Gunnar, for joining us and thank you all for listening in.
