---
title: "Evidence Over Certificates: John Ellis on the Eclipse Trustable Software Framework"
date: 2026-08-13T08:00:38Z
modified: 2026-08-10T11:43:38Z
permalink: "https://redmonk.com/videos/john-ellis/"
type: video
status: publish
excerpt: ""
wpid: 5036
categories:
  - Podcast
series:
  - Conversations
channel:
  - RedMonk Video
featured_image: "https://redmonk.com/wp-content/uploads/2026/07/John-Ellis.jpg"
timestamp: 2026-08-10T11:43:38Z
tags:
  - Podcast
  - Conversations
  - RedMonk Video
---

What does it mean to trust software? For this RedMonk Conversation, Kate Holterhoff sits down with John Ellis, President of Codethink and the contributor to the Eclipse Trustable Software Framework, to pull that question apart. Ellis leans on an old image: the bridge builders who once slept under their own bridges to prove the work was sound. Modern software rarely faces that kind of test, even when it steers a car or flies a plane. He explains where trust tends to break, especially the integration step where hidden dependencies finally show themselves, and points out that a safety certificate almost never uses the word “safe.” Rather than pass-or-fail box-ticking, the framework asks teams to state their confidence and back it with evidence that others can inspect and challenge. They also dig into AI-written code, the EU Cyber Resilience Act, and why rising software recalls suggest current habits fall short.

This RedMonk video is sponsored by the Eclipse Foundation.

## Links

- [LinkedIn: John Ellis](https://www.linkedin.com/in/johntellis/)
- [Eclipse Trustable Software Framework](https://projects.eclipse.org/projects/technology.tsf)
- [Eclipse SDV](https://eclipsesdv.org/)
- [CRA information](https://orcwg.org/)

# Transcript

**Kate Holterhoff (00:04)**
Hello and welcome to this RedMonk Conversation. My name is Kate Holterhoff. I’m a senior analyst at RedMonk. And with me today I have John Ellis. He’s the president of Codethink and the contributor of the Eclipse Trustable Software Framework at the Eclipse Foundation. John, thanks so much for joining me here on the MonkCast.

**John Ellis (00:20)**
Morning and welcome. Thank you. Appreciate the time.

**Kate Holterhoff (00:23)**
So let’s begin with some definitions here because I’m sure not everybody is familiar with some of these things. So what is the Eclipse Trustable Software Framework and how did the project begin?

**John Ellis (00:34)**
first code think is the contributor, not me. I an Aaron Joe sort of went just to make sure, but so the Eclipse Trustable Software Framework actually started as the trustable software framework. and it started Codethink started that journey a good number of years ago, about 2016. Bosch asked the question could open source be used to deliver safety critical context? we being a Bosch partner, we looked to in look into that question and we came quickly to the determination that it could

But it wouldn’t work commercially because of the just the opaque, just the contribution. Open source is open, the safety critical is closed. There’s contracts, there’s NDAs, there’s confidentiality, so there’s a mismatch in the business model. But in doing that work, we actually came upon a deeper question, which is: well, why do we even trust software? And the question actually prompted us to look for getting the safety certificate, why do we trust software? and as professionals, we had a vast amount of examples.

Which suggested maybe we shouldn’t. and thus was born the project and a whole bunch of other sort of steps behind that. but it then resulted in the the trustable software framework, which we got to a point where like, this is this is gonna something that’s gonna work and then we realized it needed to be open and so then we contributed it to the Eclipse group, specifically the SDV working group.

**Kate Holterhoff (01:50)**
Okay, that’s a that’s a good pithy history here. I like that. so you’ve been doing a lot of talking about this. I I’ve been fortunate enough to watch a couple of your presentations so I I feel like I have my bearings at least a little bit here. And so I’m hoping that we can dig into some of the the things that you you have said already about this framework. So just to begin with, I was really struck by your metaphor about

Bridge building and and how that relates to trust here because I feel like trust is kind of one of those ambiguous terms, like, yeah, we want trust, okay. But like what does it mean specifically to have trust in software?

**John Ellis (02:30)**
yeah, so it it is it’s a great it’s a great metaphor and it does often, you know, poke the audience in a in a very specific way. Right. The metaphor very specifically is is it in ancient times, right, the anecdote is is that architects and bridge builders were forced to sleep under the bridge, right, to prove to society that what they built was was in essence safe or usable, right? and if they were if they refused to do it, then you know society should take a step back and question like what actually happened.

**Kate Holterhoff (02:38)**
Yeah.

Mm-hmm.

**John Ellis (02:57)**
So the metaphor says exactly the same for software today, especially as we’re talking about the software in such embedded and regulated markets, right? Aviation being one, but autonomous vehicles, we are we are asking society to allow for, you know, basically in essence technology companies and engineers sitting at a desk to make decisions that impact society at large. and so, you know, we’ve actually provoked the quit engineer, would you be willing to put your loved ones on the road, laying horizontally in front of an oncoming vehicle?

and again it it’s it’s it’s intended to provoke deeper thinking about like what does it really mean to trust software? Are we s are we comfortable that a safety cert is sufficient? Are we are we comfortable that commercial contracts are sufficient? Like what does it really mean to to have trust? and what part of the trustable software framework and sort of the journey, right, in terms of getting there is those at those those individual points.

Are not sufficient. Like I can’t have just you saying, here’s a cert, trust me on this date. Like I it’s a continued relationship between me, me society specifically, and and how this works. So we hope that in framing that metaphor, right, we get people to really pause and say, like, wow, like, yeah, this is a really big deal. And a contract or a license or a cert or whatever else we currently use today is is really insufficient.

in allowing for us to actually have that conversation.

**Kate Holterhoff (04:27)**
Okay, that helps. And I love thinking about software as a bridge because I feel like you could take that image anywhere. But one of the places that I I I find interesting that you take it is to this conversation around integrations So you have described integrations as this moment of truth. So it’s like this, button push where everyone they’re just hoping that it works, right? And these unknown dependencies and non-reproducible binaries.

they suddenly surface and and everything is kumbaya. So talk to me about integration and how it factors into your thinking around trustable software.

**John Ellis (05:03)**
yeah, I I just as with everything I like to do, there’s there’s analogues and metaphors and and the reason for integration, in in part one of the conversations that I had in at least at least one of the talks that I think you’ve seen is we talk about sort of like the you know the idea of a factory or the idea of where integration occurs. We can kind of get a factory metaphor. and at least for most of of the audiences working in equipment like if it’s automotive or medical or others, having very similar models, right? They

They have an OEM that has a whole concept of a factory and building of a physical product. And it is incredibly interesting that when you talk about the factory, down to the nth detail, they have exactly what they’re expecting, when they’re expecting, how to expect it, how to integrate it. Right? The factory is where there are no surprises. Like there are no surprises. You do not, in fact, everything is about making sure the factory works. Unfortunately, that mindset.

hasn’t actually come over to software. And so because it’s not in software, that’s what we mean by the integration. And so the one talk that you saw where we had the picture is like all these things, very similar to a physical factory. I’ve got a package coming from a vendor or a silicon package with software coming. I I have deliverables in essence coming to me, but they’re not governed as they would be governed in the physical world. And so because of that, this integration process is where surprises come. And again, that’s where software

becomes the reality of the integration, becomes the moment in truth when like, for the first time ever, I’m now looking at all the dependencies and I’m seeing all the different things. And that’s where things and surprises happen to come from. So again, we start from the basis of that physical world model and we’re like, there’s should be no reason why it’s not in the software world. We call that the and that’s where the trustable software factor framework basically embeds and embodies that in some of the tenants.

**Kate Holterhoff (06:46)**
Yeah.

Interesting. And can you give any history to that? Like why is it that it’s this moment of integration that we have, tolerated that this please work moment is is is gonna be so important. where does this come from?

**John Ellis (07:09)**
You know, it’s it’s I think it’s it’s a couple different things. Number one is is we have historically f certainly from purchasing, we have brought people over from the physical world who would you would think would have that, but when they came over to the digital world, they they didn’t bring the the model with them. So there’s not the same level of of diligence, I would argue. number two is is is we have some you know, oftentimes it just worked.

Or or there’s been a disconnect to work, meaning, it it it was able to be shipped and then sort of afterwards the recalls or the other points and the in the sort of you know failure thing. We just haven’t applied to the software engineering discipline the same level of rigor that we applied to the physical world of of factory and and and physical engineering.

many of my colleagues at Codethink would argue software engineering is not yet really a discipline. We we haven’t gotten there yet because we we don’t have all of the other things that you would expect to see in some of that engineering discipline in terms of the rigor, in terms of the post-analysis, in terms of the society understanding something failed, we understand it, we make it, we remediate it, we fix it, and we go back. and so i I think it’s a multi-part problem, right? There’s a lot of different stakeholders for the reasons to the why. and I think

what we’re finally seeing is is is at least in some of our our example products, the rigor is now drawing into question because we can see the recalls or the failures. I mean, I think you saw the one slide I talked about, there’s trillions of dollars being accounted for economic loss. Like like we have the data point. And so I think it’s and and again the Eclipse Foundation and the SDV working group specifically is trying to address that. Like they’re acknowledging that. And the Trustable Software Network is sort of a step down that path to try and

be the place where integration is contemplated and understood. Right? One of the tenants, understand what you’re getting, understanding what’s coming to you, right? Is tenent number one. Understand what you’re building. and that is kind of that first step.

**Kate Holterhoff (09:07)**
Okay. And as a former QA engineer, one of the things that I was really attracted to in in this argument is that the the beginning of the SDLC, the beginning of this production or the factory metaphor, it it gets more attention than the end. and that’s where things kind of fall off the rails.

do you feel like the SDLC is is equipped to to deal with these challenges or like how is is your how are your ideas about this framework really challenging the way that software engineers have functioned up until now?

**John Ellis (09:39)**
so the trustful software framework is well, Bosch asked the first question, and then there was a whole generation of there was a period of time where there was an open, like in good software and in good open source form, right? Codethink provoked the question publicly and invited a public discussion about like how can we trust software? How do you get to trust software? Which those those six tenets kind of come as the sort of summary of the

**Kate Holterhoff (09:58)**
Yeah.

**John Ellis (10:04)**
twelve to fourteen months, whatever the duration of the time was behind it. And it in and so doing this, right, there was a lot of questioning about like what does it mean to actually engineer software and how do we do it and and and what was there an well there was an acknowledgement of was like software is accessible more so than so much in the hardware world. Like in the hardware world you need certain things that are not necessarily easily combiable.

And yet we can do in software. Like, mean, you could write a macro in Excel and suddenly you’re doing software, right? How many businesses have built themselves around macros built by people sitting at a dash and they don’t even understand where that comes from, right? I mean, you’re smiling, but like that’s that’s that’s how easy this is. So as we as we talk about what does it really mean, I think there’s we’re differentiating, I mean, more broadly, Codethink would argue, and I think the Eclipse Foundation would argue software engineering writ large, right, needs needs to be thought of.

**Kate Holterhoff (10:47)**
Yeah.

**John Ellis (11:01)**
But when we talk about the the the concept of safety critical or regulated industries, this is where we’re really focused on and and the idea that software under contract isn’t you know, putting it under a confidentiality agreement, putting it under a contract, put making it hidden, making it less accessible, right, it that’s not the right approach. Like exactly a bridge metaphor.

Like if somebody built the bridge and said, you can’t study it, no third-party structural engineer can come in and look at this and do weight measures and analysis, like you can’t do that. I think we’d all as a society balk at that. So why do we accept the alternative, like the opposite side of that on this on the software side? So when you say, like, is the software development lifecycle or the way engineers are used to building software ready for this or whatnot, there are changes, right? There are demands, there are understandings, but it comes down to

being really engineering and and we really do want people to think deeply about what they’re building ’cause again, society is impacted by anything and everything that these product companies release.

**Kate Holterhoff (12:06)**
Yeah, that’s huge. okay. And so just thinking about this in terms of brass tacks, organizations they’re already using like SBOMs, CI Pipelines, testing requirements management, and so my the question that I have then is what is the ETSF connect or add that these practices aren’t already providing?

**John Ellis (12:25)**
It well, I mean i in and that’s a great it’s a great observation, right? We do have a lot of the different things and and we do have them, right? And they do work to whatever limit they are, like the SBOMs and CI/ CD testing and others. And I think what what I was highlighting when I talk about this publicly is they all exist, but in some cases they all exist in either silos or fragmentation. Like we look for an SBOM by the SBOM point. We don’t think of SBOM as an every everyday evidence that’s coming in the course of a pipeline, right? Again, going back to that.

mainstream idea of a physical factory and the walking through a factory and what gets delivered and managed and how do you manage a factory and looking at like throughput and you know failure rates and others, right? there are things to be learned in that way. And so what the TSF, you know, the Eclipse Trustable Software Framework brings is is a way by which to contemplate that in its entirety, reason about it, and then reason about it in its entirety, not just I have an SBOM and the SBOM is SPDX 2.x or 3.what

Or even cyclone DX, right? Not just SPDX, it could be cyclone. We allow for the reasoning, because what we’re trying to get to at the end of the day is we don’t think you can make a blanket statement, this is X. It’s X for a period of time T against some understanding Y of the world. So at some at the end of the day, there has to be a level of confidence. Like we’re confident that this does what we say it does.

against these credentials or against these things and here’s why and here’s the evidence. And so ETSF, as you’re referring to it, right? The Eclipse Trustful Software Framework is intended to be that holistic way by which you can come to make the confidence statement, provide the evidence, and then allow a third party to reason independently and be able to say, Yeah, I’m I’m in agreement or no, I have a different view of confidence, or yeah, I’m good. I’m good and I’m let’s go release this.

**Kate Holterhoff (14:16)**
Okay. Yeah. And I yeah, I’m I’m interested in in how that ties into the way that developers often have this fear of touching software, right? So, in most regulated industries, they’re treating changing shipped software as this terrifying thing, right? it adds this this level of uncertainty to something that they already know works,

but my sense is that that the ETSF allows you to be able to make changes and reason about whether or not they’re actually gonna work in a better way. And so that that can help organizations feel better about making these changes. is that a fair way of framing it? And how does that even tie into like where the regulators sit into this, like how how they’re changing their posture in in relation to the these deployments.

**John Ellis (15:07)**
So I think I mean there’s a couple different maybe threads to pull on on there. I th the and probably the biggest one is the idea that this product, whatever it is, has to be has to ha has to continue to operate in some level of of your statements for fifteen to twenty years. one of the things that I like to when we talk to customers and and clients when I talk publicly is software.

**Kate Holterhoff (15:11)**
Yeah.

**John Ellis (15:32)**
At the base level is just a mathematical or mental model of how I see the world at a point in time. and as I as an engineer or I as a company, you know, spend time, what I see the world today may or may not be the way I see the world in a year or two years or three years. I will have learned more about the world, I will have better understood the world. And so that mental model has something called decay. I know we don’t talk about it, but just again, like the physical world.

We’ve spent so much time trying to figure you know figure out exactly where the tensile strength of steel fail fails when it’s bent a certain way. We understand the incredible, you know, physics and chemistry of everything that we build so that we have an understanding of tolerances and and we can think about them and and we have an understanding of that world. Very similarly, we have the same thing in software. We have created a model about what the world looks like or what we think the world looks like at that point in time.

Model has to change because our understanding of the world has to change. So we there’s a construct that we we have to contemplate changing our model over time. With that, against the 15 to 20 year viewpoint, somehow, some way we have to accommodate the idea that we’re gonna have to make changes. So when you ask, like, well, how does that work, or how do we get to it? We would just offer and argue if you fully understand what you’ve built, if you fully understand where it comes from, if you fully appreciate it, then you should be comfortable.

making changes. You may choose not to make changes, but that’s a that’s a that’s a decision as an engineer that you’re making with all the facts. Not because, I can’t change it because I I don’t know what’s gonna happen or I’m afraid of what’s gonna happen, right? so our tenent three is the ability to make changes and understand, right? And be comfortable to make those changes. Again, when I say that

Please don’t take from we were talking making changes willy-nilly. It’s not like some guy today is just gonna wake up and tomorrow make a change. This is the understanding that I can make changes. I do the deep engineering, I understand it, and I’ve come to an agreement as a community, whatever that community definition is, that yes, this is an appropriate change, it will work, we’ve studied it, we understand it, and we’ve made the change. And we’re not afraid to make the change. Moreover, we’re we’re willing to study the world’s impact as the change propagates and again understand, etcetera.

**Kate Holterhoff (17:25)**
Yeah.

**John Ellis (17:50)**
I always like to point out to people, right, if you look at any of the Voyagers or any of the stuff that NASA and JPL have done, like where they have stuff built from the 70s or 80s out, and they’re making full software changes years later, it just doesn’t happen. Like the engineers come and they deeply think about it. That’s the kind of stuff we’re talking about, and that’s the mindset that we’re trying to get into. Do not be afraid to make changes. That doesn’t mean you do make changes, just do not be afraid to make changes. Because of X. That’s what we’re trying to get across to people.

**Kate Holterhoff (18:21)**
Okay. And my sense is that versioning and git are extremely important to how this will look in practice. Can you talk more about that?

**John Ellis (18:31)**
I it’s it’s it’s versioning and git, but I mean it’s gonna be down to business practices as a company, writ large, right? The ability to actually manage again, this comes down to if I go back to physical factories, right? It is incredible the amount of documentation and detail we have about anything that we received, how we manage it, how we manufacture it, how it comes to like we we have all that. That same level of engineering understanding is what we’re looking for as our sort of North Star goal as we talk about software.

**Kate Holterhoff (18:36)**
Okay.

**John Ellis (18:59)**
So yes, it’s Git, it’s versioning control. It’s not just versioning control, it’s like understanding edits. Did I allow, you know, Kate to make edits independent to did did Kate’s edits get reviewed? Were Kate’s edits, did they touch upon anything safety critical? If they did, what do we do about that? What does that look like? I mean, it’s not just versioning, it’s the entirety of how do I manage this as a thing and how do I understand the, you know, the the management of its changes, its evolution, its life.

And then again, am I still able to say that with X confidence I’m good that this thing will will do what it’s supposed to do and is is gonna behave in the way it’s supposed to behave and I’ve and I have a level of confidence that it won’t harm people or you know or goods.

**Kate Holterhoff (19:45)**
so I I f I feel like to some folks might listen to this and be worried that this is gonna result in higher headcount, less lean, it’ll decrease velocity. they they might be worried about the bottlenecks that this might create. how do you address that fear?

**John Ellis (20:02)**
I mean I I it could could it? I mean maybe, maybe. but I think that’s a bubble. Like it’s at the beginning part. If you if you know, in our experience with customers, once they’ve gotten a factory model, once they’ve adopted the idea that this factory is no different than the physical factory that they run to build their product, it has the same governance, the same discipline, the same things. They get that and once they have the necessary governance in place.

Then you can start optimizing throughput for bills, others, et cetera. It’s only once you have that in place can you then have the ability to start figuring out how to optimize where necessary for you, right? At the end of the day, the Eclipse Trustable Software Framework leaves a lot of latitude to what does it mean to be trustable. Right? It’s really up to the company in question to start saying, like, here’s my evidence for the claim I’ve made, and here’s why I think this is good. Like now others can look at it.

But once you have the instrumentation and the machinery in place, now you can start thinking about, well, how do I optimize? Do I have development in a specific region? Now how do I start changing things? How do I start optimizing my the contracts that I’m entering into? How do I optimize again? One of the things that we we we look you know deeply on is how do I optimize, you know, again software sovereignty? Do I own my own software? Or am I or am I just a massive integrator at Scale and I’m just getting all the software from third parties and shipping it and putting a label on it, right?

These are the questions that come up naturally, commercially. Once you’ve answered them and you have the instrumentation, our our our experience is you can get speed, but you have to start from the concept of this structure. You just can’t you just can’t try for speed for speed’s sake.

**Kate Holterhoff (21:45)**
Mm-hmm.

That’s reasonable. I I want to return to the bridge metaphor because I’m thinking through the sort of positive things that this enables versus I guess the negative and the absence. So it’s very clear with a bridge. You either sleep under the bridge or you don’t. With software, it’s how are you testing this negative? Like the how are you determining an absence of bad behavior here?

it it just seems like it’s it’s hard to check for the absence of a happy path, right? So, like, how do you actually generate continuous evidence for this negative, this untrustworthiness?

**John Ellis (22:23)**
Yeah, so it’s it’s it’s a it’s a great question when we we get often and and for for us specifically we we’ll talk to you a little bit about so we I wanna be very clear, TSF does have as as part of like some of the expectations. So, you know, tenents four and five, the expectations and the evidence.

We actually do employ a risk analysis. Like it’s not like just a bunch of people sitting in a room. Like there is an actual formal risk analysis. And in the case of of the Trustful Software Framework, what we actually came through was it’s predicated on STPA, systemic theoretic process analysis, which came out of MIT, Nancy Levinson and John Thomas. and was originally designed for or originally envisioned for aviation and the idea that you don’t get safety by plugging two safe things together. You have to look at things in a in a in a very in a in a much higher level.

was what the the premise of the of the approach was. through working with MIT and and and this, you know, Codethink came to understand that it wasn’t necessarily tuned exactly for software. And so we created something called RAFIA, also open source, but the Risk Analysis Fault Induction Automation. So the risk analysis is all the SDPA. You do your loss scenarios, you understand and try and for the context that you’re talking about, you try and identify what can be done or what can’t be done, i.e. the loss scenarios.

Our insight in terms of the fault induction is that well, it’s software. So we can actually either write, change, modify to induce the fault that we think shouldn’t happen in the world and see how the system performs and validate that the system holds true in the case that that negative actually showed up. So we actually do that, and in so doing it, we’ve uncovered problems.

For any package that we’ve looked at, where again they’re corner cases or others where we’ve made changes and upstreamed them. Once we understand it, we can then automate the test to validate that di is does that does that still hold true over time? And so again, the STPA is where we do all the analysis and come up with the ideas. The fault induction is where we actually take advantage of the fact that it is software and we can we can force the changes that we think shouldn’t happen to happen, and then we can see how the system performs.

Then we automate it and then we just run that every time. And as as we like to say, if it squeals, then we know something’s wrong. We have to stop, we have to go look at it and understand. But again, if it doesn’t squeal, we’re we’re able to continually keep that. And so yes, that’s where the testing and the body of testing increases over time. Every single time we find where there might be a loss and we’ve forced it to happen and proved that it shouldn’t, or we’ve proved how the system behaves.

**Kate Holterhoff (25:01)**
So I and what interests me about that is, there’s there’s such a I guess a binary between this passfail mindset that you think of with like regulators and assessors versus this statistically driven like, you know, we are confident to X percent that this complex software i i is gonna be trustworthy.

**John Ellis (25:13)**
Yep.

**Kate Holterhoff (25:23)**
How do you get regulators and assessors on board with this understanding that’s a little bit more nuanced and a little more, effective in the real world?

**John Ellis (25:33)**
similarly to how like so you’re right, historically their view of the world has been in the physical world, it’s all about confidence testing. It’s all about statistical analysis and failures and understanding failure rate, which is why you get the hours of use or the exceptional sort of testing rigor that takes you to the negative exponent of of of time, right? And so part of our journey has been to raise the awareness that software should be treated no differently. Like the st

Physical confidence is something necessary. And in fact, code think’s not unique in this. There’s a lot of embedded developers who are putting forward that the idea of claims should be Bayesian based, not just analog, true, false based, because it’s only true for 85%, but you’re in an analog, then it’s false. And then the engineer’s like, well, but wait a minute. Like 85%, like so this is the exact same world that we live in in the physical world.

**Kate Holterhoff (26:15)**
Mm-hmm.

**John Ellis (26:28)**
You’re right, it’s a change. That is a mindset change. And it’s a mindset change that, especially in the embedded world, we think is is overdue simply for the fact that the complexity of the software we’re talking about. Even if the software was written by hand, the complexity of what we’re talking about in terms of the world is far greater than I think the idea of just binary true-false allows for. And so we know how to do it. Statistical process controls in in the real world. We understand how to handle it and accommodate it.

And it turns out even some of the standards bodies are beginning to acknowledge in software that there might be a need for some statistical understanding. Not there yet, but there’s at least a growing conversation broadly that’s just not just about the TSF. And not even because we necessarily said it. We’re just amplifying that message.

**Kate Holterhoff (27:15)**
Well, I mean, that certainly seems to resonate with what we’re hearing in our AI era, where there’s this whole new breed of chained vulnerabilities and we’re sort of in this new era about what the CVE landscape looks like. So sort of applying that to what you’re is talking about here, we’re seeing the need for changing our priors, right? That we we are all adapting to know, make sure that we’re we’re creating the best software possible.

in this new era. and a lot of it’s attributable to AI, but a lot of it’s been here forever, right? This is this is a historical situation that is perhaps shifting but isn’t going anywhere. and as maybe part of that, I I want to talk about the black box nature of certifications historically. So traditional certifications, they’re kind of opaque for folks on the outside. So I’m interested in,

how you’re grappling with this sense that teams are unsure what’s actually happening behind the scenes with the NDAs, the the fact that they’re just not entirely sure what what these assessors are looking at. can you speak to that at all about like trying to break through that veil?

**John Ellis (28:19)**
yeah, sure. So bit as just background, right? So Code Think has gone through a number of different certification processes. We’ve actually achieved an ASIL D 26262 for some of our building construction. we’ve actually gotten, you know, the TSF itself certified against 61508, and we’ve gotten our version of c of control, which is Linux, right, to a baseline assessment through 615. So we’re we’re familiar with the process.

And I only give that as context to to talk about then sort of like why and how do we take our position in in terms of trying to, as you said, break through. I think the first the first thing to acknowledge is the the idea of a safety certificate, I think itself is a misnomer. There is a certificate from an organization, AK Exida or TÜV or UL, against some standard. But if you actually read the certificate, at no point is the word safe present.

**Kate Holterhoff (28:46)**
Mm-hmm.

**John Ellis (29:18)**
What is present are words like, you know, we have seen, you know, ample evidence of the company in question following process for this product in question, right? We we’ve seen and therefore they have received this certificate, right? We don’t see the idea of the third-party intermediary saying, we the company hereby attest that this product is safe based on XYZ. So and this is all with respect to software, right? I’m talking about

software and so we find that super interesting. and when we’ve pushed on it it’s like, yeah, because at the end of the day, right, they’re not making a claim of safety. What they’re making a claim of is a process was followed, and then it’s up to the industry to recognize whether or not they accept that process as ergo being something that’s safe. Why that’s interesting is because that ties back to your earlier question of like statistical process control. The idea of it’s if it’s if you build it by hand and it’s binary and all these things are controlled and it’s contained and it never changes.

We can make some statement about something and so yeah, if you follow the process, we’ll equate that to safe. In the complex world we live in, we know that that is not really the case. And I might offer as evidence is, you know, as I said to everyone, if if if if it were the case that it was right, then in theory we shouldn’t see what we’re seeing in the marketplace, which is the idea of a safety recall, a recall from the United States government, is a safety issue.

And most and increasingly these recalls are software related. So if in fact it were working, we shouldn’t see an increase in recalls. We should in fact see a decrease in recalls because we’re actually doing the right things. So what we offer as evidence is there is something about the complexity nature of software. There’s something about the integration. There’s something about everything that we’re talking about that demands us as engineers to really deeply study this and understand how it is that we construct, build, and deploy this software.

ergo why we go back to traditional methods I think are gonna they’re they’re short in terms of being able to actually have somebody sleep on that road while you know, a multi-thousand pound vehicle at 80 miles an hour is traveling down the lane and we expect it to stop or avoid the person on the road. So that’s where we would put our our our our our thought process at.

**Kate Holterhoff (31:35)**
Do you suspect that this will have some upstream ramifications, this being the the framework and trustable software. do do you see that the assessors are then going to need to shift their way of addressing the industry?

**John Ellis (31:51)**
I I don’t think it’s just assessors. I think it’s assessors, regulators, society. I mean, yeah, yeah. I think all of us, all of us need to shift how how we manage, how we deal with, and how we understand and deploy, right, this this level of software. and again, the data points in the marketplace, whether it’s the increase in s in safety recalls or if it’s just the losses economically, right, suggest that what we’re doing isn’t sufficient.

**Kate Holterhoff (31:57)**
Okay.

**John Ellis (32:16)**
And again, we would offer the Eclipse Trustable Software Framework in in in support of a change. We’re not suggesting that it is the only change, but we are suggesting it is it is a piece of a change. and again, offering to stand by it as part of like how do you come to be able to say the word? I’m I’m confident or I can trust that what I’m doing, you know, won’t harm.

**Kate Holterhoff (32:36)**
with so many stakeholders involved, I suspect many of our listeners are going to be you know, are gonna have their skeptics hat on of like, is this a foxes guarding the hen house situation? Like who’s really making sure that this is something that we need to all be following here. So I just wanna make sure that we’re addressing those those questions.

**John Ellis (32:52)**
But okay, so to just add the plus one color to what you just said, like our suggesting for a moment that you know that the the ETSF by itself is the is the tool. Like there is still somebody or some entity that has to say I’m confident. But what we what we are suggesting is this framework then allows for you, the recipient of my statement of confidence, to actually independently look at that and look and understand it and study it and be able to come to your own conclusions.

**Kate Holterhoff (32:56)**
Yeah.

**John Ellis (33:19)**
Was the evidence provided sufficient or is it sufficient? Is it sufficient for the statement of confidence that they’re giving us? And if it’s not, to challenge them. Like, I’m sorry, but that evidence is insufficient for your strong confidence level. You’re missing the following, right? It just it allows finally for the ability, right, for that structural engineer on the outside looking at the bridge to say, independent of you sleeping underneath it, I have the issues with how it was built and how it’s supposed to work. How society is supposed to accept this into into the lexicon of of things that society depend on?

**Kate Holterhoff (33:52)**
All very reasonable expectations here. I I I like that. As someone who could potentially drive over a bridge with my kids in the back seat, I do want that. yeah, so this is this is all for the good here. so thinking of regulators and folks who who might be looking into the the trust issue, let’s talk about the CRA so we at RedMonk have spoken with the Eclipse folks at some length about the importance of this. I feel like

**John Ellis (34:00)**
Absolutely.

Yeah.

**Kate Holterhoff (34:18)**
you folks have really taken a leadership position in, trying to express to to the industry why this is so important. And and so the CRA being the EU Cyber Resilience Act. and so the consistent signal that we are hearing is that businesses are unprepared. And I I’m interested in your read on the CRA readiness issue across Europe today. What is driving that awareness gap?

**John Ellis (34:47)**
well I mean so first off, as you can tell by my accent, I’m not European and I don’t I don’t live in Europe. so anything I’m gonna share with you is really a a a recital of what comes from my time with Eclipse. so I think broadly, regardless of Europe and all those disclaimers I just made, I it is my as I say in most of my conversations when I talk about the fact that so much of our world is governed by software.

**Kate Holterhoff (34:53)**
Yeah, sure.

Okay.

**John Ellis (35:14)**
It still to this day shocks me at to just the gross misunderstanding or lack of understanding that sit at board levels, at executive teams, for what software really is. You know, as as it’s super interesting when I when I get asked questions and I always point to this, right? If you want to be a board member or if you want to be someone a CEO or an executive or a COO, you better understand the factory. You better understand supply chain and finances. Nowhere is it, you better understand software.

Or you better understand how software is built, even if their product is so dependent on it. So when we talk about the CRA gap, in part it’s because these executives don’t fully appreciate just how much software, right, is in or part of what they do, number one. I think it it it’s also the fact that I don’t know how much in the US, I’ll just being through the lens of the US, did we talk about the CRA? Like I can remember when

GDPR was coming up, it was a brand new novel concept. But guess what? Lots of law departments were all over our stuff raising this awareness. When I was back, you know, working in a in a in a a for-profit publicly traded company, the lawyers were all over us like this is a thing and we’re gonna have to comply with it. And I don’t know that that’s actually happening at the same level because I don’t know that there’s an awareness of that necessary requirement. And again, unfortunately, unless the attorneys are telling you you have to do it.

Sometimes that gap exists. So I think there’s an awareness at the level of the executives. I think there’s also an awareness broadly. That said, once you go through it and you start pointing these things out, I’ve been in meetings, I’ve been in sessions where, you know, we’ll go through the the key points. So when we first start, how many believe it covers them? You know, no hands up. By the end of the session, how many believe it covers you? And all the hands are up. And you’re like, okay. So now what?

**Kate Holterhoff (37:01)**
Wow.

**John Ellis (37:04)**
It’s it’s a journey and I think it’s gonna be a journey just like the GDPR. I mean to this day we’re still kind of fighting through key points of the GDPR. Like what’s really the rights in all this? It’s more settled than it was, but that’s almost fifteen years, right? Roughly, ten, ten, twelve, thirteen years. I it’s been a long time. I suspect the same journey will happen for the CRA.

**Kate Holterhoff (37:24)**
Man, so my takeaway from what you just said is that yeah, the awareness is still lacking, but also that this is not gonna happen anytime soon. I mean what I keep hearing is like it’s coming, it’s coming, it’s coming. But even when it’s here, we’re talking years in the future we’re still gonna be grappling with what that actually means my goodness.

**John Ellis (37:41)**
And I think so because at the end of the day, there’s no as as we point out, right? There is the d I think the what is really well defined. We can argue how it’s defined, like who’s a who’s a who’s a who’s a who’s a a vendor and who’s a who’s materially responsible. We can argue the definitions, right? That’s the what. And it’s it’s obviously codified and it’s been made law and we can debate that, but it’s it’s codified. So I think we could argue that the what is defined well.

What’s not defined or at least not defined well is how do you actually show when up when asked compliance or that you are in in in in in in line with, right? Whatever the word right is. I don’t know if it’s a compli well w how do you do that? Like and and sure, there are things now that are that’s where I think gaps are, and that’s kind of why we are positioning the ETSF as the way to show the how. Like we are arguing that the evidence required to support

The know what you’re building tenent one, being able to build reproducibly tenent two, being able to make changes and and understand them, right? Tenents one the evidence just behind those would be strongly sufficient, we argue, for what the commission’s looking for when they’ve laid out the CRA what. Again, that’s still to be debated and it still has to be tested. but that’s what we think is missing in the CRA is the how. And therefore, until there’s a real definition of how.

I think it’s gonna be time until people understand and settle on okay, we’ve all agreed, this is a how, they’ve agreed and and we’re now showing you the how on a regular basis or whatever the requirement is.

**Kate Holterhoff (39:16)**
Okay. I’m glad we did this check-in here, because I think it was last fall that we we published our episode on the CRA with Mike Milinkovich. So this is this is all good. We’re we’re checking in. okay. I wanna return to a subject I alluded to briefly, which is AI. so a growing share of code is written by AI now, unquestionably. and that means that the provenance issue is becoming more murky, right?

**John Ellis (39:27)**
Yeah. Yeah.

Mm.

**Kate Holterhoff (39:45)**
interested in how you factor into that. Like you’ve got this evidence based model. seems like it’s becoming more important now. and I guess I’m just I I would like to have your sense of like, can a framework accommodate the code whose origins are so difficult to trace?

**John Ellis (40:02)**
so to start, I would argue that like the framework just gives you a framework for making sure that you’ve answered the question on providence. Like the definition of acceptability of providence is ultimately the company in question or the entity in question who’s making the confidence claim. So, you know, and that’s where again the ability to reason about this. So if I show up to you, Kate, and say, Here’s my confidence, you look at provenance like you didn’t tell me where the f you didn’t tell me where that stuff came from. Like how can you make that claim? That’s you can only do that.

**Kate Holterhoff (40:10)**
Okay.

Sure.

**John Ellis (40:30)**
In light of the framework giving you the tools and the in the quote unquote framework by which to analyze, study, and challenge me, right? We don’t actually make a claim for like what is pro like provenance is a framework. You still have to do the thing necessary for a company to but we give you all the tools, we give you guidelines, and we show you this. We what’s interesting for us at Codethink is we are using this to take our Linux distribution through the actual same process that we’re expecting anyone else to take in an open source.

**Kate Holterhoff (40:37)**
Mm-hmm.

**John Ellis (41:00)**
Or even a closed source project through, right? We’re not we’re not sitting here pontificating on high. We’re actually trying to do the actual work necessary. And so as I mentioned earlier, like we actually have a baseline safety assessment that says, yes, yeah, you you can actually use this Linux in an engineered system up to SIL 3, ASIL D. Like it said that. The the ac you know the the assessor came back. It’s a properly engineered system, right? It’s not just generic. So when you ask me about the provenance, basically our answer for AI is

absent any ethical concerns that we might have or anyone might have, it’s very easy to say, is did did you check to the was did you do you have a claim or a requirement that who gave you this, who is delivering this to you, attest it or made a claim for is there AI? That’s like an early step. And if there is, you make the decision, are you comfortable with that? Like you have to make the at the end of the day, the company still has an obligation as engineers. Are you comfortable that AI was used?

and if you are and you’re confident, then you give me your evidence for that confidence level and and we can now talk about that. in terms of us personally, if we find AI involved, like we what we guide our clients and customers to is like, listen, the framework allows for you to provide a lower score or a lower trust or a lower factor. So if if we say like we don’t know where this provenance is, it’s AI, or for example, w a a vendor only gives me a binary, like they will not.

ever give me the actual source, they only give me a binary. We would also offer and argue that that’s lower trust. You you can’t again I can’t inspect it. I can only make a trust claim that commercial rules allow me to make. AKA I asked for this. They gave me this, they say it’s the SHA, I can verify the SHA, but that’s it. Is that enough trust to put the vehicle on the road? I I don’t know. That’s up to you as a company, right? But the framework gives you finally, we believe, the ability to ask the questions.

challenge the responses and then come to independent conclusions and then debate it societally. so we don’t make a claim right or wrong about whether AI should or shouldn’t be used. We just say that the framework allows for you to determine if it’s used how do you assess its ability from a confidence standpoint in what you’re shipping.

**Kate Holterhoff (43:14)**
fair enough. okay. Well as we think about winding down our conversation here, I do wanna ask you about the roadmap because I suspect that the ETSF is not a static thing that you know, this is this is shifting, this is dynamic. So talk to me about what the project is is contemplating next.

**John Ellis (43:34)**
well, I mean there’s a there’s a couple of different elements. one is we b you know, Codethink as as one of the contributors and one of the one of the users of of the framework demonstrated the ability to take a third party right or to take a standard, aka in this case 61508 and map it TSF and then show how we could take a thing against the TSF and be able to satisfy 61508.

We would want and look to see any number of those kinds of mappings happening. We believe CRA is one that’s that’s rich and we’re working through that. we’re also working, you know, internally with a number of the different stakeholders from different companies, right, to take the Eclipse development process and to also make it part of like a TSF. So generally speaking, anyone can say, I want to be assessed against TSF, and they can ask for what the assessment should be focused on. 61508, Eclipse Trustable Framework, 26262.

thirty one name your favorite standard, right? And the mappings exist and suddenly you’re able to prove this. And so what that does is it in essence simplifies for development engineers the ability to say like what am I needing to worry about? Because unfortunately one of the other outcomes of our current approach is all the standards are proprietary. And you need to need a license.

And the licenses, generally speaking, are to a single person. So UK can get a sixty two or twenty six, twenty six two license, but it’s only to you. You can’t share it with your colleagues or your friends or anyone else. Regardless of its expensiveness, that doesn’t that doesn’t foster the kinds of engineering conversation that we think needs to happen in a world of such criticality of the software in question.

So being able to say, yes, you’re reasoning at the TSF level, you’re debating at the TSF level, you’re debating in open, you’re debating at a societal level, right, is where we’re that’s one of the big roadmap items to try and get to. and we started, you know, Codethink was able to contribute, we did, under an early release, our 61508 mapping that got us to where we were with our assessments. one of the next ones up is gonna be the CRA. You know, the Clips group is trying to figure out how best to do that. and then we’re open to anyone else.

you know, coming along and saying, I’ve got another standard that I want to be able to look at and okay, great, how do we do that?

**Kate Holterhoff (45:46)**
that is a full plate. Sounds like you got your work cut out for you. I I look forward to seeing where this goes. okay, so before I sign us off here, John, where are you directing folks who want more information? Where should where should they look?

**John Ellis (46:01)**
in short, it’s the Eclipse Trustable Software Framework project, which exists inside of the Eclipse organization.

**Kate Holterhoff (46:06)**
I will include a link in the show notes here to that so that folks can follow up. I have really enjoyed speaking with you today, John. This has been an absolute pleasure. and again, my name is Kate Holterhoff. I’m a senior analyst at RedMonk. If you enjoyed this conversation, please like, subscribe, and review the MonkCast on your podcast platform of choice. If you’re watching us on RedMonk’s YouTube channel, please like, subscribe, and engage with us in the comments.