Ep 26 – AI Security Guidelines

Paul Lacey: All right. Hello everybody. Welcome back to the show. This is The Data Aware podcast by Ascend, the podcast about all things related to data engineering. I’m Paul Lacey, your host, and I’m joined by Sean Knapp, the CEO and founder of Ascend. Sean, welcome. 

Sean Knapp: Thanks, Paul. Hey everybody. 

Paul Lacey: Hey Sean. Good to have you with us.

So we have a really topical thing to jump into this week, Sean, which is around AI safety and regulations. There was a new set of guidelines that came out recently endorsed by, I think it was 18 different governments, and tons and tons of companies in the Valley that are working on AI systems, guidelines About things you should think about with regards to security of AI systems, that may be over and above what you would do for traditional software systems because of the differences of AI systems: how they work, how they’re so data driven and data based, and that introduces new ways in which people can try to compromise those systems, and also new risks of those systems malfunctioning.

And so it’s a quite interesting topic, Sean, to talk through, and I know that there’s been a lot of stuff happening in this space. And you talk to people all the time, Sean. You talk to investors, you talk to executives that are trying to build these things. You talk to engineers, including internal resources here at Ascend that are building AI based systems.

What’s your take on this whole emerging landscape of security around AI? 

Sean Knapp: Yeah, I think it’s a really interesting set up. So if we zoom out a little bit and look at the backdrop first, it’s this perfect storm, if you will, where you take it into any space, introduce incredibly powerful new technology, generative AI, that is, I would contend, not super well understood.

In many ways, it’s magical to many of the would-be adopters. And we see that, in many ways, because even the problems that people want to apply it to, they’re like the South Park underpants gnomes: step one collect underpants, step three, profit. And they’re like, I don’t know, I’m just going to throw Gen AI at my really hard problems, and I don’t know how it’s going to solve it, but Gen AI. And so it’s not a well understood technology.

But there’s so much excitement around what it can do. And first, incredibly powerful technology. Second, not super well understood. Third, the pace at which everything is moving is so high. Things are going so fast that it’s actually creating a little bit of frantic behavior too. So now there’s a sense of urgency that we’ve not seen around adoption technologies.

We didn’t see the sense of urgency in moving to the cloud. We didn’t see the sense of urgency in adopting a data lake strategy. But we’re seeing an incredible sense of urgency for people to adopt Generative AI strategies. That urgency generally means a lack of time to really slow down and think through not just can you do something, but should you do something? And so that introduces more risk. 

And so when we zoom out, it’s this perfect storm of, oh my gosh, magical, new, superpower. Don’t know what it is, but I got to get there before my competitors get there. And I got to do something. It runs a lot of inherent risk. So I think a number of folks who have been working with AI technologies for a prolonged period of time actually are trying to publish a lot more of their thoughts and their guidelines and their recommendations for the industry, to help people navigate what is going to be a chaotic and very fast paced part of their journey, So at least have some careful thoughts around what things should they do, not just can they do . So that’s why, this is how I look at it. The people who have been at this AI game for decades get to share some of their thoughts, guidelines, and wisdom to benefit the broader ecosystem, which I think is very helpful.

Paul Lacey: And it is one of the things that they called out, I think. The report uses the typical four phases of a software development life cycle where you’ve got the design, you’ve got the development, you’ve got the deployment, you’ve got the operation, and the maintenance of these models and production.

And they called out, in the development part of their recommendations, the fact that technical debt is going to be higher in AI systems than traditional systems. Because they’re so new and people are moving so quickly with them it’s gonna naturally happen that there aren’t a lot of best practices out there in the industry. Everyone’s making it up as they go. And so as a result, you’re gonna have a lot more technical debt hanging around in these AI systems. And if people don’t take the time to recognize that and acknowledge that, hey, we’re probably going to have to refactor everything at some point, and if we don’t, that’s a major security risk, then you could wind up in trouble. 

Sean Knapp: Yeah, totally. If you think about these systems, all software has bugs, all software has holes and vulnerabilities. And if you have a piece of software and you haven’t found any of them yet, it’s just that you haven’t had enough time or hardening or, you know, at bats to find those bugs, vulnerabilities and debt.

And so again, against this backdrop of, incredible new superpower, so amazing, everybody, we gotta go fast, fast, fast. What we’re going to see then is these teams are going to not follow best practices because there are none yet. And they’re just going to sprint as fast as possible to the next impact point, if you will. And there’s going to be a lot of chaos inside of that codebase. And, as we see on a pretty regular basis, there will be bugs and there will be vulnerabilities. And that’s part of the risk, assessing how likely it is to impact your business. And if you expose too much of your business to these vulnerabilities without safeguards that’s pretty scary.

And especially because a lot of Gen AI is still very much a black box for most companies too. It can do some pretty serious damage if you try and all of a sudden propel it into a very influential and very impactful part of your business operation or your product or your technology.

Paul Lacey: That’s absolutely the case. And it is called out that for models that make automated decisions based on their data that’s being fed in, that companies create some sort of a fail safe around them that keeps them from, I don’t know, ordering a bunch of unneeded materials, raw materials, if you’re managing a supply chain within a LLM, for example, right?

Investing all the company’s resources in steel when you only need a little bit of it to try to get ahead of what the LLM sees as a coming shortage, as an example, right? Just things that would, if the system went haywire and malfunction, would cause significant damage to the company or to society with these models that are out there making recommendations and doing things.

Sean Knapp: Yeah. I think of it in many ways, like a crawl, walk, run kind of strategy where, I think back when Tesla first introduced autopilot cars, for example. They didn’t give you a self-driving car with no steering wheel. That would be terrifying. Nobody would be down for that.

But instead they say, Hey, we really have smart cruise control. And hey, now it’s a self-driving, pseudo autopilot-y thing. And you build comfort and confidence in the system as it actually matures at the same time. And, I think if I had to predict, part of the trend that we’re going to see is cautiously wading in, hopefully, building more confidence over the course of time where we go from Gen AI in an advisory capacity, to probably Gen AI in an execution capacity, making decisions.

And we’ve already seen that with some embarrassing… let’s call it blowback, for various companies who have propelled it too fast into the control seat as opposed to, say, in a navigating seat, and making bad decisions. And so that’s what I think gets a little bit scary. I have a hunch what happens in that wave after is what you described, which is AI in a driving capacity, but with guardrails.

And I have a hunch that what comes after that is probably compounded AI models where we actually get a Gen AI model auditing the decisions of another AI model and actually having them be fundamentally different AI models, perhaps with different goals and different parameters, and probably still with some guardrails.

In many ways, I think we might start to see that take hold. So you have independent systems with checks and balances on each other. Just as we do, with humans in real life scenarios, which is human in the loop for decision with some escalation path, with some approval process for another human to make the judgment call as we make these critical decisions and execute on them.

So I think we’ll start to get into that mode at some point too. 

Paul Lacey: And that’s a really interesting take on almost an adversarial design of Gen AI, right? But where, you’re literally trying to slow down the decision making process. And in the case of Gen AI, it might mean slowing things down from microseconds to, to, milliseconds of getting opinions of other models.

But basically, deliberately adding bureaucracy to it in order to provide it an additional level of safety. 

Sean Knapp: Yeah. Yeah. In many ways we could take a scenario, for example, claims approval, hypothetically speaking. And having agents, multiple agents actually contend and contest and debate on merits and probability, where one is on the approval process, one is on an auditory stance, and a third is on the patient advocacy stance.

And as you go through, as you hit certain thresholds, if you’re training different models to take different perspectives and different views, actually allowing them to explore as part of that evaluation itself, see how that should fundamentally play out to be really interesting.

Paul Lacey: Yeah, absolutely.

There’s also, in this report, a number of new attack vectors that the authors call out against LLMs. And they call out things like the inputs to LLMs can be attack vectors, which we’ve known about in software for a long time. Sean, you probably have written quite a bit of code to sanitize inputs on the web and look out for SQL injection attacks and things like that.

But what they’re finding is that, inputs coming from users into natural language models, there’s all kinds of new ways of, via prompt engineering, finding ways of getting the model to, for example, cough up sensitive training data and things like that, which is really interesting to see come out.

And then the other thing that, that they mentioned was it’s now possible for attackers to use a vector that they call data poisoning, which is where someone can actually break in and seed bad data, malicious data, into the training data set, in order to bias a model towards a behavior that they can then go and exploit during runtime.

A very insidious attack because a lot of people probably wouldn’t be anticipating someone would be trying to corrupt their training data. Because training data typically comes from internal sources or, governed by third party sources. But what does this mean, Sean? Because I know we think about things in the world of data movement and data processing and things like that on the back end.

What does this mean to like a data engineer when you start hearing about stuff like this. 

Sean Knapp: Yeah, it becomes very scary in many ways because LLMs are built on top of incredibly large volumes of data. And it’s hard to trace back all the way to a particular decision or outcome that was suggested because of a particular input.

Whereas, we’ve dealt with this for a long time in the software space, where I install one package of software here and it actually has a dependency chain where it installs 300 other pieces of software. And there’s, five steps down the chain that actually is a malicious piece of software.

And so we know how to do things like scan our Docker images or our runtime environments for those malicious packages and are pretty good at it as an industry, I would contend. But we’ve had, gosh, a decade plus to solve that problem. And when we think about it from a LLM perspective, we still have no fundamental ability to audit what the inputs were.

And it, again, going back to the comment of most LLMs are black boxes. They’re just delivered. And there’s all this stuff that goes into that LLM, created it, that we actually don’t have a lot of knowledge as to what went in there. And so in many ways, this is where, the fingerprinting and providing a SHA, of the LLM, the model that actually is producing outcomes as a trusted, vetted, safe system. I think, for us, that’s going to become incredibly important for pipeline developers, to know if whatever that foundation you’re building on top of is busted. Because if you can’t trust that foundation, everything else falls apart. 

It’s like, imagine if you had an EC2 server that every time you launch an EC2 server, but you didn’t have a foundation of trust that everything you wrote to disk was actually kept private for you.

The whole world would fall apart at that point. You couldn’t write your software and run in the cloud. And so there’s some of those fundamental guarantees we need to have in these building blocks, if we trust them as fundamental building blocks we architect our systems on. I think we need those kinds of guarantees.

And then, similarly, if we’re adding and augmenting or enriching LLMs, we need to then, as we put in new data into the ecosystem, we similarly need to be able to filter and validate that it is actually trusted data, not malicious data. I think that’s going to be a hard thing. That may end up being other LLMs and classifiers that analyze the data before it’s allowed into the ecosystem, to solve for that.

I think that’s going to be a really critical thing we have to solve. 

Paul Lacey: Who knew that metadata could become even more important than it already is? But I think that’s exactly what you’re saying, Sean, is that metadata becomes a security consideration going forward. Yeah, absolutely. Yeah that’s, it’s amazing to imagine, how this changes the game in a lot of ways for a lot of people and accelerates some of the things that we’re already doing.

And in terms of security, governance, control, control over pipelines becomes even more important than before, because now we can imagine that somebody could alter a pipeline to significantly alter a dataset. They could be able to corrupt the data without even injecting bad data.

They could just inject bad transformation logic onto the dataset and they could achieve the same ends. Correct? So you know, there’s so many other things that we have to consider as an industry as a result of this, which is going to be a fascinating thing to watch Sean. 

We’ll keep an eye on this over the next coming months and years and talk about more specific examples as we start to talk about the design of these systems going forward and the use of data in AI and LLM type applications. And that’s what this program is all about. So yeah, so even though we could continue talking about this for hours, Sean, I think we have to cap it there and let folks get about their days.

But yeah, it’s super interesting. And I also encourage folks that are listening to, join the conversation and reach out to us on LinkedIn. If you’ve got interesting perspectives that you’d like to add, or questions, or things that come up as you listen to these conversations.

Because there is a lot happening in the industry, and we’re all learning together. So we would love to bring other voices in on this and have this be more of a community sourced conversation. More to come on this for sure.

But Sean, thank you so much for your time. Thank you for your insights today. They’ve been fantastic, as always. And hope you have a great rest of your week. 

Sean Knapp: My pleasure. You too. Thanks, Paul. 

Paul Lacey: Thank you. And thanks to all the listeners. We’ll see you next time. Cheers.