Governing AI That Keeps Evolving With Maryam Ashoori

[00:00:00]

Introduction & Guest Welcome

[00:00:06] Krishna Gade: Welcome to our AI Explained. today we have a very special guest, uh, coming on our webinar, uh, Maryam Ashoori. Uh, she's the VP of Product and Engineering at IBM Watsonx Governance, where, uh, she's spearheads product delivery and strategy. And prior to that, she led engineering teams at Lyft and, and, uh, spent, you know, many years at IBM Research and, and worked a lot in AI and quantum computing, holds a PhD in systems design and engineering at Waterloo. So we're very excited to bring, uh, a veteran in AI and an expert. Uh, Maryam, welcome to the podcast.

[00:00:46] Maryam Ashoori: Krishna, thanks for having me

[00:00:48] Krishna Gade: Absolutely. So Maryam, um, you've, you've probably seen AI from enterprise AI, from traditional machine learning to generative AI, and now agentic systems. have you seen the, like, the fundamental changes each time, you know, in these transformations?

[00:01:05] Maryam Ashoori: Even prior to that, Krishna, funny things, 20 years ago I did multi-agent systems, like two master's degree, two dissertation working on MAS well before they were under spotlight by

[00:01:21] LLM. So we've seen it all from traditional, like with agents being on the papers to generative AI to agentic AI. But you know what's interesting? The lesson that I've learned over the years is not to follow technology, follow the trends that solve problems. And when you look into enterprise challenges, the core to those challenges haven't changed much, like understanding the risk, governance.

[00:01:46] Like GRC, governance, risk, and compliance, for 20 years companies have been doing it. The nature of risks changes, but the foundation doesn't change. And I think, um, there are a lot of excitement on where it's going, but also lots of uncertainty along with the foundations that are not uncertain that much that we can potentially go and look into and focus and overcome a lot of, um, challenges that I keep observing in the market for the enterprise

[00:02:21] Krishna Gade: Yeah, absolutely. So, uh, you know, the fundamental difference that we have observed is, of course, models make predictions. You know, large language models came about and sort of saw all that content generation. agents are taking actions, right? So how does, how did, like, trust and governance change across all of these different phases?

[00:02:42] You know, like what, what have you seen in terms of that?

AI Governance Foundations: Visibility, Control & Accountability

[00:02:45] Maryam Ashoori: Um, back to foundations. The foundations doesn't change much. Like, when I think about what makes trust, um, it's really the three layers that I've been thinking about and talking about. The first layer is visibility. Like, you need to know what you're doing, right? Um, in terms of AI assets, I keep hearing from some of the customers, like, the term shadow AI.

[00:03:10] The developers are building at a faster rate that the organizations can govern them or even understand the risk associated to them. That's why the question of the can I trust that workload? Maybe not, because I don't know what that workload is, right? So the first layer is visibility, which is the foundation to assess your exposure.

[00:03:33] The second one is control. Trust is all about control, understanding what are the risks associated, what are the constraints that I have, and do I have proper guardrails to control that, right? Any, any, uh, risk is mitigated against a control. A control is measured by a metric. Now you have a way to go and actually see if you have proper tooling in place to monitor those metrics, then you have the control, right?

[00:04:00] And the last piece is, um, accountability. Control without accountability is just good intention, right? We need to make sure that there is enforcement. You have the control to enforce them, and you have an option to pull back information to make sure those controls are working. Once you have these three elements together, visibility, control, and accountability, that's really the foundations to what makes people to trust that the system is working

[00:04:33] Krishna Gade: So why do you think the enterprise adoption... I know it has picked up in, in the agentic era compared to the model era quite a bit, but it's still as much, right? Like, you know, why do you think like, you know, companies are struggling, you know, to kind of put a lot of this stuff into production quickly and, know, where are there some of the challenges in your sort of, uh, you know, experience working with them?

[00:04:56] Maryam Ashoori: Yeah.

Why Enterprise AI Adoption Is Stalling

[00:04:57] Maryam Ashoori: I'd like to take you on a very quick journey of the past three years. At the beginning and after ChatGPT. When ChatGPT happened, we saw a rush on doing something with genAI. Like, I, I got many CIOs, CEOs coming to us, and they got a mandate from board to just do something with genAI. Very soon we saw that the applications and the use cases were focused on a very niche area around a series of well-defined use cases like information extraction, summarization, content-grounded question and answering.

[00:05:33] Later in the year, code generation was added to that. Um, so if you had a scenario that fit that, you're golden. If not, okay, what problem does it solve for me? So at the second year after, um, ChatGPT, we started seeing companies are struggling to see the value of ROI because they thought they could just apply that to every single scenarios, and this gives them efficiencies, which wasn't.

[00:06:01] So they were struggling. And then mid-year 2024, we started seeing LLMs taking action, as known as agents at the time. The market got super excited about this because they perceived this as an opportunity to go and apply that niche set of use cases, the productivity and acceleration that you get here to every single part of your organization, including legacy systems through API calling.

[00:06:31] And that's why, like, suddenly we saw a lot of excitement around, oh, agentic AI, automation, it can come and solve my problem. And again, boards now are asking about having a couple agents in production by the end of the year. This is 2024. Which my question was for those execs was, to do what exactly? Like, when you go to the core of not chasing technology, but looking at what problem does it solve for me, and if this is the right sol-

[00:07:06] But back to your question, why the market is excited about, because it's, it's very clear the opportunities that this set of technologies represent in terms of automation, in terms of generation, in terms of productivity, in terms of, uh, accelerating enterprise workloads. The promise is there.

[00:07:25] Krishna Gade: Hmm.

[00:07:25] Maryam Ashoori: The experimentations are done.

[00:07:27] They have seen the values. Now the blocker is the reality of, okay, things can go wrong. There is a certain level of autonomy associated with agents. Do we have the proper instrumentations and guardrails and guard lines or processes in place to protect ourselves?

[00:07:46] Krishna Gade: Yeah

[00:07:47] Maryam Ashoori: They can't solve that problem, they can't really utilize any of the accelerations that comes with the new tech

[00:07:55] Krishna Gade: So I think, you know, when like as you mentioned, when you kind of went, when we went back like maybe three years ago when, you know, ChatGPT came about, you know, models, models were actually the thing, right? Like, you know, you know, uh, you know, model, model layer had the most mode.

[00:08:10] Um, you know, and, and of course now three years later, uh, you know, models are becoming increasingly interchangeable. You know, I mean, yes, there are differences, you know, between, you know, one model and the other, but, you know, those are becoming more and more subtle. So where will the durable enterprise value accumulate, in your opinion? Uh, there's model, there's proprietary data, there's workflows, and there's of course the control guardrails policy layer. Where do you see like enterprises-- Where should you enterprises focus on, you know, as they kind of go about this enter- AI stack?

Where Durable Enterprise Value Will Accumulate

[00:08:42] Maryam Ashoori: Well, there are multiple places when you look into the entire AI stack that depending on what kind of enterprise you have and what kind of assets that you have, you can shine and expand your market share. Let's say if you are sitting on enterprise data, most of these LLMs are trained on public data, which is a fraction of the entire data that is available as of like 2026, right?

[00:09:10] So if you have access to that data, specialized data, that's your moat, right? So that's one. If you go on the application layer, monitoring, governance, like if we truly believe that the cost of software development keeps going down, and that's something that we saw in the past with X-ray. You remember like when at, at, at first when not LLMs, uh, the generative...

[00:09:37] What did they, we call them? Like, um, the,

[00:09:41] Krishna Gade: LSTMs or,

[00:09:42] Maryam Ashoori: the

[00:09:42] Krishna Gade: are they?

[00:09:42] Maryam Ashoori: Early generations of that. They came in, we are like, "Oh, okay, so we don't need the, um, uh, we don't need the person to actually look at the X-ray to figure out what's going wrong." So they started, radiologists, they started losing their jobs because we didn't need them at the time.

[00:10:00] But the cost of X-ray went down to the point that we can now, we go to the doctor, they send us to X-ray. Guess what? The number of requests for X-ray keeps going up, and suddenly we are like, "Oh, so much complications, we need radiologists." So the need for radiologists kept going back up. I think we see the same for software.

[00:10:23] The cost of software development keeps going up, but the generation keeps going down, but the generation of software keeps going up. The demand for that is gonna go up, to the point that we automatically create software at the point of interaction. With all that automation, the key point here, and back to opportunities, is governance.

[00:10:46] You need to have and make sure that the, the guidelines, the controls, the risk, the tracing, the security, everything is there to accommodate that new world that collectively in the market we are not equipped to today. So depending on where you are operating, what part of the stack there is opportunity to figure out where the market is going and how can I utilize the advantage that I have to be differentiated versus just take the technology and go chase adoption of it.

[00:11:21] Krishna Gade: Yeah, makes sense. And so I think that's a good segue, right? So essentially, you know, that you mentioned like, you know, models and proprietary data and then governance and controls. So let's talk about the governance and controls. I would say maybe in the last six months, this this phrase called the AI control plane has just become uh, a, a popular, a popular category, right? What do you, what's, what you, what is your opinion? What are, what are the capabilities a control plane should, should, should have, you know, so that it sort of like, you know, basically can help these enterprise customers?

The AI Control Plane: What It Is & What It Should Do

[00:11:53] Maryam Ashoori: Yeah. So as AI becomes the backbone of how enterprises operate, and AI is changing rapidly, when you're going at that speed, control is not the enemy of the progress. It's like a race car.

[00:12:08] The, the brake is not designed to slow you down. It's designed to make you go faster, corner harder, and stay on the track.

[00:12:16] That's your governance. That's why now we are having this conversation about controls and control planes, because we need to be able to accommodate that speed of moving really fast. And the reality is when you look into the market, the founders that are putting products out there, the startups, like 75% of the business executives, they are not confident they would pass an independent AI audit within 90 days, and yet they have their agents and their products in, like, production.

[00:12:50] Krishna Gade: Yeah. Yeah

[00:12:51] Maryam Ashoori: And the cost is real, too. Like, I'll give you an example. A 20-billion enterprise with weak governance in average is losing about 70 million a year, not from random errors, not from unavoidable glitches, from preventable oversight. So how do they get that control plane? It, it's the promise of giving them a chance to monitor all of those, to understand what they need to control, meaning that...

[00:13:19] And controls, you know where these controls are coming really from. Corporate policies that you have, enterprise AI risk that you're exposed to, regulatory compliance that, um, that the market is defining for you, and operational efficiencies like the payload uh, cost and everything that you care about, really understanding them, mapping them to metrics that you can go enforce.

[00:13:44] And that's the challenge that the market is facing now. So now control plane, the promise of control plane is really three things. One, to help you define those controls.

[00:13:56] The second one is to implement those controls. Let's say if we are building an HR system, we are, uh, using sensitive employee information.

[00:14:07] When sensitive information are involved, you need to put together guardrails to mask the information or make sure that none, none of them are, um, shown up in the output. This is a guardrail. This is a definition of control. So control plane should help you to implement this either out of the box or with AI-assisted coding, um, just to implement what you need.

[00:14:30] And then last piece of this is enforcement. Actually, four, four, four parts. Enforcement is the third one. You need, now that you have implemented that, you have to go and enforce no matter where the workloads are being deployed. And the last one, which is extremely important, is to close the loop, is enforce- of controls happens, what was the impact to my business if the control is working, do I need to adjust the control and come back? Really defining, implementing, enforcing and enforcement tracking is what you need to implement this cycle.

[00:15:01] Krishna Gade: Super. That's, that's awesome. And so in your opinion, uh, should like enterprises like say banks, insurance, or like any large enterprise company, should they look at a control plane about every model application tool, whatever their, you know, data source, or they should look at from a... Do you think this will be a fragmented set of control planes, or do you think there should be an, kind of a unified control plane?

[00:15:25] You know, where do you see, where do you see like how do, how do you think the deployment and sort of the adoption will be?

[00:15:31] Maryam Ashoori: Well, here, um, uh, the answer depends on who you are asking this question. If you are asking the risk and compliance officer in a company or you are asking CISO, they care about the entire stack. It's like at the application layer, if I'm exposed to risk, you need to observe that. At the AI layer, if I'm at, um, exposed to that, you need to observe that.

[00:15:53] At the infrastructure layer, the same, right? But then when you go to the business, um, pillar, depending on what, what you are controlling, do you have an application? Do you have an AI asset? is it a GPU, um, infrastructure-based layer? You are subject to different things. I'll give you an example. Security risks.

[00:16:16] A risk and compliance officer or the CISO, as part of their work, they need to understand the risk they are exposed to. security risk is a category, like OWASP top 10 agentic, uh, application risk, right? But then now that they are defined, you remember the four pillars of control. Now they are defined, now we need to figure out how to implement them and enforce them.

[00:16:38] Usually, the implementation of security risk are within a security tool. It's not an observability dashboard. But if it... If the risk is monitoring to make sure the sensitive information is not surfaced at output, the HR case that we mentioned, this is at AI layer. This is not a security enforcement. You're gonna need to go to observability dashboards to make sure that the tracing is capturing it.

[00:17:02] So really depending on what the use case is and what layer of AI assets are covering, you wanna make sure that you have a full coverage across your stack on a series of different observability platform, but a unified single view of risk across the organizations and all the controls pulling back to, um, the we call it governance console.

[00:17:27] The governance console that you have to track if breach happened or if the controls are working.

[00:17:32] Krishna Gade: No, it's amazing, yeah. That's really cool. So, uh, you know, how sort of, uh, important it is for the control planes to remain independent of the platforms that are building AI? So there's this whole sort of AI factories that have emerged, right? So this foundation model companies, cloud companies, they all have AI factories.

[00:17:50] What-- So is, is there, like, a role for an, an independent control plane or what, what do you think about that sort of?

Independent vs. Integrated Governance

[00:17:56] Maryam Ashoori: Well, I wouldn't, I wouldn't call it an independent control plane because control plane is just a tool. I would come one step back and look into the needs and the foundations. Like control plane is to implement some sort of governance, right? If you like to control and govern something, as a best practice, it's better to have a third party than the party that is building it.

[00:18:22] Like the, um, compliance certificates, like SOC 2, like there's the- company that is defining what the standard is is not necessarily the one that is building the applications, right? The same concepts here. But then for the implementation, it can be implemented in different ways. If there is a technology company that has the expertise to accelerate the implementation of it and you are an enterprise, you need a generic dashboard, that's probably your option to go.

[00:18:51] If you need an specialized thing on a cert- a certain part of the stack that is not covered by the market, you probably wanna build that specialized control plane. So there is no standard definition of what this control plane is. It's really back to those four categories of what you I- call it the flywheel of control

[00:19:13] And what is the best way for you to implement those control all the way from definitions to enforcement tracking?

[00:19:19] Krishna Gade: Hmm. Awesome. So, so the, the other interesting thing that's happening now is actually on a customer call yesterday, and the the, the customer was from a bank, and he was asking that, "Look, you know, like, agents are becoming like these digital employees. You know, are, are you sort of, like, managing them like people do?"

[00:19:37] Like, you know, and, and that was a very interesting thing, right? You know, he was asking, "Do you sell to the chief people officers and head of HRs out there?" And I was like, "Not yet." But, like, but that's actually a very interesting thing, right? Like, will... And do you think enterprises will manage agents the way they manage employees, like with identities, owners, roles, permissions, you know, performance expectations, right?

[00:19:57] Kind of like how you manage people. Or where do you see this going?

Agents as Digital Employees: Identity, Accountability & Risk

[00:20:01] Maryam Ashoori: Well, I will bring up two stories for you. The first one is the role of human is evolving. Historically, we were like, uh, the role of AI was to accelerate our human processes. And by historically, I mean like the last three years. Like I write an email, an agent can help me... Not agent AI, a generative AI can help me with a better written email or the content that I generate.

[00:20:29] So it's a need that I had, and now AI is coming to help me deliver that. Moving forward, the role of the human is gonna change to an orchestrator. So instead of... Like just, just picture this. I use AI to write an email, send it to someone. Someone else is using AI to read the email. So at some point we don't need to exchange these emails, right?

[00:20:53] It goes away. It's a temporary thing that like... And it's just an example.

[00:20:58] Krishna Gade: Yeah.

[00:20:58] Maryam Ashoori: But it's a, it's a temporary thing that for a period of time we are gonna handle, but eventually everything is gonna be automated, and the role of us is really an orchestrator. So now in enterprises, in the current format of the processes and hierarchy, we do not have mechanism to accompany this.

[00:21:19] And, and even, even I don't think market collectively understand what it means in terms of the identity of the agent, what it should be, who is the accountable party. So for example, agents. Let's go a little bit deeper. If there is an HR agent Let's even simplify that. A customer care agent and a bank is using to interact with their customers, end users.

[00:21:46] A bad actor comes in, does the jailbreak, and things happen, right? You don't know what exactly happened. This can be potentially who is accountable here. Something went wrong. We need to figure out what happened. It can be potentially the bad actor, or it can be the business unit that made the call to expose that to the user, or it can be the provider of agent itself.

[00:22:15] Let's say it was a third-party agent that you brought into your custom workflows, or it can be the model provider that trained the model that powered up this agent, or it could be the tools that the agent called. So I don't think this is collectively clear, uh, in the market when you talk to lawyers, when when you talk to regulations, when you talk to the business owners of who is accountable here and what is really the identity of the agent.

[00:22:45] At least in the near future, the person that is using it or impl- uh, applying it is responsible. So, like, if you send a agent to do something, it inherits the access control that you have, and you are accountable if something goes wrong. But there are lots of fascinating unknowns that collectively with the market, uh, we are discovering as we move forward.

[00:23:09] Krishna Gade: And so it's very interesting question, accountability, right? Like in the past, you know, you know, where I used to work in social media companies, you know, was like one incident where the site is down, you know, for many hours and, you know, who did it? Who made, who made the change? You know, the, the intern made the change, and essentially that's the accountability would rest in a person.

[00:23:28] But as you said, you know, where does the accountability fall when an agent goes wrong? Is it like the creator of the agent? Is it the model that powering the agent and, or the one that's the identity that the agent is mimicking? But it's also it can be possible that I can create an agent, I can give it to you, then what...

[00:23:44] That you ran the agent and that agent went wrong,

[00:23:47] Maryam Ashoori: Exactly. Exactly

[00:23:48] Krishna Gade: who sort of is accountable? That's actually another problem, right?

[00:23:52] Maryam Ashoori: But you brought up a very good point that reminds me of a different aspects of this that we didn't talk about. Like in this climate, if I'm an enterprise, what should I do? Because I have to adopt these agents and the technology. I can't, I can't just say, "Hey, I wait in five years to see where the market is going, and then I come in."

[00:24:13] That's not an option. Um, the example that you, um, used. Historically, let's say I'm a, um, major provider of networks across a country. Historically, I was looking into it's a risk category that is called business continuity. They were looking at earthquake happens, fire happens, how can I have backups, backup MZRs, data centers, backup things to minimize that?

[00:24:45] Now moving towards AI, AI everywhere, literally everywhere. Now we need to ask the question, let's say for AI, if the API for the model provider is down for two minutes, what is the impact to my business? How can I future-proof and minimize that impact? So meaning that, like if things happen, do I have a, a, a mechanism to come in and protect myself?

[00:25:14] Because for agents and everything, we know what can go wrong. It's not a matter of, "Hey, don't adopt it." No, it's like before you adopt it, put together a mechanism to protect yourself for whatever you care about. Like business continuity, for example, for years in GRC world, for 20 years peop- enterprises have been using this.

[00:25:34] Now it's applicable to the AI world. Third-party risk is another one. Historically, they were using it for defining like what, what, what company to partner for procurement. Now agents sprawl, AI assets are coming from everywhere. How would you pick which third party to pick, um, um, to trust and bring their workloads to us?

[00:25:56] And if that happens, how are you protecting yourself? Um, so instead of like, uh, we, we obviously have to, we as enterprise look into what are the technology available to me, the best, the state of the art to adopt, but at the most important thing is what are the non-negotiables for me? Do I have the right system, processes, and the tools in place independent of what technology I'm consuming to protect myself and control that?

[00:26:26] Krishna Gade: That's awesome. So let's go a little deeper into your sort of subject matter expertise, which is governance and continuous assurance, right? So, so traditional governance is kind of seen as this, okay, it's a compliance checkbox. It's often based on documentation approvals, you know, some six-month review or whatever. Uh, is that sufficient for systems where, you know, this is like now all sort of agentic and, you know, sort of any one interaction can mess things up, right?

Continuous Governance & Agentic Evaluation

[00:26:54] Maryam Ashoori: You know, like you and I working in software development, historically we used to have a year-long plan with quarter deliv- plan delivery, like a roadmap. It's like my roadmap for next year and every quarter I like to deliver this. And then you take it to risk and compliance, they evaluate, they say, "Okay, we are gonna have a quarterly review for you," and all of this.

[00:27:17] That's the system that was working. Now, roadmaps are so agile, continuous delivery. We are changing things fast and deliver fast, and assets are being developed at a faster, much faster speed that it can be governed. So in this world, the question becomes, how can we protect ourself? How can we accommodate this so we are not slowing down the systems but we do the right thing?

[00:27:47] And I think that's really the core to focus.

[00:27:50] Krishna Gade: Interesting. So, what is the difference between a governing and AI system and continuously assuring that it is behaving as intended? You know, like how has... know, is it like governance has become more runtime now because of agents? You know, have you seen that sort of shift? Yeah.

[00:28:06] Maryam Ashoori: Yeah, it's the, it's the, a very good question. It's the entire life cycle. Historically, if someone wanted to build an application, they would come in, submit a use case. They would say how they are planning to use AI for. Someone would review and approve and say, "Hey, okay, you can start building it."

[00:28:28] Krishna Gade: Mm-hmm.

[00:28:29] Maryam Ashoori: The reality of today is our developers...

[00:28:31] or not just developers, everyone is a builder. Literally everyone is a builder. They go to an AI-assisted development building environment, and they start building. Let's say I'm a developer. I go in, and I start. The first thing that I say is, "I would like to make a customer care application that does this and this."

[00:28:53] The first moment that they think of an idea, so nothing has yet built, we need to be there. Governance needs to be there. We do a quick use case analysis, similarity analysis to figure out behind the scene, figure out if there are similar applications and use cases that are already in the system. And if they are, and they are approved with certain constraints and guardrails, we come back and say, "Oh, good thing you asked.

[00:29:21] Implement these three guardrails, and you're good to go," right? Or if it's a high-risk situation, we are gonna notify risk and compliance that, "Hey, someone, um, is thinking of it. Flag it. Go, go look into that."

[00:29:32] Krishna Gade: Someone's

[00:29:32] Maryam Ashoori: So

[00:29:33] Krishna Gade: a nuclear bomb in our

[00:29:37] blanket

[00:29:37] Maryam Ashoori: So this, not even... nothing is built yet, and you are there.

[00:29:41] Um, and obviously then, um, preparation to go to the runtime, once you have the controls well-defined, they need to provide evidence that they are actually implementing it as part of your approval. And once you get that, then you go to runtime and the traditional things that we had. It's even more important to continuously monitor this in runtime because of the certain level of autonomy that the agents have.

[00:30:09] Um, so just looking at the tracing is not enough. You'll like to make sure... Because, because looking at tracing is showing you what happened in the past. Something went wrong. The breach happened. The control cut that, but maybe it's too late. If it's, for example, leaking information, it's already leaked, right?

[00:30:26] So you wanna be ahead of it, proactively look for patterns to figure out what's, what's the area that is missing or, like, how can I go and do policy enforcement before this action has happened. So, um, obviously the traditional, um, governance was not looking into this continuous compliance and regulations.

[00:30:48] The traditional one quarterly was not designed for this. Everything is so, uh, continuous, and in order to deliver that continuous delivery of automated reports, applicability analysis, and, um, monitoring and enforcement, you really need to have a good grasp of the context Um, and, and, and this is, this is, this is really the part that fascinates me about governance versus just understanding AI inventories or other assets, because governance is about relationships.

[00:31:20] Relationships from AI assets to use cases, from use cases exposing risk, from risk mitigated by controls, controls measured against metrics, metrics showing that the business objective is achieved. If we look into end-to-end, and like in, in our product we call it governance graph, really trying to extract all the relationships all the way from AI assets back to your business initiatives,

[00:31:47] Krishna Gade: Yeah

[00:31:47] Maryam Ashoori: you have a complete 360 view of what exactly is happening.

[00:31:52] And if control is breached, you know exactly what you are exposed to. That historically there was not much need on a continuous basis for this, but now there is

[00:32:03] Krishna Gade: Makes sense. So now, uh, many, uh, these large companies have vendor solutions, right? So you may have like a model from someone, data from another agentic framework, some some other one, maybe some other tools. So how do they think about AI governance, you know, across all of these things, you know? What, what's the, what's a good way to think about it?

[00:32:25] Maryam Ashoori: No, that's, that's a very good question, and this is where most governance approaches fall short because they treat enterprise AI asset and AI risk in isolation from enterprise operations. Take third-party risk as an example, the one that you mentioned. AI assets are coming from different places when different vendors, and you need to know who you are trusting, right?

[00:32:48] Um, there is a third-party risk management in the GRC world that we brought to our world. So for example, in my product, we have partnerships with companies like D&B, RiskRecon, Security Scorecard, Rapid Rating. These all give you deep vendor assessment coverage on a daily basis, including... And some of them even including real-time incident monitoring.

[00:33:10] So if you are using one of these third parties, you need to have the information about the supplier of these assets, not just the assets itself. It's like, what other risk I'm exposed to, and monitoring them and putting together a control around it. And unfortunately, most of the AI governance, um, solutions out there that I see, they focus on just the AI assets

[00:33:35] Krishna Gade: Mm-hmm.

[00:33:35] Maryam Ashoori: in isolations from all the operations, and this operational risk is not just third party.

[00:33:40] We talked about business continuity risk. We talked... Uh, other examples are IT risk, for example, or the environmental risk. So there is a series of them that come into the picture really connecting the AI assets to the operations of the enterprise.

[00:33:55] Krishna Gade: Very cool. One of the things that you mentioned as part of this is that you need to monitor the agentic behavior, and tracing is not enough, right? So there's this concept of evals and evaluating the agentic behavior that is a must because, you know, can collect the agentic trajectory, but if you cannot evaluate it, then it's not, it's not much meaningful. So, know, traditional, you know, model evaluation score, just the final output. Right now with agents like this, you know, we've seen the whole OpenAI Hugging Face incident where something happened and, you know, agent hacked into production servers of Hugging Face. do you think... How do, how do, how, how does teams need to think about the whole agentic trajectory evaluation, planning, reasoning, tool selection? Any thoughts on that?

[00:34:42] Maryam Ashoori: Well, back to, like, I'm coming from the governance world, back to control. So, um, and, and controls are not just risk-driven. Sometimes it's, for example, cost-driven. If, uh, a agent is using more than certain tokens, you want to stop it before you get the bill, Right?

[00:34:59] Krishna Gade: right

[00:35:00] Maryam Ashoori: So... or sometimes maybe it's the payload or access things that you like to, um, stop them.

[00:35:07] Um, so back to this, I would, I would start with, one, what this agent is supposed to do.

[00:35:16] Krishna Gade: Yeah.

[00:35:16] Maryam Ashoori: What are the risk or dimensions that I care to control around the behavior of the agent?

[00:35:23] Krishna Gade: Yeah

[00:35:24] Maryam Ashoori: Can I map them to some metrics that I can go back to tracing and start measuring as, like, the definition of signals? Like, what is the threshold that I need to be notified that it's working or not?

[00:35:37] And then throughout the life cycle, when it's the build time, I need to be able to evaluate them. Like, sometimes benchmark comes in, like if you want to pick a third-party agent, for example, you look into their offline evaluations that the company has to make sure you get the best one. Then we have, um... so that's the offline evaluation.

[00:35:59] Then during runtime, when you go to the runtime, you want to make sure that you do overtime evaluation, like that is historically we did it for traditional ML tool. If the agent is drifting or on the behavior and stuff over time, do I have the right metrics to track that? And also, um, for the agent itself as it's, uh, making the decisions.

[00:36:24] So for example, for us, like we, we see agent is not really a black box of input and output. The input goes in. Think of a node. It breaks it down to a series of steps, and then for every step, it decides to do a number of calling. At every step in that node, you need to be there to do the evaluation. So there is evaluation in the node, there is evaluation over time, and evaluation at build time, which is most of the time offline, to make sure that you have all the metrics monitoring that you need to monitor in order to make sure your control is working

[00:37:05] Krishna Gade: Yeah, yeah. And that is also hard, right? In the sense, you know, many times you don't know what good looks like. You know, it's, it's sort of like what is the ground truth, you know? Is, is... Uh, and that's, that's the hard part that, you know, industry is facing right now in terms of how effectively you want to evaluate the, an AI system.

[00:37:22] Maryam Ashoori: Exactly. But also the goal for evaluation. Sometimes the goal for evaluation is to make sure that, um, the controls are implemented correctly, like guardrails, but sometimes the goal is optimization. Like, you like to optimize your cost. You like to optimize your prompt, right? And so it's important to understand what is the goal for that evaluation too

[00:37:45] Krishna Gade: Yeah. So, so I guess, you know, when you have seen, uh, agentic journeys of your customers or, or sort of other teams that you spoke to, what, what's, what have you seen, you know, things breaking? You know, when an enterprise goes from, say, let's say a handful of agents to, like, maybe 100-plus agents in production, you know, across different departments.

[00:38:04] You know, where, where have you seen arise, you know, agents running in production and misbehaving?

[00:38:10] Maryam Ashoori: Yeah. Uh, you know, we frequently run customer advisory boards, and in the last one I asked a similar question. I'm like, "What is your top challenge

[00:38:21] Krishna Gade: Yeah

[00:38:21] Maryam Ashoori: it comes to agent adoption?" And guess what? We already talked about that. The number one is accountability. Who is accountable?

[00:38:30] Krishna Gade: Mm-hmm.

[00:38:31] Maryam Ashoori: Especially because the leaders that we talk to, they're usually responsible for their piece of the work.

[00:38:37] And here we are talking about multiple departments within the business making the call, but also the exposure to external entities like model provider, agent provider, the tool provider, the person that makes a decision to surface it to the user or the bad- actor, the user itself. And, and, and that- that's really the number one, um, problem that they keep citing because this, this is not something that you just solve with the technology.

[00:39:11] You can... The thing that you can do as an enterprise is tracing instrumentation to put together all the instrumentation so you can go trace back and figure out where it was stemmed from. But then enforcing it to third party requires external regulation to catch up, right? Um, so it's not a one person, one company, get it done and move on.

[00:39:36] Yeah

[00:39:37] Krishna Gade: makes sense. Let's take some relevant audience questions. There's a question from Kenneth Roy. You know, what would be a good governance model organizational chart across enterprise companies when many departments and functions that are individually trying to adopt AI agents into their workflows?

[00:39:52] Is there a responsible position or committee? So it sort of aligns with your accountability question, but any thoughts on that?

Scaling AI Governance Across the Enterprise

[00:39:57] Maryam Ashoori: Yeah. Usually the way that we see it is we have the line of defense within the enterprise and the line of business. Line of defense, the first line of defense is obviously the business owner. The second line of defense is risk and compliance officers, and the third line of defense is the audit team.

[00:40:13] Those... The line of defense is usually they need to have access to the entire AI estate, no matter which department you are sitting in. So if you are part of the business side, obviously you're subject to the, um, guardrails and controls that the risk and compliance is exposing to. Um, but when you are, uh, um...

[00:40:36] Yeah, and then on the business side, you just really need to make sure that you have the right control plane to control your realm of the business versus 360 entire AI estate of the company

[00:40:51] Krishna Gade: Got it. Got it. And so maybe a, a relevant follow-up, you know, Satvik Parasa asks, "What type of metrics would convince you that an organization's AI governance program is actually effective rather than just compliant?" You know, what does-- what defines good governance?

[00:41:07] Maryam Ashoori: I, I love this question because most of the time people think about AI assets use case mitigated by risk control metric, and we are done. But if you notice, like, there was one last piece of the puzzle, metric connected to business objective. So in the governance graph, you can trace back from AI assets all the way to business initiatives.

[00:41:30] Because at the end of the day, we need to hold AI accountable to deliver on the outcome that it was promised to deliver, right? Um, and this is, this is your direct segue of AI assets to business, um, initiatives. It gives you two things. One, obviously over time gener- general layer at a, on a monthly basis or weekly basis, you have access to the dashboards that can show you the effectiveness of those AI assets.

[00:41:59] But the second thing that it also gives you, if when you have controls in place and something wrong happens, like breach of control, you can trace it back exactly to what KPIs was on breach. What is the impact to my financials and business

[00:42:17] Krishna Gade: Right

[00:42:18] Maryam Ashoori: from a metric that was breached in a tracing dashboard?

[00:42:22] And I think that's the beauty of the governance graph.

[00:42:25] Krishna Gade: Yeah, makes sense. And, and so I guess, uh, back to that sort of accountability and self-attestation, right? At, at what point do you think companies can't rely on self-attestation anymore and need an independent assurance for higher risk AI systems?

[00:42:43] Maryam Ashoori: I wish I had an answer for this. You know, recently I had a conversation with a startup founder. Um, he used to be a lawyer, and now he has a company that does this exactly for the companies. So you make an AI decision, um, at this point of time, you pick a third-party AI provider, model provider. The startup would run some tests.

[00:43:13] They would legally document all of this to show that as an evidence that at that point of time you considered this option, you evaluate it and you protect it yourself. The regulation is evolving rapidly. Nobody knows in the course how AI is gonna be held accountable. There is lots of movements in the market in terms of startups, insurance providers, but it has not yet emerged into a single unified.

[00:43:44] And, and just think about that at the global layer, it's even more complicated because every company... every country has their own regulations when it comes into enforcement of these things

[00:43:56] Krishna Gade: Yeah, absolutely. And now it's, it's actually has become a best practice, not just a regulatory shield for most companies, right? Essentially, as you articulated, you know, it helps you go from

[00:44:07] Maryam Ashoori: Faster

[00:44:07] Krishna Gade: APIs and become your agent native. Yeah, that makes sense. Uh, so there's also this whole sort of debate versus human in the loop versus human in the loop.

[00:44:18] You know, when

[00:44:19] does it become-

[00:44:20] Maryam Ashoori: what's the new thing? Human in the lead

[00:44:22] Krishna Gade: Human in the lead, yeah. So what's your take on that? You know, uh, uh, where do you suggest, you know, human in the loop versus not human in the loop?

[00:44:30] Maryam Ashoori: Well, we talked about human, um, becoming an orchestrator. Once you are an orchestrator, you are really the lead, right? That's why I like human in the lead as a good metaphor where it says, "Oh, keep human in the loop. Make sure that there is human in the loop to have some sort of oversight." No, like human is in the lead

[00:44:52] Krishna Gade: Yeah. Makes sense. Makes sense. So, so I guess, uh, you see, uh, the categories of control plane and governance kind of converging? Or do you see, like, you know, it becomes another, like, control plane kind of becoming a foundational layer, at the AI infrastructure stack? You know, where do you see five years from now?

[00:45:12] Maryam Ashoori: Well, control point to me is just a tool to implement what you need to govern

[00:45:17] Krishna Gade: Mm-hmm.

[00:45:17] Maryam Ashoori: control. Um, you can call it control point, you can call it control tower, you can call it the governance console, you can call it a dashboard. So, so, uh, to me it's really the, the concept of what it's delivering, which is the four thing that we talked about, the flywheel of control.

[00:45:37] Defining controls, implementing controls, enforcing them, and tracking them. No matter how it makes sense for your enterprise to implement that, this, these are really the four things that I don't see changing over the next five years. The nature of that will be changed, but GRC has been around for 20 years or more, uh, and it's gonna continue to be relevant for many more years to come

[00:46:02] Krishna Gade: Yeah. So, so when it comes to enforcing controls, there's actually an interesting question from Gurpreet that, "How do the enterprise customers scale agentic controls without pushing the work to individual engineers? We are trying to manage the steering files, and it has become a headache." You know, essentially a lot of the agentic controls are being pushed to engineers to encode into steering, you know, steering files and, you know, it's... And Gurpreet is not certainly liking it.

[00:46:27] Maryam Ashoori: Yeah. So, so one of the things that, uh, we recently released is we call it use case onboarding optimization or risk advisor. Um, so instead of someone going and, um, actually saying that, "Hey, I would like officially submitting a use case," and then a human comes in and reviews the stuff, uh, we are automating a lot of that.

[00:46:49] So like use case similarity analysis that we talked about automatically looks into are there relevant use cases? If there is one approved, move forward. And through that we were able to, um, in Q1 get about 55% efficiency. So if you're a developer, you don't wanna deal with any of this, right? You're a builder.

[00:47:11] You like to build the things and move forward, but also you don't wanna be accountable to figuring out what is the guardrail that you need to implement or thinking about, oh, when it goes wrong, like, what do I need to do? Obviously, you need to think about it, but then there is a structure... Just think about it.

[00:47:28] What if there is a structure that tells you exactly what needs to be implemented and help you to implement it? Does it make it more difficult for you, or make the life much easier? Because at the end of the day, you're not accountable. Someone else did the risk analysis and told you you're just accountable to implement that guardrail.

[00:47:48] So the goal of all of this is to make the life much easier for the builders, not necessarily complicate things. And that's why, like, we are packaging, uh, the capabilities that we are designing as MCP servers, as skills that can be surfaced versus a product that you need to go to the interface and, uh, report things.

[00:48:09] Krishna Gade: Yeah, maybe a shameless plug here, Gurpreet. So there's a policy engine that we are building at Fiddler where you can encode all of this at an admin level so that you don't have to deal with all the multiple steering files. But, you know, check us out and we can chat later. So let's actually go into a rapid fire section, Maryam, I think as we round this out.

Rapid Fire Q&A

[00:48:25] Krishna Gade: Uh, so, you know, short answers, you know, sharp takes, uh, uh, traditional ML or generative AI?

[00:48:33] Maryam Ashoori: Both

[00:48:35] Krishna Gade: Okay. spoken like a

[00:48:37] true enterprise

[00:48:38] Maryam Ashoori: Yeah. well,

[00:48:39] Krishna Gade: veteran.

[00:48:39] Maryam Ashoori: is the thing. I used to work for Lyft, and my job was to put the right bike at the right place at the right time. This is a classic predictive

[00:48:48] Krishna Gade: Yeah.

[00:48:49] Maryam Ashoori: challenge that requires traditional ML. Come on.

[00:48:52] Let's, Let's, not mix them up

[00:48:54] Krishna Gade: absolutely. Awesome. Yeah, great to see that. Copilots or autonomous agents? I think we kind of spoke about that, but yeah. You wanna give your hot take

[00:49:02] Maryam Ashoori: Oh, Copilot, is it the Microsoft Copilot or Copilot?

[00:49:05] Krishna Gade: meaning human in the loop copilot, sort of copilot where a human is

[00:49:09] looking at it,

[00:49:09] Maryam Ashoori: okay, so

[00:49:10] Krishna Gade: Yeah

[00:49:11] Maryam Ashoori: human in the loop versus total 100% automation. I would say it depends on the use case

[00:49:16] Krishna Gade: Got it. powerful agent or many specialized agents?

[00:49:24] Maryam Ashoori: I would say depends on the use case. Sometimes the cost of orchestration across the agents can be higher than one Uber agent

[00:49:33] Krishna Gade: Okay, here is a hard question. Frontier models or small specialized models?

[00:49:38] Maryam Ashoori: Well, frontier model is still good when you want to burn cash. But if your use case doesn't require that accuracy, small models

[00:49:47] Krishna Gade: Yeah. Open source or proprietary models

[00:49:51] Maryam Ashoori: I'm a fan of open source, so I, I, on this one I would go open. actually, I'm, I'm a fan of open source to a point that, like, you can look it up. I even designed a board game and I made it open source. You can print your own to that level, so not even code, yeah.

[00:50:08] Krishna Gade: What is it called?

[00:50:09] Maryam Ashoori: It's called Entanglion. It is, uh, a, uh, an open source game to teach people how to program quantum computers

[00:50:17] Krishna Gade: Super. We'll check that out. We'll put, put a link in the video later. Um, centralized or federated AI governance

[00:50:29] Maryam Ashoori: It depends on the industry

[00:50:31] Krishna Gade: Yeah. Yeah. Observability or enforcement, which is harder in your opinion?

[00:50:44] Maryam Ashoori: Enforcement requires observability.

[00:50:47] Krishna Gade: Yeah

[00:50:47] Maryam Ashoori: If you don't have observability, you don't know what you're enforcing

[00:50:51] Krishna Gade: Makes sense. Uh, bigger risk, what is a bigger risk, a malicious agent or an over-permissioned agent?

[00:51:01] Maryam Ashoori: It depends on what the agent is trying to achieve

[00:51:06] Krishna Gade: What is more important, model accuracy or system reliability?

[00:51:11] Maryam Ashoori: System reliability

[00:51:13] Krishna Gade: And harness as the work. Uh, most overhyped agentic use case that you have come across in the last few months

[00:51:19] Maryam Ashoori: most hyped

[00:51:28] You know, there are lots of statement that are being made that, like, agents can do almost everything which, uh, in an experimentation, sure, but in the reality I would say it's applicable to almost all of the agents, um, um, if they are not designed for production. Um, it's so easy to build a an agent in five minutes that does rocket science, but is it appli- is it practical?

[00:51:56] Can I use it?

[00:51:58] Krishna Gade: It's, it's nine- it's 90% more, uh, difficult to operationalize it. You know,

[00:52:02] Maryam Ashoori: Exactly.

[00:52:03] Krishna Gade: right?

[00:52:04] Maryam Ashoori: Exactly

[00:52:04] Krishna Gade: Yeah. Most underrated enterprise AI use case

[00:52:13] Maryam Ashoori: Well, the most popular one is still content grounded question and answering, which is a simple question and answer, but still there are lots of efficiencies that enterprises can gain

[00:52:24] Krishna Gade: Oh, oh. one metric every AI leader should track

[00:52:35] Maryam Ashoori: That metric is, do I have all the right metrics that I need to track?

[00:52:40] Krishna Gade: That's great. Very well said. Uh, what should an AI agent never be allowed to do autonomously, in your opinion? Never say never or

[00:52:52] Maryam Ashoori: Well, there are things like, um, the foundations like AI should never harm human no matter what.

[00:53:00] Krishna Gade: Mm-hmm.

[00:53:00] Maryam Ashoori: So there are a series of foundations that no matter what, and it's not just applicable to agents, it should not be broken

[00:53:08] Krishna Gade: Yeah. finally, what AI belief have you changed your mind about?

[00:53:16] Maryam Ashoori: What AI belief has changed my mind

[00:53:18] Krishna Gade: like have you... Like maybe you had a certain belief and now it has changed, you know, given happening within AI

[00:53:26] Maryam Ashoori: Well, I'm an AI fan.

[00:53:28] Krishna Gade: Mm-hmm.

[00:53:28] Maryam Ashoori: I spent the last 20 years working on AI, so I'm pretty biased that AI is

[00:53:34] Krishna Gade: dreams

[00:53:34] Maryam Ashoori: cool. But, but I would say that the thing that I keep observing in the changing people is at the beginning they were blindly trusting AI, and now with all the educations happening um, for not just enterprise leaders, like outside enterprise, the schools, uh, kids interacting with that, like, um, parents interacting with it, um, people on the street interacting with it, they are more aware of what can go wrong and limitations, which is, which is the right direction to go.

[00:54:10] But historically, like three years ago, they were like, "Oh, just trust AI no matter what." Yeah

[00:54:19] Krishna Gade: Awesome. great, uh, way to end the conversation here, Maryam. Thank you so much for joining us and, uh, uh, you know, and, you know, sort of sharing your valuable thoughts.

[00:54:29] Maryam Ashoori: Thanks for having me, Krishna

[00:54:31] Krishna Gade: Awesome. Thank you so much. Thank you, folks. Uh, we'll come back with, uh, another great guest in a few weeks. And until then, you know, have a nice day.