AI Safety in Generative AI with Peter Norvig

Krishna Gade: Welcome everyone to AI Explained, this webinar series that we've been doing at Fiddler. I am the founder and CEO of Fiddler, Krishna Gade, and I'll be your host today. Here with me is Peter Norvig. Welcome Peter to the show. Peter Norvig is a legendary figure in AI. I am sure that a lot of you have read his book on the AI Modern Approach.

I read it in my grad school. He's currently a Distinguished Fellow at the Stanford Institute for Human-centered AI. He's worked for a number of years on Google Search and improving search quality. So welcome to the show, Peter. Thank you. It's an honor to have you, Peter. 

Peter Norvig: Great to be here. Thanks. 

Krishna Gade: Awesome. Peter, we are in the midst of this revolutionary thing called generative AI, and you've been working on AI. What are your thoughts on generative AI? 

Peter Norvig: It's amazing this, this era that we're in and how fast things are moving. I started out in the days of good old fashioned AI, where things move very slowly because we said the way we're gonna put knowledge into a machine is to hand code it in some logical language.

Using the blood, sweat, and tears of graduate students. and so you could never put that much in. And there was also, a, the maximum you could reach because as you started to put more in, it started to contradict each other. And because we were basically using logic with a few non monotonic exceptions, you couldn't really resolve those conflicts.

So, that was discouraging. 

Then we moved to machine learning and suddenly said, now we have an unlimited amount of data to learn from. All we have to do is come up with the right representation and we thought that that would be the hard part is saying, what representations are we gonna use for all this information?

And then neural nets came along and said they're gonna make their own representations and we're gonna make them deep enough and have the right architecture. So that the intermediate nodes can form representations. And it seems that they do a pretty good job of that. And we still don't quite understand why, but it is really exciting to see how fast it's moving.

Krishna Gade: Awesome. And as you reflect in your book, artificial intelligence has changed over the years and from logic programming to machine learning now to generative AI. How are you seeing this whole emergence of foundation models?

Where do you see the future of generative AI go? Do you see a world where people would just use foundation models and build off of them? Or do you see old school kinds of model training still exist? How do you see the world in the next 10 years?

Peter Norvig: So I think the old techniques will still coexist. as a textbook author, I'm thinking, do I not get to throw away 25 out of 28 chapters and 

Krishna Gade: And just focus on prompting? 

Peter Norvig: and I don't think that's right, and I think you can see that, right? So, look at things like AlphaGo, the Go Playing program.

So the main innovation was saying, we're gonna represent the state of the board with the neural net and convolutional neural nets are good at representing pixels and, go pieces on a board are like pixels. so that's a good representation. and I think that's what made the difference between that and previous systems.

But they didn't just say, let's just have a generic neural network. They said, let's use these techniques that we already know about, like game tree search and so on. and I think in going forward we'll have more of that. We'll say what techniques are appropriate? Now let's figure out how to combine those techniques.

With a general representation, like a deep neural network. 

Krishna Gade: One of the things that you mentioned just now is that this deep neural network learns patterns from data and we don't know how or why it works, and therein lies a lot of risks. And people talk about concerns around generative AI. There's a lot of talk about AI safety these days.

What's your take on that? how does one an organization or anyone. 

Think about this. 

Peter Norvig: So I think safety is really important. I think there is some confusion, right? So we talk about neural nets that are hard to explain. Because it's just matrix multiplication and they're big matrices with lots of numbers.

as opposed to other techniques, maybe you're easier to explain, decision tree or just straight line code. But I think there's a confusion there that is part of the difficulty in understanding. Comes from the solution. Being a neural net. But I think more of it comes from the problem. and problems are hard to understand no matter what solution you use.

if your problem is, coding up the system, the software for a bank. then that's hard to get right, because there's a million different regulations on what taxes and what fees correspond to what transaction. but you know that there is a right answer. 

And so if you look at it carefully, regardless of how it was implemented, you could say, yes, this is right, or no, it's wrong in this case.

But I think with AI, problems, part of the difficulty is that often, there is no ground truth. So here's a picture. We do object recognition. Some of the objects are, there's an obvious correct answer, but other ones, this is a dog or a wolf. Nobody knows for sure. So there is no ground truth.

And, that's where the difficulty comes in. And, one of the things, The example I like to use is we say, neural nets are hard to explain, but decision trees are maybe easier and, maybe sometimes it's a good idea to say, let's build a neural net now let's translate it into a, the closest decision tree and see how the two compare.

And if they're close enough, maybe you wanna use the decision tree at one time. and we say, decision trees are easy to understand because it says if A and B and C, then X. but if that were true, then you know, regular software also has IF statements that says if A and B and C, then X and regular software has bugs in it.

And why does it have bugs? It's because somebody looks at that and says, that statement is correct. And then at runtime there's an error, right? And then they say, oh, of course I meant if A and B and C, but not D. Then we should do vax, right? It goes without saying that it doesn't apply when D is true.

And that's where the bugs come in. And so that type of bug, that kind of weird exception that you didn't think about. That comes about because of the problem, not because of the solution, whether it's a, 

Krishna Gade: in the human understanding problem. Subject matter expert. So there's like a human in the loop in software building and so you're able to, yes, so I think you want a human in the loop. 

Peter Norvig: You want automated tools in the loop. You want different types of representations and explanations. You want lots of tests. You want red teams to try to attack it. but there is no one solution. It's going to say this is the answer. 

Krishna Gade: It's a very interesting point, right?

So essentially problems are harder and you need human subject matter expertise to decipher them, and you build software and then software can have bugs and therefore you still do this software practices. AI makes it completely different. AI itself has a black box, there's no ground truth, and so therefore you need to invest in these things.

Now, one of the things that I'm actually very curious about is that you were a big proponent of simpler models and large data. back in the day, I remember watching your lecture, when I was working at Bing and you were Search Core at Google search and you had proposed that, hey, great data with simpler algorithm would always be like, is how are you thinking now at the models becoming more and more complex and zero shot learning, one shot learning, know, emerging. How, how do you view this world, today?

Complexity of algorithms. 

Peter Norvig: So, I think that's right. I've seen that progression and we can see that going back to the additions of the textbook. So in 1995 was the first edition, and my co-author, Stuart Russell, I think we felt like AI is part of computer science.

Computer science is about algorithms. So this book is gonna be mostly algorithms. with some explanations of what's going on. Then we got into the second and third editions. You get up into the two thousands. we have this era of big data and we said, we're still gonna have all the algorithms.

It's still gonna be a thick book. Sorry. but as we were saying, if you wanna improve, There's probably more leverage in getting better data Than in getting better algorithms. and I think that's proven true. In the latest edition from two years ago, I think we had a revelation that said, up to now we've been saying, you get your algorithm, you get your data, somebody hands you an objective function, and your job as an engineer is to maximize that.

but that objective is a given, I think we're saying now, maybe that's the hardest part. So don't just say that's outside of the field that was handed to you by somebody else. Say, figuring out what your objective is, what you're trying to optimize, What's fair? That's the hardest part.

And if you get that then, the data and the algorithms, that's easier. 

Krishna Gade: So we are now living in the age of complex algorithms. The models have become complex and they're things that we are seeing, hallucinations are a good example, models potentially leaking private data.

bias is always an issue that people are concerned about. Let's see. If you think about this hallucination, which is essentially something a lot of our customers are implementing generative AI. What's your take? First of all, why do these generative AI models hallucinate?

Peter Norvig: So I think, a synonym for a hallucination is creativity. 

And if you ask the system, design, an outfit that I can wear to a party, and it comes up with an outfit that's never been seen before, you say, awesome job. I love it. That was very creative.

Kudos to you. But if you say, I'm filing a court case. before a judge and it comes up with precedents that had never been seen before. That's illegal. 

And so the problem is we haven't told these systems when they should be creative. And up with something new and when they should be reporting. and you it's various approaches to do that. I think we can do it, there's been experiments saying, you should have access to knowledge bases. So maybe you should be looking some of these things up. If you think about it, how do we humans operate? Yep. If you ask me the capital of France, I can say Paris.

But if you ask me, the capital of every county and every country in the world, most of them are gonna have to look up. And so our systems should be doing that, or they should be calling out to other expert systems.

And putting pieces together. So I think it's, as you say, the architecture gets more complicated. And one of the things we have to do is separate out the creativity. From the factual reporting and and, the documentation of where the arguments come from. 

Krishna Gade: So attribution is a hard problem if you can make the models attribute to where they're getting the content from, do you think we arrive at technology that would allow us to do that in the future?

Like where we can, models can actually self attribute, like in references from, document data? 

Peter Norvig: I think so. I don't see any barrier to that. It's just we thought the easiest thing to do was just say, let's feed in as much text as we can. 

but I think we've already seen that with the, if you annotate the text then you can come up with better representations. So I saw, it's a nice paper. I've gotten the name of the author now, where they are trying to train a model to be a dungeon master to play Dungeons and Dragons. And you could just train it on the transcripts. Yeah, but then it doesn't do very well.

But you can annotate the transcripts to have a kind of the state of the game and you know what the players are and how many hit points they have and so on. And when you put in those annotations, then it does much better. Right. So I think that's the kind of thing we should be doing is saying this is a citation.

There are things like journals and articles and facts and databases and so on, and don't just dump everything out of there as text, annotate it to say where it came from. 

Krishna Gade: So one of the things with hallucinations is, especially, it's okay as you said, for a ChatGPT or something to be creative and hallucinate.

So we work with enterprise customers and we were helping one of the health insurance companies to create a chat bot recently. And so it was, so there was a simple question called, what is a, well known soft drink that would, help, human health condition positively or something?

And so the right answer is there's no such scientifically proven soft drink. but you can make the question slightly different. And, then you can have text-davinci hallucinate and say answers like water. Red wine and stuff. So this is, potentially, if, for example, if a bank wants to use it for wealth management or, a healthcare company wants to use this for clinical diagnosis in front of a physician or something.

Hallucinations can be problematic, right? how do, how does one think about, preventing these things, like using whatever existing techniques and tools they have. 

Peter Norvig: I guess the first thing to think about is, this is not a new problem. That comes from generative AI, right?

So we're exposed every day to advertisements and, some of them are outright lying and most of them just go right up to the barrier where they may imply claims, about health or whatever, but don't quite make a medical claim. and they're designed to trick us into believing those claims.

so it's not surprising that our AI models pick up on that because it's everywhere. and to. Combat that you need, more critical thinking. So you need that as a consumer of the news and advertising, and you need that, as a check on these models. 

Krishna Gade: So you, bring up this point that AI models are not perfect, they're error prone, and therefore you need some sort of a monitoring of AI models.

And some humans in the loop. The other sort of concern is like the bias thing, right? So we have now these foundation models scouring the web and processing the content, and then, they can be learning all the biases that people have written on the web. And now, that's a big concern.

How do you, how do we, create a world where gendered AI does not advance discrimination? like that's a big problem. 

Peter Norvig: So I think a couple things. So one is, you wanna be measuring how you're doing. and, one is you wanna have, just be aware of this. and I think part of that is just having diversity on your team in terms of what groups are represented, what nationalities, what cultures, and so on.

And because if you don't have that, you won't be able to recognize some of the biases. And we've seen that in things like the major search engines. A few years ago you did an image search for brides and they're all white dresses and now, you do that and it's more diverse and then you say, gee, there are different cultures and they have different traditions and they have different kinds of traditional dresses.

And, that came about because someone noticed that it was an issue And said we need to add some diversity here. And then, went about doing it. so you're always gonna need teams that, diverse teams, that are aware of the issues, and can combat that. to some extent some of that falls out, in that there's always this idea that diversity is good.

And, the first reason it's good is because, going back to search, if you put up 10 links, it would be bad if all 10 links were mirrors of Wikipedia to the exact same article, right? So instead you say, we'll put the best one first, and then the second one should be the link that adds the most information, assuming that you've already seen the first one, right?

And so that kind of automatically gives you more diversity. Because there's only value of information to something that's new. Then there's another kind of bias, besides just, so there's bias. In the data sources. because societies are biased, but there's also bias for the majority because machine learning models work better if they have more examples.

So even if, some minority groups, there's no prejudice against them. Everybody likes them just as much as the majority group. They have less data, they're going to do worse. And then there's bias in terms of, as an enterprise, You have to decide who your customers are. And at some point you're gonna leave some of them out.

And again, this is not new to ai, right? For decades we've had companies that say, we're shipping a product. We're gonna put an instruction manual in that product. Majority of our customers speak English, so it'll be in English. And then one day they say, we have a pretty good minority that speaks Spanish, so let's put in a Spanish manual as well.

And that's an additional cost and translation and paper and so on. And then you have to say, am I gonna add a third and fourth and fifth language? And at some point there's a cutoff. And some people get left out. And that's always been the case that, if you're in a minority group, you're gonna get less attention.

Krishna Gade: That's a big problem. Because, someone is asking, I think Charles, wait, Charles, so it seems like, The internet is authored by a majority group, like the privileged group. And so there's, most of the sources, data sources, as you just articulating, came probably from the people that had access to the internet first and they could write content.

And now generative AI is gobbling up this information. Could actually undermine those like other, emerging countries and people who don't have the same kind of access to the internet and probably have not had the chance of putting their content out as much. how do we, and then the, now there is a problem that we are, we might be living in an age where generative AI models are feeding each other off.

Because if the content generation is gonna be automated, Now a model could be generating content, which would then feed off information to the other model that is coming in the future. So now you'll have a propagation of this sort of bias, like at, a much larger scale. I guess I'm not too 

worried about that.

Peter Norvig: I could see a future where that's a problem but a couple things. One, we were worried about this a decade ago with machine translation, right? So we were offering a machine translation service. People were using it, posting stuff on the web. We said, what if we're getting feedback from our own output?

so we didn't experiment where we, watermark our output so we could probabilistically tell which was ours. And it turned out it was a very small percentage, so it wasn't a really big deal. So one answer is if it's gonna be a small percentage, then you shouldn't worry about it. The other answer is if you can detect the difference between quality and non-quality, then why should you care?

Whether it was generated or not. 

Krishna Gade: Makes sense. So, it seems one of the things that we need to pay attention as AI progresses is to have some. Practices and to implement it, safety, to have humans in the loop, but maybe drawing from your experience of building probably one of the world's first large scale ML application, which is Google search in some ways, what advice would you give for companies to build large scale ML applications, from whether that's predictive ML or generative AI. How does one go about the process of building a great and responsible and safe AI application? 

Peter Norvig: I guess the first thing, and, I spent a lot of time, advising startup companies on, on their machine learning practices. And so many times we've had this conversation we're coming in, they say, oh, there's this deep learning stuff and it's so complicated and there's all these maths and there's partial differentials and so on.

And, I'm nervous about that. And I would say, get started a couple months from now, you're gonna be completely over that nervousness. And, this big black box in the middle that you think is so big and scary, that's gonna look small and tame. And instead, what's gonna matter is this pipeline of the data coming in and then how you serve your customers and all your attention will go on that.

Finding the right data, generating new data if you have to, having human oversight of that, having automated ways of cleaning it up, and then how do you serve it? So on that whole pipeline becomes the important part. so, I think that's the first lesson that people come to.

Krishna Gade: Sure. And where does AI Observability fit in this picture? I'm sure like when you were building these ranking models at search, you were having some sort of a feedback loop right here. 

Peter Norvig: Yeah, so we thought of it at three levels. So one is the clicks. And both short term, did people click on the number one?

That's probably a good thing. Unless they come back right a second later and then click on something else, then maybe it was misleading. And a little bit longer term, are we keeping our customers or are we losing them? Yeah. so that's at one level. 

Krishna Gade: So like business level data.

Peter Norvig: Yes then at the next level we said, we can't do it by clicks alone. We really need some human judgment. We had paid workers to say, is this a good result or not? Was the user happy with this? Of these two possible versions of the system, which one produces better results?

We didn't call it reinforcement learning with human feedback, but that's the kind of thing it was. and then the third level was, we would bring people into the lab or we would go into their home. And we would observe one person at a time. Super carefully. We had the lab with the mirrors and the cameras behind the mirrors and so on, and said, do a search.

How would you do this? Think aloud as you're doing it, say, what's confusing? How is it working for you? And so from having that, billions of clicks. Thousands of human judgements and a single person at a time interview that those three levels put together gave you a better picture of what was going on.

Krishna Gade: So like business level data, human labels, like getting the ground. Data In some cases, ground truth is hard to achieve. For example, we work with large insurance companies, banking customers, you wouldn't know until someone pays their first installment Whether they've issued a good loan or not, and things like that.

What's your take on data drift? And people are starting to monitor these signal drifts. How do you think it will help, ML team?

Peter Norvig: That's hard. And I think you're right that, maybe within this it's a little bit easier.

Are people actually paying or not? Yeah, you can see that there's other places where it's even harder, right? So there's been a lot of controversy over AI systems helping in the justice system, right? Making recommendations for, should you get parole because are you going to recommit a crime?

And part of the problem there is. We don't have the ground truth. What we really wanna know is, are you going to commit a crime? From that, we look at similar people and say, have they committed a crime? We don't have any records on who commits a crime. All we have is who's been arrested and who's been convicted.

And both of those processes can be biased, that the selective policing in certain areas is a selective choice to arrest or not. And then bias in the juries of who they convict or not. So we don't have any ground truth. And in order to deal with that, you have to estimate, how much bias is there?

In the data that we have. And that's hard to do. 

Krishna Gade: And then what would one need to do? Like in the case of where the ground is not available? Is there another way to like proxy, get some proxy measure in terms of how well your model is performing at any given point of time?

Have you seen anything work in practice? 

Peter Norvig: You can, look in depth, you can sample A few cases. And look in more depth. And, we see that in the justice system. Like these Innocence projects of saying, here's somebody that was convicted in the past.

Let's look more closely. We have better tools available today, like DNA analysis, right? Were they wrongly convicted, right? 

Krishna Gade: So having some sort of a human in the loop in the work.. 

Peter Norvig: And you only have to sample a small number of cases and look at them in depth to have a better idea of are you making systematic mistakes.

Krishna Gade: Switching gears a little bit into, sister topic, right? AI from a technology point of view is fascinating. It can enable so many things. But then what do you think is a social responsibility for companies implementing AI? How do we create, I think this is a question coming from Deborah Adams now, how do we create an ethical standard that is being woven into the most of the tools that we use today?

How do we wanna make sure that AI, it will be used for good, without slowing down. 

Peter Norvig: That's hard. And as I was saying, right? So in the latest edition of the book we said that the hard part is saying what's fair? What do you wanna optimize? What are your true utilities? And I think we've seen a lot of effort In that way. All the big tech companies, many of the governments, and many other organizations have their AI principles. I think that's good as far as it goes. I wish there was another level to that, right?

Because, you have AI principles that say things like you should, respect the user and their data. And that sounds good, but. Say, I've got a database of faces. What am I allowed to do with that? And what am I not allowed to do? That principle alone doesn't really tell me what the limits are.

So I would want another set of principles that says, In terms of surveillance and facial recognition. Here's the allowable things and, here's, things that you shouldn't be doing, so I would like more detail on that. It'd be great if we could get consensus on that, right? we'll never get complete consensus.

The things that we think are right in the US will be different than what they think is right in China. and that'll be. I think you're reconciled. Maybe we can reconcile the US with the EU, but we're not gonna get there, completely. I think you have to say what is it that you're trying to achieve?

How do you agree on these things? And then how do we implement a system that checks that? How do we put in training so that Engineers will build the systems that respect it. One of the things that was important to me was broadening the viewpoint. So I recognize that, as, software builder.

I was mostly thinking in terms of the user. and that's not enough, right? So, as the question says, you gotta think about society as well. And, I think of it in terms of three levels. So if we go back to this criminal justice system, there's one user and that's the judge. And if you're building a system to assist them, you wanna have a great interface and there should be fancy charts and tables and all the information in the right place so that the judge can make a good decision.

But you're not done then because the judge is just part of the whole system's implications on this. And so the next level is, who are the stakeholders and that's the defendant and their family. And, the victims of many crimes that the defendant commit, committed, or may commit in the future, and their families.

Those are all important stakeholders and you wanna think what's fair to them. And then there's society as a whole. You wanna think of, what are the effects of mass incarceration and, and bias of various kinds on the system. and so just optimizing the screen for the user. That's not enough.

There's a lot more left to do. 

Krishna Gade: That's a great point. I think it just spans beyond criminal justice. you're not building AI for the underwriter in the bank, but also the person who was applying for the loan and the society, and it just extends to so many things. you're not building AI for a physician, but like the patient and everything.

Amazing. So again, that's where it comes down to this whole aspect of how we create an incentive for organizations. To do this right? To build safe AI, do you think regulations have a place here? AI Act as well is coming. 

Peter Norvig: So part of it is saying, you should do this cause it's the right thing.

Part of it is saying you should do this because otherwise your business is gonna get in trouble of various kinds. and so I think there should be a multifaceted approach. I think there should be regulation, but. Regulation does not move at the speed at which technology moves. So it'll always lag behind.

And, regulators are not technology experts, so they may get some things wrong. So that can't be the only approach. I think internal self-regulation is important. You're seeing the tech companies do that in part because they think it's the right thing, in part because they wanna stave off the regulation that would make mistakes.

I think there's a role for technical societies, right? So the ACM and IEEE and so on, they have codes of conduct. they could do more of that in terms of education, so on. I don't know if certification would be part of that process. So right now, anybody can call themselves a software engineer and go out and build a piece of software.

You don't need any degrees or certificates or plaques on the wall or anything. That's not true in other fields. I can't say I'm a civil engineer, I'm gonna go build a bridge, right? I need proof that I'm allowed to do that, right? so I don't know if we need that for software, but maybe we want that as an optional approach.

And then another part of the puzzle could be, third party certification, right? So I just recently joined an AI safety group that's being put together by Underwriters Laboratory, and I think they're interesting because. A hundred and something Years ago when we had elastic technology that was going to kill everybody, it was electricity.

And there were these big public scares of what electricity's gonna do, and you could see these old cartoons of people getting electrocuted and so on. Underwriters Laboratory came along and said, okay, we're a third party. We don't have anything to do with the government, but we're gonna do an inspection and we'll put a little stamp on your toaster.

And if it says UL on it, the customer can be assured that it is probably not gonna kill them. 

Krishna Gade: So this was a private lab? Private lab? 

Peter Norvig: Completely private. 

Krishna Gade: Interesting. 

Peter Norvig: Completely optional to get certified from it. But, the companies saw that the consumer trusted the brand mark, and so the company said, it's worth it to us to get certified through this.

And so that could be another possible route. And maybe, maybe these third party independent companies can move faster than regulation can. 

Krishna Gade: Great point. We probably have another question. Are the business owners and shareholders also stakeholders? How do we reconcile doing the right thing for society and influencing investors in C-Suite?

This is very interesting, right? So recently I was at a conference where, There were customers who were telling us that AI is now like a board level thing, Unlike the last few things, like the big data cloud computing, it's, it seems like the boards are very much in influencing the C-suite in many large companies about AI strategy, and there's almost like this gold rush thing, right?

What are we doing? And you were articulating, everyone wants to create a chat bot. How do they need to think about it? I'm sure a lot of people are not aware of all these things. What we just talked about, how, what, do you think we should do here?

What do you think organizations should do? 

Peter Norvig: I think successful organizations worry about the bottom line, but don't obsess over it. And they think we're optimizing for the long run. We want to be here. So this quarter's profit is not the thing that I have to maximize, rather I have to build a sustainable company.

I remember Tim O'Reilly of the O'Reilly Publishers saying, revenue, I guess he said this number of years ago. So he said, revenue is the gas that makes the car run. Now maybe it's electricity that makes the car run. but he says A road trip shouldn't just be a tour of gas stations.

You're there for a purpose. and so I think companies should think first, what's our purpose? We're going to make society better while making a profit. but the goal isn't only to make a profit. 

Krishna Gade: Right, makes sense. I think basically ethical AI will, safe AI will result in actually building better AI products, and eventually it'll help the companies as well.

So rounding off. So there's basically coming back to like where we started. The foundation models are now the rage. And now there's open source availability of these foundation models. There are dangers of people even abusing them, potentially building ransomware programs, daily viruses, and whatnot.

what are, what do you, what's, your take here? how do we, how do teams think about it? How do we, I guess it probably just summarize all of the conversation, but what's your sort of, overall take in terms of technology, ethics, safety?

All of these things combined. 

Peter Norvig: So a couple things. One is, safety is really crucial and it should be an important part of everything you do. Secondly, I don't quite understand why there's so much emphasis on, some of these results, of saying, look, I can jailbreak a chatbot to tell me how to build a bomb.

But it was a lot easier just to type a search. How do I build a bomb? And that information was already out there and the chatbot resurfaced it. So it doesn't seem to me like that's a new threat. So the new threat would be if it is synthesizing information that's hard to find in order to do something disruptive.

And most of the examples I've seen before are not like that. They're just resurfacing something that's already out there. So I don't see that as a huge threat. but we do wanna defend against that and, we do wanna make things safe and, We have safety at a lot of different levels, we put cheesy little locks on doors that any professional thief could get through in a second.

But it keeps the casual person out right? And so I think a lot of this AI safety here is for keeping that casual person. Now the person who really wants to make the ransomware, we're not gonna be able to stop them anyways. So, how do you do that? I think, red teaming is really important to say, let's think about how people are trying to break these systems.

So the base level is the reinforcement learning with human feedback and the fine-tuning to sort build systems that are resistant to that. And then as new attacks come up, you should go in and make your system safer against them. 

Krishna Gade: Can you double click on the red teaming thing?I don't think a lot of people know what the red team is and what they do.

Peter Norvig: So it means, the engineers should say, okay, we're gonna try to build a safe system. We can, but when they say, Okay. It's passed all the tests we thought of. That's not enough. Rather you should have a separate team that says, my job is, to try to break the system.

QA the model basically. QA the model in every way I can. And you wanna bring in people that are experts in that. So the people that. Are making ransomware and so on and and know all the tricks and and let them try as hard as they can to break into the system. And then you go back and forth, And try to fix that. Makes sense. And then I guess the other thing is, I'm of two minds about this open source, right? So we saw in the beginning, companies and maybe it was a PR ploy saying, oh, we're not gonna release our models cause they're too dangerous. and one way to make customers want things even more is to say you can't have it.

there may have been some PR involved there, but I think it was a good idea to say you are only gonna get access to these models through an api. And that means, we can monitor what's being done. If we see something being done wrong, we can either fix that particular attack or shut the whole thing down.

If you open source a model, you lose the ability to do that. Now the bad guys can do anything you want and you have no way to monitor what they're doing. So I'm a little bit worried about the ability or the ability to take these models and work on them. Unconstrained. 

Krishna Gade: Awesome. This is great. Thank you for sharing your words of wisdom. Peter here, it seems like, organizations need to invest in AI safety and, responsible AI and, how do we make this, not just for the user, but all the society as a whole.

There's probably one last question that we'll wrap up. what's to prevent a developer from creating responsible AI that would be unsafe, biased, unhygienic, inaccurate, that could cause humans harm? In other words, how do we humans control AI? I guess it's probably just a summary of the entire conversation.

Any closing thoughts. 

Peter Norvig: I would broaden that to say, how do we control against malicious uses of technology In general. So a lot of the technologies we have dual purposes. and all our kind of. Energy technologies, right? So as a species we invent fire.

And that's good for a lot of things, but it can also be destructive. Exactly. And we invent vehicles that go fast. but then you can crash them into things. so, any technology can be put to bad use. We talked about ways to make it harder to do that, to put them behind APIs so that you can lock them up.

To put preventions against the casual user. but I think with any kind of technology, it's hard to have preventions against the professional dedicated user. So if something has a capability, they're gonna be able to use it. and, I'm not sure the extent to which AI makes that.

Particularly worse. So I think you were already able to do a lot of the things you could do. without the AI technology, maybe it makes it a little bit easier. 

Krishna Gade: Makes sense. Awesome. On that note, thank you so much for joining this webinar. We'd like to thank Peter. So there's probably three takeaways that I got from this. One is, essentially every organization needs to self-regulate, and have to put processes and tools in place so that they can. They could deploy safe, responsible AI. And it seems like there's potentially a need for third party ratification in the future so that AI engineers don't go and build their own things.

And, then obviously we all, as a society, we're responsible, teams are not building AI for a particular user but the entire society and have the deep ratification. but with that, thank you so much Peter, for coming on our webinar. It was a pleasure. And, thank you for everything that you do for us.

Peter Norvig: Great to be here. Thanks. 

Krishna Gade: Thanks.

‍