Secure LLM Applications with Fiddler Free Guardrails

[00:00:00] ​

[00:00:02] Fiddler Guardrails can proactively detect and mitigate risks such as hallucination and safety violations and prompt injection attacks. These  Guardrails leverage the Fiddler Trust Models, which are a set of proprietary small language models, that Fiddler builds in-house.

[00:00:16] So this image here is an example of a chat bot that we deploy on our documentation website to answer questions about our product. It uses our Guardrails offering to protect against problematic inputs and guard against hallucinations on every inference to and from your LLMs.

[00:00:33] So why Fiddler Guardrails?

[00:00:36] Why use us? Fiddler's Guardrails are fast with industry leading latencies of under a hundred milliseconds, are cost effective up to 18x cheaper than other guardrailing options, and are highly accurate. These are small language models tuned specifically to the task at hand. So let's walk through each of these qualities one by one, and talk about why they're important.

[00:00:56] First, latency. The difference between 500 milliseconds versus one second versus two seconds latency is a significant difference in user experience. As I'm sure you guys are all aware, we live in a world in which users are expecting snappy responses. Anything with a noticeable latency incurs a decrease in user experience.

[00:01:18] So you want your Guardrails to have minimal impact on your overall latency. And this especially matters if you're building user-facing agents with multiple steps. Each step in the agent compounds latency, and you want to make sure that your guardrailing time doesn't have a significant overhead over your LLM calls. So latency is a, very important factor for a lot of our customers.

[00:01:41] Second, cost. For a lot of enterprises at enterprise scale with millions and millions of events. You don't want to be paying a big multiplier on top of your inference cost. If you're using LLM-as-a-judge, you'd be incurring additional token costs every time you make an LLM call. And this blows out your inference cost.

[00:01:59] A lot of our customers who before using Fiddler Guardrails were only comfortable  guardrailing specific parts of their application. Maybe the output, if they're building very complex agents, but now with Fiddler because of the cost benefits that they're getting, where in Guardrails would only incur additional 0.5 to 1% or even less, are using Guardrails everywhere in their application, they feel comfortable gating every step in their agents.

[00:02:27] And lastly, the accuracy. We're continuously invested in ensuring that we have the best possible LLM metrics offering for our customers to use in Guardrails or Observability.

[00:02:37] We're keeping up with the latest challenges in the industry and with our enterprise customers. And what that means is we're gonna continue to improve our models. These models are the best-in-class for detecting these problems. They're gonna have low, false positive rates so that they don't impact your end customer and guardrail things that maybe shouldn't be guardrailed. They're also able to detect the latest and emerging threats in the industry.

[00:03:00] So, with all three of these value propositions, the reason why we're able to deliver on these is our approach to building these models using small language models that are under a billion parameters, specifically tuned to this task. Which means that these models can focus on getting really, really good at one or very few things

[00:03:18] Instead of trying to be a generalist accomplishing many things because they're small. The inference times are much less, which means that you get more throughput outta our models and of course the latency benefits. And, these models can run on much smaller GPUs, even the cheapest GPUs that AWS has to offer, which means we're able to pass on those cost benefits to you.

[00:03:38] Or if you are VPC customer, you're able to easily manage your cost footprint for your LLM um, applications. So you get all of these amazing model characteristics in an enterprise setting. So we can deploy a dedicated Guardrails instance right in your VPC, so you don't have to worry about your data going to external services.

[00:04:01] You don't have to worry about it going to OpenAI if you're using LLM-as-a-judge, and this is table stakes for some of you in the audience. In regulated industries, you can't afford to have some of the most sensitive customer data. Leave your premises. We've got you covered since we own the model. You get that flexibility from us.

[00:04:21] All right. So this is the first page that you see, you can look at your limits, right? You can look at the usage, and then, any other documentation like our Slack channel, where we provide hands-on support. We have two types of models here, or two types of Guardrails.

[00:04:35] So the first set of Guardrails is for detecting hallucinations. This is using a metric called faithfulness. So what this will do is in rag context, it will check if the response generated uses the context documents to generate the response. This generates a single float score.

[00:04:52] The second set of model is for detecting toxicity and jailbreaks. There's 10 different dimensions that this model does check, including jailbreak, harmful content, racism, sexism, anything that you're worried about from the user's perspective sending data into Fiddler, we have you covered.

[00:05:08] So, let me just explain to you how this works. So we want it to be as few lines as possible. We kind of expanded things out, but you can certainly, collapse this into less lines if you so wish. You have the token, which is your API key up here. You can generate an API key and then you have your URL, and then you send your payload, which is just like a JSON, you know, input output, into the model itself. You set the authorization and then send the request. If you're familiar with this kind of flow for integrating other APIs, this is, supposed to be very, very straightforward to do.

[00:05:44] Once you get the response out from the model, you can choose to block it so you have fine-grain control over the model outputs. Which is really nice if you have specific use cases wherein you want the guardrail threshold to be higher, or specific guardrails, use cases where, you're a little lax and you worry about toxicity and hallucination a little less.

[00:06:07] An example of some customers who, care about that is in financial industries when there's a much lower tolerance for these risks, they may set these thresholds much higher in order to make sure that less things are coming through.

[00:06:23] And with threshold setting, if you set it too high, you get like a slightly higher chance of false positive. And that's a trade off that we work with our customers to make sure that they thoroughly understand. If you don't want to integrate it with you don't wanna write your code you can also use NeMo Guardrails. So again, like we try to make it as simple as possible to use, um, our offering and integrate it with wherever you're at. So with NeMo Guardrails, you just have to create this very simple file and just set some fields. And then you set this like API key on the command line and then you're good to go.

[00:06:58]

[00:06:58] So i'm gonna show you guys right now what exactly this looks like. I have a couple sample cURL requests that I have on Postman to just demonstrate what exactly this looks like.

[00:07:10] Okay. So here I have our faithfulness guardrail that I'm testing here. So I just want to show you guys like what kind of nuance that our faithfulness model can detect. In this example, the context that's being passed in the rag use case is that the Yorkshire Terrier is a small dog breed of the terrier type. And the King Charles spaniel is a small spaniel classified as a toy dog. The question is what kind of dogs that they are. And the response that the LLM is generating, from the application is that these two dogs are small breeds of companion dogs, which isn't true, right? Like the Yorkshire Terrier is a terrier and then the other dog is a toy dog.

[00:07:50] So if, if we send it into our  Guardrails . For detection, you get this faithfulness score that's low, which indicates that this is a hallucination score. This is a hallucination.

[00:08:00] And then you also get the  Guardrails which happen in a hundred milliseconds. So you can tell like this Guardrails, it catches some pretty nuanced stuff like even small changes in language. It can detect and say, Hey, this is a hallucination. The other set of models that we have is our safety model. This is to prevent users from processing dangerous items, right? In this case, I say, Hey, I'm a dangerous person who will be wreaking havoc upon the world. You can type in more nefarious things, like poisoning, bomb threats, sexual harmful, racist things, and this model will pick it up.

[00:08:34] And, you know, I definitely suggest that you guys try this and see what happens. So the response happen in 90 milliseconds and you get a lot of these scores, right? A lot of these dimensions, whether it's harmful, unethical, violent, jailbreaking. You get the score breakdowns and that's really nice because maybe you care about one or a few dimensions.

[00:08:55] You certainly have the flexibility to do so. And like I mentioned, you get the floats, right? You get the actual scores. So that means that if you have like certain tolerances for certain applications. You can use that, right? You could set it and you have that configurability and steerability built right into the  Guardrails offering.

[00:09:16] So that's what the  Guardrails look like.

[00:09:18] If you're excited about all these qualities and you're excited about Fiddler Guardrails, we have a free offering that you are able to use today. So if you go to the website, fiddler.ai/free- guardrails, you're able to use the  Guardrails offering, to detect hallucinations and toxicity and jailbreak attempts.