Transcript of Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS
AI Engineer
0:00Hi,0:01today we are going to talk about how to0:04stop AI agent hallucinations0:07with five techniques beyond the prompt.0:11Each one is a code change, not a prompt0:15change.0:16Let's see.0:18Every time your AI agent responds, you0:21are paying for the words going in and0:24the words coming out.0:26And your bill, you will see those0:28calling tokens. Yeah.0:31And the more tokens you are sending, the0:34more you pay.0:36And if what you send is not quite right,0:39too much, or missing something0:41important, your AI agent start to0:45hallucinate.0:46There are five techniques0:50to help reduce tokens waste,0:52improve accuracy, and catch failure0:56before you submit them. And each one is1:01a code change,1:02not a prompt change at all.1:05So, let's see it.1:08First, we have semantic tool selection.1:12You filter which tool go into context on1:17every call. The model only see what it1:20needs for that specific query. Second,1:24we have graph rack.1:26For precise queries like aggregation,1:29counts, multi-hop reasoning,1:32and you replace the text retrieval with1:36a structured graph query.1:38The model gets a compute a verifiable1:42answer, not a sample1:44as rack do.1:47Three, multi-agent validation. A second1:51agent can check every response before it1:55reaches the users.1:57And four,1:59neuro-symbolic guardians,2:01you rule life in Python, not in the2:04prompt, and the model cannot escape2:07them.2:08So,2:10five, runtime guardians, because if you2:13don't want to block, you can steer, and2:16you don't you don't need to block2:17everything. And when a rule fires, the2:21agent self-correct and complete the2:24task.2:26No hard stop, no user advice. So, for2:29each technique, I will show you the2:32agent without it, and then with it, so2:36you can compare. And all the demos2:40uh I'm using a travel agent that I built2:43using a Strands agent.2:45And a Strands agent is a open-source2:48agent framework that we maintain on AWS.2:56And2:57I am Elizabeth Fuentes Leon. I am a2:59developer advocate for AWS. I'm focused3:03on agentic application.3:05And here, in this QR code, you will find3:09everything that you will need3:12to recreate all these techniques that3:15I'm going to show you in a moment.3:21So, let's get into3:26Let's get into it.3:31So,3:32semantic tool selection,3:35our travel agent has 29 tools.3:39Flights, hotels, payments, weather,3:42cancellations, all all our dummy tools3:47are not like a travel agent for real,3:49but3:50every time that user sends a message,3:54all the 29 tools description go into the3:57context windows.4:00The model reads all of them before4:04deciding what to do.4:07And if your agent has memory, the rows4:11to every conversation adds more context4:16that gets sent with every single4:18message.4:20And you pay for every single one of4:24those tokens,4:25whether the model ends up using the tool4:28or not.4:30To understand where those tokens come4:33from, you need to see what a tool4:36actually looks like to the model.4:39In a Strands, you write a function with4:42the4:42tool decorator4:44that's the tool,4:46and a name, a description, and a4:49docstring4:50typed parameters.4:52Then, Strands take that and generate a4:55schema with name, description,4:58parameters, and the schema is what goes5:02into the context windows on every call.5:07Each tool schema is about5:1017 or 200 tokens,5:13depending of how many parameters it has.5:17If our travel agent [snorts] has 295:20tools, that adds up to somewhere around5:243,000 tokens per call,5:28just for the tool description.5:31Before your message, before the5:33response,5:35every single call.5:38By creating a tool database,5:41we can filter the tools that the agent5:44may need before the agent the symbol.5:48With this filter, the model sees only5:52three most relevant tools.5:55Tokens usage drops from thousands to6:00fewer than 300.6:02Let me show you in the code.6:05Here in my cure ID, this is the Let me6:09clear all the outputs.6:12So, first we need to install the6:15requirements.6:16Here is the requirement file. This is a6:19Jupyter notebook because it's more6:21simple to show everything, but there is6:23one application that you can run if it6:25is more comfortable for you.6:27So, here are the requirements. I have6:30the A strands agent and because I'm6:32using open AI as a model invocation, I'm6:35using the API from open AI.6:38And6:39I need a strands agent for open AI.6:41You can use open AI strands agent with6:44all our my god, almost all the model6:46provider. And with a strand we Amazon6:49better of course because we as AWS, we6:52are the maintaining of this framework.6:55And to use a Amazon better, you don't6:58need to add7:00the model provider.7:02And7:03because I need to embed in I need to7:06create embeddings for my vector tools7:09database, I'm using the sentence7:12transformer. This is a super simple7:14model that runs locally, so it's free.7:17So, you want you can run this locally7:20you computer without expense another7:22model embedding.7:24And7:25you can use a strands with all llama7:28too. So, if you have a local model, you7:31can run everything7:33in this in your computer for free7:35without expense any tokens.7:38And7:39I use it files as my vector store,7:42local, super simple.7:45And well, right now I'm going to use a7:47Neo4j. This is for other7:50demo that I'm going to show you in a7:51moment. And because I'm using some7:54environments, I'm using Python of7:56And let's see. So here I want to run7:59this. Let me put it bigger. I know that8:01you have it some problems.8:03Bigger. Little more little less.8:06And here, this is better.8:10So.8:11I style my requirement. I already did8:13that. And I'm using OpenAI, so I need my8:16my API.8:18My API key. API key.8:21And here, I invoke my strands. I need8:24the agent because I want to be like8:25agent that I'm using OpenAI. And I have8:28a bunch of tool dummy tools here. I let8:31I going to show you in a moment. And I8:33want to use this because I have some8:37um8:38functions that I'm using to8:41uh create the build the index for my8:43vector store. And I need the search8:45tools when I'm using the vector store, I8:48put my query there. I search for the I8:50search for the tool that I'm going to8:52use using vector um8:54um search for vectors.8:57And I swap the tools. I going to show8:59you that in a moment. So let me show you9:02all my dummies tools. Where is the9:04engage register here.9:08Here I have all my tools. Uh this is for9:11swap. Search, build index. And where are9:14my tools? Shoo shoo shoo shoo shoo here.9:18So here are all the dummy tools I9:20created for this. You know, it's super.9:22This is a demo, please. It's not9:24something that you can use to go to9:27production. No, please.9:29And okay. So where is this this one? So9:32I run this.9:34And I build my semantic index. I already9:37did that via Yeah. So, I have 29 tools9:41here, 29 dummy tools, and I want to test9:44this with a lot of different queries,9:47and this is the ground truth. I have a9:49ground truth because I know what is the9:52best um9:53the best tool to answer the question, so9:56we can know if our agent is okay or not9:59okay. So, let's run this.10:02We have 90 queries and 29 tools. So,10:07this is some helper functions. I don't10:09care.10:10Yeah, I care, but I don't want to10:12explain you that. So, this is my agent,10:15my traditional agent, where I'm putting10:17all my 29 tools inside the agent.10:20I have my model, some helper function.10:24This is my agent. I only This is the way10:27I create a agent using a strange agent.10:29I put all my tools,10:31and my system prompt is you are a travel10:34assistant. You select correct tool to10:35answer question. Oh my god, super.10:38And the model.10:40Let's run this.10:41And this is whatever. This is some10:44function to count the tokens because in10:47inside this strange agent, you can you10:49can count the tokens. You You can know10:52how many tokens are using for in and out10:55the the agent.10:57This is some helper function, and yeah.11:00This is Every time that I put a11:03uh11:04a prompt inside this agent, I'm only11:07sending the prompt, and the agent11:08respond. Then, I I use a new agent11:11because it's a11:12a fork. So, I'm not having a11:14conversation with the agent. I only send11:16a uh11:17a question, and I receiving an answer.11:20So, for each question, it is spend11:22around 2,000 tokens.11:25So, in and not always it give me the11:28right answer.11:30Here. So, yes, I have um I'm not so good11:34at accuracy, and the average is 111:381,00011:39tokens. Now, let's use my new AI with11:42the semantic approach.11:44The thing The first thing that I do is11:47for every single query, for every single11:50prompt, I send the query to my search11:53tools, and I going to receive the top11:56two the top K three11:59most relevant12:01with more probability to response my12:04query because this is a semantic search12:06inside my vector store.12:08And12:09then I use that response. I going to12:12select that name, and I going to put12:15that tools inside my AI. So, I only12:19going to use the three tools that this12:21semantic search12:23retrieve me for my AI, and I send that12:27only tools. And this is my function12:29helper, and that's it.12:31So, I going to send the all the queries12:34again the same. And for the first12:37question, I have 4,000 tokens three blah12:40blah blah. And yeah, we have a huge12:44different because I don't send the all12:47the 29 to tools in all the queries.12:51So, yeah, bam bam.12:53And12:55okay, semantic memory. Yeah, okay, now12:58come on. Finish. So, let's let's let's13:01go to the next one. So, this engine is13:05only sending a question, and I receive a13:07response. I don't have any conversation13:09with this engine.13:11What happen if I start to have a13:13conversations? If the engine remember13:15me? So, we need to we need to send only13:19the tools that this engine is going to13:21use. Because if I give the all the tools13:24I'm using in the conversation history,13:28uh then I it's going to be in a moment13:29that I want to have the 29 tools inside13:32my engine. So, we are not resolving the13:34we are not having a a13:36um13:37We are not resolving our problems in the13:40engine when we having a conversation13:42with the engine.13:44So, we need to put the tools and then we13:46need to remove the tools.13:49And we can do that with the swap tools.13:53And let's run this. And what swap tools13:57I got run this, no. Let me Let me show14:00you first the swap tools14:03functions here the register.14:06I delete this14:08because with a strands14:11in each invocation, you have complete14:14control of the status because this is a14:17a gentle loop and you can14:20we use this a gentle loop14:23how for14:28because this is a gentle loop and you14:30can do14:31almost whatever you want. You can put14:35and remove everything inside the loop14:38with some few lines of code. So, let's14:40see.14:41Here we have the engine. We have the14:44tool register through your through tool14:46registers.14:47It's something inside the engine status.14:51So, we can clear the tools14:54and that's it. So, in the next14:56invocation, we can clear the all tools14:59and we can add the new tools.15:03Here, where is my I have a lot of things15:05here.15:09So, I run this.15:16It's going to take a moment.15:28>> And yes, we can see that in each15:31invocation, my amount of talking is15:34increasing.15:35Why I have more? Because I'm sending the15:39chat history as well. So, I send the15:41tools that I need, only the tools that I15:44need, and the chat history. So, that's15:47why I can see that my amount of token is15:51bigger and bigger.15:59And the see,16:00the accuracy is better. And yeah, we16:04have more tokens.16:08So, yes, probably we don't have the best16:11accuracy here because this is a super16:14demo with super dummies tools, and some16:17queries are ambiguous on purpose because16:21search for something, for something,16:24check something. And the demo has a16:27generic tools, dummy tools,16:30with some similar name.16:32And when all the 29 tools are visible,16:37the model sometimes pick the wrong one.16:40And with the filtering, those generic16:43tools uh only appears in the query16:46actually match them.16:48So, you can run everything and16:50you can test this. So, this is only that16:54run only locally. But, what happen when16:57you want to put this demo in production?17:00You can of course, you can build a17:03bigger, better store. I don't know, you17:05can use Postgres. I think it's too much.17:08But,17:09we on AWS, we have17:12Amazon Bedrock Agent Core.17:14So, Amazon Bedrock Agent Core is a17:17service dedicated only to agents in17:21production. And in size of Agent Core,17:24we have Agent Core Gateway. So, Agent17:27Core Gateway Gateway17:30allows you to build this index17:33[clears throat]17:34uh17:35alone. So, you only have to say, "Hey,17:39this is my tools." And Agent Core17:41Gateway is going to build everything for17:43you. And it can have the17:46vector search inside. So, the routing17:48layer is inside the Agent Core and it17:51handles the tool selection17:53automatically. So, you register your17:55tools once and it find the right one for17:59each request. It's the same principle18:02but without infrastructure to manage.18:06Now, let's see the next one.18:10GraphRAG.18:12You know,18:13RAG is retrieval augmented generation.18:16Is how agents access your own data.18:20You take the user question, search your18:23documents for the most similar content18:26using vector search and pass what you18:29find to the model.18:31The model answer from that, it's work18:34well for open question.18:37Find me something about this topic.18:39But there is a category of question18:41where that breaks down.18:44What is the average rating across all18:46hotels in Paris? How many hotels have a18:50pool?18:51Vector search always returns something18:55even when nothing is truly relevant. And18:59the agent only sees the top end chunks19:02of your all data at a time.19:06It cannot aggregate, count, or traverse19:09relationship across all the full data19:11set.19:12So, it estimates.19:15And it present that estimates as a real19:18fact, you know, a real answer.19:22So, here we have graph19:25you build a vector store and it19:27retrieves three chunks from 30019:30documents and the model guesses.19:34If we are using graph rack, you can run19:37a query across all the and returns a19:41compute results.19:43Graph address this differently19:47because instead of a retrieving text19:50chunk, you can build a knowledge graph19:54from the documents. Knows relationship19:57and structured data.20:00For the demo, I'm going to use Neo4j20:04locally and the model is going to write20:07a Cypher query to search it.20:10Cypher query is Neo4j query language.20:14It's similar to SQL. So, the graph run20:18that query across all the data. And the20:21model gets back a compute verified20:24results,20:25not a sample.20:27And before running this demo, I'm going20:31to start the dependents. Let me show20:33you. Let me Let's go to the code.20:37So, graph20:39This is the notebook I'm going to share20:41with you. So, we need to install the20:43requirements again.20:45What we have here, let me see.20:48This is the first one, yeah.20:50Requirements.20:51I have I'm going to use OpenAI again.20:55Let me just close this. I'm going to use20:57Neo4j and we need Neo4j for graph rack.21:01And for the21:03engine that is using normal rack, we are21:06going to be a super simple21:09uh vector store in files and we are21:12going to use again the sentence21:14transformer.21:15And let's see. I already have all right,21:17I invoke [clears throat] my open AI.21:21This is my tools. I I because I want to21:24create some tools here, I want to use21:26open AI and I use in graph database.21:30And this is my I have my Neo4j locally.21:34And let's run this.21:37And this is something to check Neo4j.21:39So, I'm building my21:41uh vector store. This is my files and21:45this is my Neo4j that I have here21:46locally. I don't know if I can show you.21:48I think not.21:49And I have a tool, a normal tool created21:53with a decorator. This is to search21:55inside the21:57my vector store. You see that's super I21:59I have the query. I I create I I create22:02the the embedding for my query and then22:05I search inside the vector store and I22:07have the query for the knowledge graph.22:10So,22:11here the model22:12need to understand that to search in the22:15query in the graph knowledge base, it22:18need to build a Cypher query. So, I put22:21that in the context because we already22:23know how to build uh tools, right?22:25I put that in the context. I have my22:28driver to send the data to uh to connect22:31with the vector store and to send the22:34Cypher query that the model is going to22:36create for me. And this is to read the22:39results and that's it. That's the only22:41tool that I need to search in my22:43knowledge graph.22:44Okay, this is my model22:46open AI. This is my rag.22:50My rag engine and uh this is my graph22:52engine. I have two different engine to22:54compare the results.22:56Let's see.22:57Well, I have a problem. Yeah, I'm ready.23:00So, let's do this round the first test.23:04Aggregation.23:10Here, what is the question?23:13What is the average guest rating across23:15all hotels in Paris?23:17So, meanwhile this engine is Oh, it's23:20already done. So, what I have here23:23I have the average guest rating across23:25the hotels listed in Paris is23:28Um it calculate, you know.23:31When something When I'm using rag, it go23:34to the to the vector store, it receive23:37the n uh possible answers for this23:40question. Because this is an23:42aggregation,23:44it's going to use the data that it23:45receive to build a mathematical23:49uh operation. This is the thing that I23:51have here. So, everything that we see23:53here is the is the model reasoning.23:56So, when I run23:58the graph engine on the mean,24:01the engine is only going to give me the24:03answer, you know, only these tokens24:05here.24:06Because the Cypher query, it can give me24:09the mathematical operation itself. It24:12can do that. So, we don't need the24:14engines, the LLMs, the model uh do that24:18for me because the Cypher query already24:20give me the right question. So, it is 4724:25and the average. So, here in this uh24:29first we are lucky24:31because probably they are Yeah, only two24:34hotels. But, what happened if this24:37vector store have24:39more than two or three hotels in the in24:43the vector store? So, we are not going24:45to give a real answer. It's going to the24:48LLM is going to calculate with the only24:52three answer that it receive. So, it's24:55going to build this operation with only24:57three hotels.24:58But, if the vector store have more than25:02three,25:03we are going to have some problems.25:05Something that I am you know uh25:07yeah.25:09So, let's run the next one. The prices25:11counting.25:13Something similar too. How many hotel25:15have a swimming pools as a amenity?25:19How many again? So, it give me the the25:23traditional25:25It's appear that the search did not25:27return some specific information about25:29hotels in Paris. Okay. What happened25:32with the other one?25:33There are currently no hotel that offer25:35swimming pools.25:37No I. There is no hotels. So, the other25:40one is like mm25:42would you like to ask about hotel with25:43other specific amenity or information?25:46It's like mm maybe it is or maybe I25:49don't know.25:50Yeah, it's not so accurate, okay?25:54So, mostly conversation.25:58Oh, what is my question?26:00What are the room types and price for26:03the highest rating hotels?26:05What are the rooms types and price? This26:09two question.26:10So,26:11it search the fact I had had only one.26:14The highly rated hotels in any company26:16Paris blah blah blah. I I don't speak26:18French.26:19With the guest rating is that one.26:22However, I currently don't have Okay,26:25blah blah blah. A lot of data there.26:27So,26:28what happened with the other one?26:30Receiving notification for Oh, I have a26:32error.26:35I hear in the second one I receive the26:38the question. The highly rated hotel is26:39harmony blah blah blah. They offer26:41following types. But, unfortunately26:43these rooms are not available.26:45So, I receiving a26:48answer. I don't receive a lot of blah26:50blah blah there. I only receive the26:52answer what I need.26:54And for the next one, how was the26:56meditation?26:59What is the question? Tell me about27:01hotels in Antarctica. A spoiler, there27:04is no hotel in Antarctica. There is zero27:07hotels in Antarctica and let's see.27:10It's appear that the certain order to27:13specific information about hotel Okay,27:14because they are I search for the other27:17have the tail blah blah blah in the27:20If you are looking for particular that27:21experience specifically inquire about27:23visiting Antarctica.27:25No.27:26Please let me know and I can assist you27:28for it. Okay, a lot of tokens that I27:31spending there in that answer for Elena.27:34What happened with the other one? They27:36are currently no hotel listed in27:38Antarctica. Of course, because it create27:40a cyber query, the cyber query and it27:42receive zero.27:45So,27:46it no, it give me a honest answer.27:49Here's summary that close grade for me27:52for this Neo4j notebook. But here is27:55something that I want to show you before27:58go to the other one.28:00So,28:02here something that I love for Neo4j28:05because why I'm using Neo4j? Because in28:07the library28:09the Neo4j give me28:12uh it can build28:15it use a LLM. I'm using the OpenAI as28:18well to build the knowledge graph. So,28:22how I build my knowledge graph? I only28:24have a bunch of data, a bunch of txt28:29only text and I send that data to the28:32Neo4j library that I miss out.28:36Okay, I send that data to this28:38and Neo4j using this all this um library28:43just right here, it can understand all28:46my data and it can build the graph for28:50me. So, I don't need to create that28:54using the simple knowledge graph28:57pipeline inside the knowledge graph28:59library. So, that's why I'm using Neo4j.29:03It's amazing. It's super simple to use.29:06So, I invite you to use it at that. So,29:09let's go to the next one.29:13So, multi-agent validation.29:17Sometimes an agent fails, and nobody29:20find out.29:21It calls a tool, and the tool returns an29:23error, and the agent does not surface29:26that error.29:28It generate a confidence success29:30response instead.29:32The user thinks it work. You think it29:35work. It didn't not. The agent acts and29:39validate its own output in the same29:42loop.29:43There's no separation, no second29:46opinion. So, when something goes wrong,29:49it rationalizes and tells you it's okay.29:53It work.29:55Here is what happens. Inside a single29:58agent, when it fail, it calls the tool,30:02gets an error, rationalizes30:07it, and returns a success response.30:10The user never sees the error.30:13You can address this by adding a30:17validation layer. You can have three30:21agents in sequence.30:24One acts, one checks, and other approve30:28or rejects.30:30Strands agent has a built-in class for30:34this called swarm.30:36It manage the handoff between agents30:40automatically.30:42You just30:43the role of each agent in a system30:46prompt.30:48And let me show you that.30:53So, this demo only needs a Swarms agent30:56with open AI integration. Let's see the30:58requirements here.31:01All you need of is Swarms agent. We31:03don't need anything more.31:04Let's go to the notebook here.31:08And31:09the key important thing here is the31:12already running is Let me see here. Is31:16the Swarm.31:18The Swarm it's what lets you to connect31:21multiple agents together31:24without creating like a for a while31:27manually to put the all these agents31:29together. It can build a chain and31:32manage the handoff between them31:35automatically.31:37So, here we are going to create three31:39agents.31:40Let me go down because we have the31:42normal and we have some ground truth31:45data.31:47So, here is to create a a single agent.31:49So, we have we are going to test three31:52different scenarios to validate a31:53booking. This They we expect true to31:56enable hotels. We expect false and31:59non-existing hotels and missing booking.32:02We expect false, too. This is how we32:04created single agent. We need a prompt,32:07some tools.32:08And we are using the tools that we have32:10here.32:12And here we know that this is the data.32:15So, the agents can can give out the32:18answer that we are looking for.32:20And that's it. Let me go to the other32:22ones. Oh, wait here.32:25So, how we build the Swarm?32:27I already run this. I want to show you32:29this one. So, we are going to build32:32three different agents. The executor,32:34the validator and the critic.32:37The32:38we have a system prompt. You are32:40executor agent for a hotel booking32:42system.32:43>> [clears throat]32:44>> Use the provide tools to fulfill32:47requests accurately and blah blah blah.32:49The validator is you are a validator32:51agent. Review what the executor did and32:54output exactly on off.32:58And we have the critics that is going to33:00set here approve or no reject.33:03Let's run this.33:05And oh, what is here? Our model didn't33:07define the model. Sorry.33:09Sorry, please. Forgive me the life33:12because I didn't run this one.33:17Let's run now because this is the33:20the normal.33:21And the model is there, right? What?33:25A little bias. Run that.33:29What's happening with you? Let me go33:31this. Let me copy and paste.33:33This one.33:34I don't know why this give me problem.33:39Multi-agent swarm is the time the same33:41name. Come on.33:43Yeah, you can see this is life. I don't33:45going to edit this.33:47So, yes.33:49Okay, we have33:52Thank you. And the swarm, how we how we33:55create the swarm? We have swarm and33:58the swarm function, we put all the34:01agents together and the entry point is34:04the executor. So, all everything is34:06going to start in the executor and then34:08it's going to hand off.34:10And six time because you can handle34:12that, too. So, get the swarm final34:14response. Let's do a for and they send34:17all the response.34:18And book the Grand Hotel.34:22I have booked a Grand Hotel for Alice34:24tonight. Booking blah blah blah. Hand34:26off.34:27Valid. Okay.34:29Looks good.34:30Critic, approve.34:32So, the critics is approving the this.34:36This is the other one.34:38And we can run some comparison here if34:40you want.34:43So, here we can see that the swarm34:46handles the flow between the between all34:50the uh three different saying. It was34:54what happens when the single agent tries34:56to conflict something that does not34:58exist in the system.35:00And now the same request through the35:03swarm, the executor gets the error, the35:06validator catches, the critic rejects35:09this, and the user never see a35:11fabricator response.35:13So, you can we can test here with the35:15single agent that it's a suggested35:18unknown entry and return success.35:23And the swarm, it have a executor that35:26go the error, validate saying, "Hey,35:29come on, man. Hallucination and the35:31critic they reject." So, the user sees a35:34clear failure.35:36The fabricator confirmation so missing35:39the issue here.35:41And you can run this to compare the two35:44different agents. So, here we can see35:48that the single agent it have everything35:51like a yeah, it's okay, it's correct.35:53But in the multi-agent swarm, you can35:56see that the last one is a you know,35:58missing issue here. Something is36:00happening.36:02Now, let's go to the next one.36:05Okay. Neuro-symbolic Guardians.36:09You have a rule for your agent. Let's uh36:12say maximum 10 guesses per reservation.36:15You write in in the system prompt.36:18You even write in in the tool36:21description. And the agent still calls36:24the tool with 15. No, because it is36:28ignoring you.36:29Because prompts probably are36:31suggestions, not constraints.36:34The model process them as a text.36:38Not as a logic it has to execute.36:42It's probabilistic.36:44Only code execute logic.36:47A rule in the prompt the model read it36:50as a suggestion.36:52A rule in the code the model can not36:55escape it.36:56Neuro-symbolic guardians rules put the37:00rules in the code. E-strands A and has a37:03few that call him hooks.37:06The the function that is trans called37:09automatically at specific moments in the37:12A and loop.37:14It is this case right before a tool37:17executes. So you have a hook.37:20You write the rule, check the parameters37:23and if they fails you cancel the call.37:27Let me show you this in the code.37:31Here we have the Jupiter notebook.37:34We have the requirements here.37:37Um what is the same requirements as the37:39previous demos we only need I will37:41install everything again.37:42We only need E-strands and open AI. We37:45don't need anything else.37:48And um my37:51API.37:53And here is the thing important.37:56In this demo37:57are these three hooks provided in this38:00trans base class. So you38:03allows you to create the hook. So we38:05have hook provider, hook register and38:07before to call event.38:10And the tool register is what E-strands38:13pass38:14you to register your call back. And38:17before to call event is the event that38:21fires every time the model is about to38:25execute shadow.38:27That last38:28one is what makes it possible the38:31intercept the call before it runs. So,38:34you have the before two calls even and38:37call we have after two calls even that38:39we are not going to use that here.38:42And38:43let's see the rule in the prompt.38:45Remember, the model reads as a text and38:49it may follow38:51or not follow them. So, let's see this.38:53I want to run this.38:55So, here we have some similar state38:58that we are going to use in this engine.39:00And39:01let me this is the symbolic rule.39:05So, booking rule if if rule booking rule39:08as a this is something that39:11we create. Let me show you the rules39:13here. So, we have39:15the rules.39:17Rule one.39:18Validate dates. Check in must be before39:21check out. If what is validated they39:24check out.39:25This is something that it check. This is39:27a something that is going to invoke when39:31the rule is39:33is uh39:34is must have to use it.39:36And39:39uh to do39:42We have another rule for max guesses.39:45Maximum take guesses per booking. So, if39:48I want to use I don't know I if I want39:51to book in uh39:52uh39:53I want to do a booking for 11, it's39:55going to block me. It's not going to39:57reject the booking.40:00And we have some confirmation rule.40:02Payment before confirm. You know, you40:04can confirm if you don't have the40:06payment. Cancellation rule.40:08Cancellation within the cannot cancel40:11within 48 hours of check in. So, this is40:15something that I've created here, you40:16know, to this demo.40:18So, I have booking rules and I have40:20confirmation rules. I don't I use in40:22here the cancellation rule.40:24So,40:25create validation hook.40:27We create the validation hooks here,40:30right?40:31Like this, neuro-symbolic rule, and we40:34use the hook provider.40:35So, we have a bunch of code here.40:38Then, we add the booking rules, the40:41confirmation rules, and the state.40:45And you can check this by yourself40:47later, but we want to see the demo40:50running. So, that we define the claim40:52tools that we are going to use them for40:54the booking hotels and the process40:56payment.40:58This is the the normal tools, you know,41:00and we need to add that tools to the41:02hooks. I think I add that.41:07Yeah. Tool name, book hotel. So, this41:10thing is going to get the eight when the41:13book hotel is using the41:19Yeah, the book hotel.41:21Okay?41:22So, let's create the agents for the41:24comparison. So, we are going as the41:27other demos, we have three different41:29scenarios. Confirm booking without41:32payment. Payment must be verified before41:35the confirmation. This is the rule that41:36I must trigger this.41:38We have booking hotel exceeding guest41:40limit, and we have valid booking for41:42five guests.41:46So, yeah, we have the three scenarios.41:49And we have the normal engine, the41:51normal engine, and the vaseline engine.41:54And we have the engine with the neuro41:56the neuro-symbolic guardians. So, we41:58have the hook, the neuro-symbolic hook.42:02Uh this is the same. Here, yeah.42:05And42:07we add this42:09If you can see, the normal engine42:11[snorts] is only three lines. The tools,42:13the model, and this is the line that42:15give us the difference between these two42:18engines. So, I already run that.42:21Now, let's go to42:25confirm a booking without payment.42:29So, I run this my engines. The first one42:32is42:32Abis. Come on, man. I Yeah, because I42:35Yeah, it confirmed the booking without42:36payment because it doesn't have any42:38rules and the promise is super basic.42:41And the other one is42:44Let's open this and is42:47booking blocked. Payment must be42:50verified before the confirmation. Thank42:53you. So, it's okay.42:55Now, let's run the other one. Test42:57second scenario.43:01What is the question?43:03Booking hotel access guest limit. So,43:06the question is I Yeah, I have more than43:10So,43:12here43:14the hotel Abis successful booking for43:15Yeah, because the promise is basic and I43:18don't put in any rule there. And for the43:20other one, it seemed that the Grand43:22Hotel has maximum capacity of 10 guests43:25per booking. Additional booking must be43:27made at last 1 day in advance. I don't43:30remember the day. Probably This is43:32something that they give them.43:34And to validate booking is both engines43:37execute all rules pass.43:39Because they Yeah, it's only a43:40validation for booking. So, book hotel43:43for blah blah blah. All rules passes.43:47It seemed that the booking is still43:48needs to make.43:49I don't I didn't remember the question.43:52So, booking43:53is made provide I Yeah.43:55Okay.43:57Okay, I get it.43:58And43:59Okay.44:01Now, run all the thing this question.44:04Confirming without payment, booking save44:06max.44:06And what happened here? This is all the44:08scenarios that we just run. So, this is44:10awesome thing to compare the results.44:12So, Bali booking king secure five44:15guesses, allowance, correct, wrong.44:1950,44:20blocked, wrong, correct.44:23Confirm, so there we have here the44:27comparison between the two ends.44:33So, what we have what we have here is44:36same model, same tools, same prompt.44:41And the different that outcome because44:44the rules are in Python knowing the44:46prompt.44:47This pattern of enforcing rules in code44:51before the tools run44:54is also what Amazon Aion core policies44:57service that we have44:59does at the structural level. So, the45:02same concept but managed for you in45:05production. And you only have to create45:08the rules. So, the hooks are all or45:11nothing.45:12They block everything or approve.45:16But sometimes you want the agent to45:20adjust and keep going. No and stop and45:24leave the user waiting.45:26That is what I will show you in the45:28next.45:32Run time guardians, run time is still45:36is still don't block. This is the next45:39one.45:40Hooks blocks unconditionally.45:44The agent is stop and the user has to45:47retry.45:48For a hard constraint, that is exactly45:52what you want. But sometimes the rule is45:56soft.45:57Maybe room fits for four guests, but a46:01group of six could go book two different46:06rooms.46:08Or a flight is full, but there is46:10ability on the next one.46:13You do46:14No, you don't want to block everything.46:17Probably, you want the agent to find a46:20option and complete the task.46:23That is a steering.46:25Here, we have the hook that fires and46:29the task failed.46:31And with agent control, it steered the46:34models and the task complete.46:37The other different here is operational.46:40Because with the hooks, it's uh changing46:43a rule means changing code and redeploy46:46the whole harness, whole the engine. And46:50with the agent control, which is the46:52name of the open source library that we46:55are going to use here,46:57rules are registered on a local server47:00via API.47:02You open the agent without touching the47:05agent code because the agent picks them47:08up immediately.47:10Let me show you that here in the code.47:14Yeah, this is the notebook.47:17In this demo, we only need a extra47:20packet.47:21The47:23agent control is the key.47:26So, the agent control is the one that47:29helps us to create a steering. And here,47:33we are using the setup control.47:36This is a a little application, a little47:39app that I create with the local server47:43and the steering rules. Here, we have47:47the local server. We have the control.47:50They [snorts] are the steering rules.47:51First, we have the47:54the steer master guest.47:56A steer, you know, guy. So, guy agent to47:59reduce a guest count48:02when exceeding maximum of 10.48:06And he's going to steer.48:08And we have some control that deny. For48:12example, deny no payment. Block blocking48:16confirmation without PR payment.48:20And48:21here, well, this is to48:23to create the48:25um48:25the service. The service. And let's go48:28to the48:30uh48:31notebook. So, I'm ready.48:33Um48:34Let me48:35go here. Yeah. So, I need the48:38environment.48:40So, this is48:41the agent.48:43Um48:45I'm going to have a error here. Wait.48:46Wait. Wait. I'm going to use bedroom.48:48Wait. Wait.48:50No, then let me comment this one because48:52this going to give me a error.48:57And let's run this.49:00Yeah, so we have the hooks as we did the49:03demo.49:04So, we have book any company Lisbon and49:07we have some prompts.49:08You are a hotel booking assistant. When49:11booking, first describe what you will49:13book in blah blah. And this is the49:15prompt for my agent. Where is my agent?49:18Agent control. This is for the49:21Well, this is on some helper. And this49:24is my hook that you already know because49:26we create this. So, we have the system49:28prompt.49:29And we have the hooks. So, let's test49:31this agent with the49:34uh book any company Lisbon for 6049:38guests. And if you remember, this only49:40can book for uh49:43less than 10 guesses.49:46And of course, he's blocking.49:49Now, let's go to the AIN control new49:52agent.49:54Here we have49:55two key importances or two key imports49:59that are important for the AIN control50:01SDK. So, we have the AIN control plugin50:05that captures AIN events and sends them50:08to the AIN control server. And we have50:11the AIN control student handle that50:13listen for a student decision for the50:16server and delivers them back to the50:19model.50:21Together here50:24they are50:25what connects a student to the AIN50:27control student logic.50:29So,50:31let's go this with this.50:35So, the student agent50:39is going to a river room for any company50:43list board 50 guests. Let's see what50:46it's doing. Yeah, I have to set for50:48booking for a stay in the company list50:50board for 50 guests50:52for May.50:53So, it's the reservation have been split50:56into two rooms. So, it took that it in50:59self.51:00You know?51:01One room for one and other room for five51:05and that's it.51:07So, use hook for hard constraints.51:11AIN control for software [snorts] rule.51:15Now, we have51:16five technique or running locally. But,51:22how do you take this to production51:24without maintaining service, without51:26building infrastructure?51:29Let me show you how.51:33Everything that I just built here in the51:36previous demos runs locally.51:39For the production version, Amazon51:41Bedrock Bedrock51:43Amazon Bedrock AIN core give you the run51:46time.51:47A gateway,51:48a short term and a long term memory, a51:52cloud watch observability, build in no51:56servers to manage.51:58Here the architecture,52:00the strands engine runs inside the run52:02time. And you know, you inside the run52:05time, you can put every framework that52:07you want, not only run a strands.52:10The gateway rules tools calls to the52:12lambda function automatically as a52:15tools. And the steering rules from the52:18previous demo live in the DynamoDB.52:22So, you change them there and they are52:26live on the next call.52:29And52:30you don't need to re-deploy anything.52:33And you want to use Neo4j, of course,52:36you can use Neo4j Aura DB, that is a52:39external graph database, which also has52:42a free tier. And the code is in the repo52:47here.52:51Is in the repo.52:53And52:54you will need AWS credentials if you're52:57using Amazon Bella Betray in core.53:00And in the resource link, there is some53:03credits. I hope that you can find it53:06because I always try to to give away53:09some credit for AWS so you can deploy53:12everything for free.53:15And uh if you are new here in AWS, I53:19have a repository that you can use to53:21deploy everything over this architecture53:25uh using a notebook, too. But, if you53:27are familiar with CDK, cloud developer53:30kit, you can do it that as well to53:33deploy everything at once.53:35Uh both options are in the repository,53:38of course. And you can go deeper in53:40Agent Core with all the documentation53:42that I left there.53:45So, please don't stop at the demo and53:47try going to production with Amazon53:49better agent car.53:52So, let me bring it all back.53:56Tokens53:57waste on every request.53:59You can fix it with semantic tool54:01selection.54:03And confident answer it's never54:06computed.54:07Possibly it's possible because you are54:10asking how many. So, you can use graph54:13rack and query the data. Don't sample54:16it.54:18Sometimes we have fabric case success54:20confirmation. You can use multi-agent54:23validation and second pass catch it.54:27And rules the models54:30uh quite a little skip it.54:32You can use54:34uh neurosymbolic guardians to enforce in54:38the code. Don't trust in the prompt.54:40And for the hard blocks that I stopped54:43the user, you can use runtime SDR,54:46self-correct, and finish.54:51Each demo that I just showed you is in54:53the in the repository as a notebook and54:56an application as well.54:58Start with the demo one55:00and then go to the demo five. And if you55:03want,55:04deploy it in the productions.55:07So, have you tried any of these in your55:10own hands?55:12Thank you to join me in this session and55:15happy building.
7,194 words · 1252 lines
