Transcript of BDD, ADR, PRD, WTF: Capturing Decisions for Humans and AI Alike — Michal Cichra, Safe Intelligence
AI Engineer
0:07[music]0:14>> Hi, I'm Michal.0:17Welcome to capturing decisions for0:19humans and AI alike.0:22Yesterday with a team from Safe0:25Intelligence, we have released Spec 27,0:27a new product to test agents.0:29Before that, I was in Microsoft, Red0:31Hat, and spent in 10 years working on a0:34single product.0:35The consistency problems we face with AI0:38and the story of capturing decisions0:40show up in every product I have seen.0:43And these notes are distilled from the0:44experience.0:46And you can find me at the booth.0:48Um0:49So,0:50BDD, PRD, ADR, like that's a lot of0:53acronyms. Uh why does any of that0:55matter?0:56So, let's unpack it from the end.0:58You probably know this story. Uh I hope1:01it's not an urban legend, uh but1:03scientists put five monkeys in a cage1:05with bananas on a ladder.1:07Then gave them a cold shower every time1:09a monkey tried to get a banana.1:11Other monkeys beaten up the poor fellow.1:14Then they replaced the monkeys one by1:16one and none of the originals remained.1:19And yet, they have beaten up every1:20monkey that tried to climb the ladder1:22not knowing why.1:26So, humans and LLMs, they suffer from1:29the same trait. Limited context. People1:31forget. LLMs context compact. Humans1:35leave. LLMs have no memory.1:36>> [snorts]1:37>> After a while of operating a product,1:40the team starts asking, "Why do we have1:42this flow? Why is this goal of this1:45feature?1:46Why is this code shaped like that?1:49Why Where does this belong?"1:51And you might not have the founding1:53engineer available to answer.1:56And these problems show in every org. Uh1:58maybe with AI much sooner than they used2:00to.2:03So, ADR is architecture decision record.2:07It records why you do something and how2:10you enforce it or how you want to do2:12that.2:13And you can cover examples by reference2:16docs and code snippets.2:18For example, we split code in layers to2:20prevent N+1 queries. We enforce that2:23split by linting imports in modules.2:25And we also enforce reading from2:27database2:28returns um plain shapes instead of ORM2:31objects, so we cannot um2:33cannot make these uh these queries and2:36to prevent duplication.2:38And also linting it by module imports.2:41And another like 50 ADRs that define2:44architecture of the product.2:47There is not a single format that that2:48you need to use. It's just a concept.2:51Um2:53It's a text, so there is no specific uh2:57way how to enforce it. You still need a2:58tool to enforce it.3:00But the tool will tell you that this is3:03the rule. Why are you doing this? And3:05how are you supposed to fix it?3:08Then the agent will go and try to find3:10this document why this reason exists and3:13more information about how to fix it.3:15Also, you can define like which files it3:18actually concerns to, like is it some3:20Python files or some folders?3:22And how you actually enforce it.3:26PRD is a product requirements document.3:28Uh that's something lighter when you're3:31building a feature, you describe why3:32that thing exists3:34and what problems it solves.3:36And how user goes through the app to3:40actually interact with it. What's the3:42journey through the application.3:45It can be very light. It doesn't need to3:47be really long and exhaustive like a3:49massive document. Uh you can just3:51capture why, the problem, and the goal,3:54and the journey that connects them.3:57And it's not just for the agents, but4:00also for you 6 weeks from now when you4:03forget why you did that.4:07Now, BDD.4:09Um it's behavior-driven development.4:12You have probably seen spec-driven4:14development lately, uh but if you4:16practiced it, uh4:18you might have suffered the same thing4:20as me.4:21How do you validate that the product4:23actually adheres to the spec?4:25It's a markdown document, you describe4:27how it's supposed to work, but how do4:28you know it actually works like that?4:31One thing harder than reading an AI code4:35is reading AI tests.4:37Um4:38so, what if you had an intermediate4:40layer that actually describes how the4:42product4:43behaves in a human language?4:46And BDD is not new and shiny, but it's4:49it can be executable and readable.4:52So, enter Cucumber.4:54It's almost forgotten, suddenly useful4:56again.4:58It's definitely easier to review than5:00your average tests.5:02You can connect scenarios directly to5:04your PRDs and critical user journeys.5:08It can be readable, executable, and it5:10closes the loop that a spec-driven5:11[clears throat]5:12development leaves open.5:15These5:16rules, uh these specs are later parsed5:20by5:21steps and they are executed as code.5:24But what you can do is that you can5:25actually write and read these. And you5:27can review these. And you can understand5:29these. The language is on you. It5:31doesn't need to be5:33um5:34enforced. Like you There are multiple5:36ways how to write these uh these5:38features.5:40And5:42that's it, but they describe5:45how you're supposed to go through the5:47application, why this thing exists, and5:50how it runs. And similarly, they can5:52refer back to all the documents that you5:54have about5:56why things exist.5:59So, and as a bonus,6:02making consistent UIs with agents is6:05just another level of hard.6:08Like6:08design system and pattern library6:11are the way to build consistent UIs.6:13Like that was the way before AI and it6:16is the way now.6:18So, you document your language. You say,6:20for example, a primary button is this6:22and that. It is blue. It has this shape.6:23It has this color and it's this size.6:26And you say your rules. You say, "We6:28will have only one primary button6:30visible on a site at any on on a page at6:33any point in time."6:35And then you can enforce these rules.6:39Similarly, you define components and6:41patterns.6:42So, for example, if you have multiple6:44colors of these buttons and multiple6:45states,6:46you define components and you define6:48previews and you6:50demonstrate how they work and you create6:52snippets of previews, so you can6:53actually see them.6:55And the agents can see them.6:57And then you can go and review and like6:58do these actually adhere to the6:59principles that I have?7:01Do they adhere to the visuals? And then7:03you reuse them.7:04As with code, you build these from the7:06ground up from small pieces into bigger7:09ones. You compose them and you reuse7:11them.7:13Otherwise, it's uh chaos like with the7:15code.7:18So,7:20cool. These are cool ideas, but how do I7:22actually enforce this?7:24So, my team and agents stick with it.7:28How do I keep it consistent?7:30Well, with the loop.7:32You probably have heard about closing7:34the loop, reinforcement loop, the7:35harness.7:37How to remind the agent that there are7:40rules and how to follow them.7:44So, our loop is simple. It is7:47git hooks, skills, CI, and linters, and7:51a bunch of other checks.7:53Agent's goal is to deliver a pull7:55request and they To do that, they need7:57to use git.7:58So, we use git hooks git hooks to run8:01predefined tasks8:02and these tasks are later executed on a8:04CI. They are the same tasks that they8:06are executed as as hooks.8:08If, for example, agents would get lazy8:10and not want to execute them or skip8:12them, then they get caught.8:14And we include linting, formatting, type8:16checking, code duplication, architecture8:18checks, document linting, everything8:20that's that's possible.8:23So,8:24there was a time where code reviews were8:26about style and tabs and spaces and8:28there is no space for that anymore.8:30All these things are not for discussion.8:33They are rules and they are enforced and8:35they are automated because there is no8:38space for discussion about these8:39anymore.8:40It's more about the high-level concepts.8:43What you cannot find, you cannot8:44enforce.8:48So, for example, we enforce architecture8:50of the product and of the code.8:53We separate modules8:55um8:56and their imports, so what you can use8:59from where.9:00For example, our end-to-end BDD test9:02suite cannot access database. So, we9:04forbid from accessing any module that9:06could access database and basically9:08force the module to the models to9:11iterate without database and really use9:13only the browser features of the9:14application.9:16Similarly, in the product itself, we9:18enforce we cannot talk to database from9:20rendering templates. So, we know that9:22there are no N+1 queries ever.9:25We just define ways to prevent these9:28problems from happening ever. You cannot9:30keep finding them. You need to prevent9:32them entirely.9:36Then the come the agent tries to commit9:38it and push it and they get feedback on9:41the commit and get rejected and they get9:44linked back to the document and they go9:45read it and fix it.9:47And iterate.9:50So, there are some drawbacks.9:52Um9:53it is Oh, sorry. It's not drawbacks. It9:55Um so, this loop is generic. This loop9:58where they do some work, they10:00they push it and they get feedback and10:02they iterate.10:04But the loop can be multiple things,10:06right? Like sometimes you're working on10:07a product feature, sometimes you're10:08working on a UI, sometimes you're10:09working on more back-end-ish10:12back-end-ish things.10:14So that loop is the same, but what10:16changes is the focus of the loop. So we10:18have different skills. There is ADR that10:22whenever there is an ADR mentioned, the10:24agent will look up ADRs, how to operate10:26with them.10:27How to find code that affect that's10:30affected by these ADRs. For PRD the10:33same. For for UI loop, we actually skip10:36bunch of checks and rather force it to10:39iterate in a browser quickly.10:42And10:43test skill that actually identifies10:45tests to run based on code coverage and10:49file changes. So we run just a focused10:50part of the suite and not the entire10:52suite.10:53And some goal execution that actually10:55keep decisions that the model makes so10:56we can review them later.10:59But all of these provide focus in the11:01loop, but the loop still stays the same.11:04There are drawbacks.11:06It is very context heavy. Like you can11:08run out of half of the context in um11:13starting the research.11:15Um but I have no fear of context11:17compacts. Like this actually for like11:19last half year actually works, I think.11:22So11:23in my sessions there 20-50 context11:26compacts and it's it's okay.11:29Because11:30the important things survive and the11:32agent will always look them up again.11:35So and that's the goal anyway, right?11:37Like you want to have multiple hour11:38sessions with a clear goal that agent11:41can operate autonomously with the rules11:42that you define. So11:44that's the goal anyway, like11:48So11:51there are decisions that you can record.11:55There are parts of the product that you11:56can describe why these exist. There is12:00um12:01cucumber or BDD that can have executable12:03specifications that you can actually12:04read and review, understand.12:08Design systems can help you to build12:10consistent UI from components. And12:14um12:15again, enforce it that for example,12:16there are no inline styles anywhere12:18else.12:19And you employ harness to loop it all12:22together.12:24So may the spec be with you.12:26That's it.12:29>> [applause]12:36[music]12:42[music]
1,866 words · 336 lines
