Human-in-the-Loop in Agentic Investigations: A Snowball Sprint Storyboard Analysis

Human-in-the-loop is a critical design consideration for your agentic flows. Most companies spend very little time doing the analysis before they jump into building. By spending just a couple of hours with your team doing the analysis we're going to walk through in this article, you save yourself a lot of headache and a lot of money building the wrong thing.

I know, because I built one of these flows.

Sources

• Flow A — the AWS re:Invent investigate-with-Bedrock pattern, from the AWS Re:Invent 2024 - Don't get stuck: How connected telemetry keeps you moving forward (COP322), by AWS Events on YouTube, starting at 53 minutes: AWS re:Invent 2024 - Don't get stuck: How connected telemetry keeps you moving forward (COP322). AWS Events on YouTube.com. Dec 7, 2024. https://www.youtube.com/watch?v=ad42UTjP7ds Collected Jan 21, 2025. AWS Security Blog post "Use an Amazon Bedrock powered chatbot with Amazon Security Lake to help investigate incidents" and the AWS re:Invent 2024 security recap.

• Flow B — a simplified storyboard of the Sumo Logic SOC Analyst Agent that Sumo showcased on the big stage at AWS re:Invent 2025 (Dec 1, 2025), reconstructed from Sumo's own public materials: the Dojo AI expansion announcement, the SOC Analyst Agent product page, and the CIO Magazine demo walkthrough.

Disclosure: I designed and built the proof-of-concept for this pattern at Sumo Logic before leaving the company. My description here uses only Sumo's published material and generic agentic AI design principles; no confidential Sumo implementation details are disclosed.

(Showing you why the Storyboard analysis matters — and exactly how to do it from public sources — is the entire point of this article.)

Why the storyboard, and why first

The Snowball Sprint Storyboard is the first tool in the framing layer:

It captures the use case — the job to be done, the pain being addressed, and, most importantly, who is talking to whom at every step — treating each agent as a person.

The storyboard is a conversation map.

This matters more than it sounds.

Look at any architecture diagram for an agentic investigation flow. Here, for example, is Flow A: Boxes, arrows, an LLM in the middle, a data layer on the right, a human in the loop:

Now look at a different agentic investigation, Flow B. This Architecture diagram shows almost the same boxes and arrows:

If you stop at the architecture diagram, you would conclude that these are basically the same product. They are not. They could not be more different. The differences live in the conversations — who talks to whom, in what order, with what handoffs, and where the human is asked to participate.

That's why the storyboard exists. It surfaces what the architecture diagram hides: the actual job to be done, by whom, and in what sequence.

Storyboard surfaces the pain that the high-level architecture diagram hides behind the boxes and arrows.

The setup: same alert, two flows

Both storyboards start at essentially the same place. Flow A: Alert — sharp increase in Faults in the critical Bot-Service. Flow B: Alert — a possible credential exfiltration from an EC2 instance to an unknown external IP.

In both cases, a Tier-1 analyst is on call. The clock is running.

Flow A is the canonical "give the analyst an LLM copilot" approach. Flow B is what happens when you throw out the chatbot metaphor and ask: what if the analyst's job is to decide, not to investigate?

Walkthrough of Flow A ("Human-in-Every-Loop")

• The analyst (let's call him Bob) sees a new alert in Bot-Service.

• Bob asks the Supervisor agent (the smart robot guy who looks a bit like Vision, the character from the Marvel Avengers movies) to investigate.

• The Supervisor wakes up a bunch of worker agents (who look like simpler square robots) and assigns each of them a task specific to their specialty.

• A short while later, some agents come back with findings. These are only shown to the Supervisor agent, who formats these findings and then presents them to Bob, asking if these are relevant to the investigation. Bob provides some additional investigation guidance.

• (…This flow repeats an uncontrolled number of times…)

• Later, one of the worker agents picks up a critical piece of information. Now the Supervisor has enough to form the hypothesis.

• Supervisor formats the hypothesis and presents it to Bob, who accepts it.

• The Supervisor then chastises Bob on his change management.

• Bob is horrified.

Look at where the human sits. The Analyst is the investigator. The Chatbot is the research assistant. Every decision — what to ask next, whether the answer makes sense, when to stop, what it means — sits with the Analyst.

The "loop" the human is in is every loop.

This is the design choice most agentic security demos make, and it's the choice that breaks at scale. Not because the AI is bad, but because the human is being asked to do the hardest cognitive work (directing an investigation) at 3 a.m., on alert 47 of the shift, with a chatbot that gives confident answers whether or not it should.

There's a second, subtler problem. In Flow A, the Analyst is the only quality control. If the Chatbot hallucinates a column name, the Analyst has to catch it. If the playbook recommendation is wrong for this context, the Analyst has to know. The architecture assumes a senior investigator and ships to a Tier-1 queue.

The human is everywhere. Which is the same as nowhere.

Walkthrough of Flow B ("Human-on-the-Loop")

• The SIEM system generates a new alert: "Possible credential exfiltration from EC2 instance i-0abc to 198.51.100.42." The alert context is immediately passed to the supervisor agent of the autonomous SOC Investigation.

• This supervisor is more specialized than in flow A -- we will call him a "Planner" AI Agent. While this agent is responsible for supervising the investigation, his primary job is to come up with an Investigation Plan.

• The Planner agent retrieves additional information as needed and creates an investigation plan, limited to a fixed step budget. Planner posts the investigation to the UI.

• (... This flow is done in parallel for each of the predetermined steps...)

• The Planner assigns each of the steps of the plan to its own Executor agent, who independently writes a query to retrieve relevant information and executes it. The query and result are written to the UI.

• The Planner then directly passes the query and results to the Analyzer agent, who scans them to figure out what the results mean. Analyzer posts the analysis to the UI.

• The Planner collects all of the results and analysis and passes them to the Judge Agent, who issues a verdict, explains it, suggests remediation steps, and posts everything to the UI.

• The verdict is "Confirmed Compromise," so the Judge elevates the alert level to "Red Alert" and notifies Bob.

• Bob enters the story, looks at the completed investigation, verdict, and the playbook suggested by the AI agents. He confirms the diagnosis and decides to remediate.

• Bob runs the recommended playbook with 1 click and moves on to the next garbage fire.

Bob, the SOC Analyst, is paged once, at the end. He opens a single decision-shaped artifact and makes one call.

The human in Flow B is not in the loop on every step. The human is on the loop — supervising at the decision point, with four agents doing the work the human used to do badly at 3 a.m.

The framing question, made concrete

Look at the two storyboards together, and the question stops being abstract.

In Flow A, the Analyst is in 15+ conversations per alert, none of them important on their own, all of them required, none of them auditable. The "human in the loop" is at the heart of the loop of every SQL query.

In Flow B, the Analyst is in 1 conversation per alert: do I trust the verdict and authorize the action? That conversation is auditable, defensible, and small enough to do well at 3 a.m.

Same human. Same problem. Same alert. Different placement of the loop. Different outcome by an order of magnitude.

The framing question, reduced to its simplest form: what is the smallest set of conversations the human must be in, and how do we design the system so that they are included only in the critical conversations?

This is what the architecture diagram could not show you. The box diagrams hid the essential complexity. The storyboards made the complex Agentic AI interactions obvious by spelling them out.

Try it on something you care about

The storyboards above are a worked example of the Snowball Sprint Storyboard Analysis — the first exercise in the Framing layer. The analysis is highly portable. Try it on your own Agentic AI product flow and briefly sketch the human-Agent conversations. What do you notice? Ask yourself the key question:

If not, what can you automate? Could you replace the human with a Judge that can do the check at machine speed? (Full Agentic Judge-in-the-Loop article: https://uxforai.com/p/kids-matches-agents-judges-and-the-simplest-soc-agent-safety-layer-nobody-built-yet)

The best part is that you don't even need insider information for this. Everything I did here is based on public materials. Perfect for comparing multiple different flows.

The takeaway, for now

"Human in the loop" is not a universal requirement. It is a decision about which conversations the human is in.

The two storyboards above are about two similar alerts handled with two completely different conversation maps. One puts the human in every conversation and exhausts them. The other puts the human in exactly one conversation and empowers them.

Storyboard your own agentic flow this week. Map the conversations. Count the ones the human is in. Then ask which of them actually requires a human in the loop. The number is almost certainly smaller than the number you have today.

That's the storyboard analysis. The next exercise in the framing ring is the Digital Twin — what each party sees, what gets handed off, and what gets withheld. Coming soon, with the AWS Bedrock pattern as the worked example.

This is exactly what we teach in the UX for AI Professional Certification — how to analyze agentic flows, place the human deliberately, and design AI products that actually work.

Cohort 1 sold out. Cohort 2 enrollment opens soon.

Join the waitlist →

Greg