Case Study

Letting AI act on security alerts

Runbooks where AI investigates on its own and a human approves before anything changes

Runbooks where AI investigates on its own and a human approves before anything changes

ROLE

Sole designer, AI product surface

TEAM

Design, PM, engineering

TIMELINE

12 weeks · 2026

COMPANY

Panther Labs

The problem

Agentic runbooks can revoke a session, block a user, isolate a host. But approval was all or nothing: gate everything and lose the speed, or let everything run and lose the control. And the old model assumed someone was watching the console, when real alerts fire at 3am with nobody there.

Key decisions

01

Put a hard stop between thinking and acting

The AI investigates freely, then stops cold before it acts. The gate shows up twice: as a visible line in the builder (“once a determination is reached, the runbook acts”) and at runtime, where every response step waits for approval. I shipped this simple hard stop before the fuller policy engine because it’s what would unblock first adopters.

02

Let people author the posture, with AI

A runbook is where an org sets how a detection gets handled. An AI co-author interviews you before drafting, or you start from a template. Then you test it against real past alerts, grouped by behavior and weighted toward the rare cases where a runbook is most likely to be wrong. Gaps come back as fixes you can apply in one click.

03

03 / Bring the approval to the analyst

If the gate only lives in the console, it fails the 3am test. So approval comes to the analyst in Slack: what the AI wants to do, why, and inline actions. All designed inside Block Kit, which had to carry a high-stakes security decision using only the components Slack allows.

Results

  • Designed the full runbook lifecycle: templates, an AI co-author, testing against real past alerts, and a hard gate before any action

  • Made testing the trust step: replay against behavior clusters weighted toward rare cases, so teams see where a runbook breaks before it runs live

  • Put the gate in two places: visible while authoring, and enforced at runtime, where every response step waits for approval

  • Panther was acquired by Databricks before this shipped. The pattern of investigate freely, approve before acting, is the same problem I work on now

← Previous case study

Next case study →