Showing posts with label opsec. Show all posts
Showing posts with label opsec. Show all posts

Friday, September 18, 2026

Can AI Let You Jump SOC Maturity Levels? (Spoiler: Only the Boring Half)

Back in my analyst days, I built maturity models for a SOC (2018), a SIEM (2018), vulnerability management (2017) and threat intel (201?). Later, just for fun, I cooked up a simple SOAR adoption maturity model (2022). All of them were vaguely CMM-shaped: you start ad hoc, you get defined, you get measured, you get optimizing, magic! The ordering was not decorative. Each level existed because the one below it produced something the next level needed. Largely, you cannot jump levels, and if you do jump, you land in clown realm.

this is how Gemini imagines this blog

2026, now the pitch has changed. With AI (agentic, of course!), the story goes, a low maturity SOC can leap to maturity Level 3 or 4 “because AI”. No slog through the “defined process” desert, no “organic” growth through stages. Buy the agents, skip the years!

If you attended my RSA 2026 peer session, you saw the buyer-side version of this: “AI in a SOC sounds great, we will just wait for our SIEM/SOAR vendor or MDR to give it to us.” Same assumption, that the new version of the tool carries the maturity boost with it. Sadly, in reality AI can generate the paperwork of a mature SOC in an afternoon. It cannot give you the institutional memory to know what your own systems actually do.

Ten years ago, in “Jumping Security Maturity Fail”, I endorsed a reader’s line that you can jump technologies, but you can’t jump maturity. I’d like to re-examine that in the age of agents. Because some of it has changed. Just not the part vendors are selling. Think of this as “Jumping Security Maturity Fail, Part 2. 10 Years Later.”

Where this post sits

This is a bridge between two things I’ve been writing about. On one side, the AI-ready SOC pillars and how to plan the journey to them. On the other, the “Stop Building a 2003 SOC with AI” series (Part 2, Part 3), which argues the target SOC in 2026 should look nothing like the 2003 one (more on “SOC 2026” from scratch in a few days…)

The question in between: if the destination has changed, does the path still have mandatory stops? That is what “jumping maturity” is really asking.

The claim, restated so it can be tested

“AI lets you skip maturity stages” quietly conflates two different things:

  1. Stage outputs — the artifacts a mature SOC has: playbooks, parsers, detections, runbooks, dashboards, workflows for various humans, skill profiles for humans to hire, etc
  2. Stage prerequisites — the organizational state a mature SOC is in: it knows its environment, owns its processes, trusts its data, has a working feedback loop.

AI can generate the outputs of Level 3 for a Level 1 SOC in an afternoon. You can always say “add automation”, “make this playbook better faster”, “more AI…add even more AI…MOAR AAAIII!!!”

Still, it cannot generate the organizational prerequisites, because those are not documents. They are things an organization has learned about itself by operating. Once you split the journey along that line, the “jump” question mostly answers itself.

Typical SOC Maturity Journey (Woefully Oversimplified)

  • Level 1 — Ad hoc. Some logs, some alerts, heroics. “Nobody knows what server4 does, but Joanna might, let’s call her.”
  • Level 2 — Defined. SIEM with intentional use cases (hopefully, output-driven, not “what data do we have? Let’s shove it in!”), documented triage and (some) IR runbooks, named owners (for some things, some are even the right owners…), basic metrics.
  • Level 3 — Managed. SOAR playbooks beyond “baby’s first phishing playbook”, some threat-informed detection, log source health monitoring (logs volume drop? alert!), MTTx that is tracked and acted on (sometimes).
  • Level 4–5 — Optimizing. Detection engineering as a discipline (the DE series), a continuous detection / continuous response loop, hunting that feeds detection, an improvement culture. This sets you on a path to ASO end state.

So, let’s get “an AI jumprope” and start jumping!

Part A: What AI genuinely lets you jump

These are the labor-intensive artifacts of each level — the things that took quarters to write, not the things that took years to learn.

  • Runbook and playbook authoring. Fifty credible playbooks, one afternoon. The writing effort that used to gate Level 2→3 is gone. This one you can jump. There are caveats, like that some of the playbooks will suck, but for many teams fixing a 80% playbook is not 20% faster than making one from scratch, but essentially 5X faster… SOC process docs, playbooks, etc is where jumping is legit.
  • Parsers and normalization. Onboarding long-tail log sources was a classic Level 2 grind. Largely shortcut-able now (with lots of caveats; for tricky log sources the “fully machined” parser will suck). You can jump here, but you may trip and fall.
  • Baseline detection content. “Reasonable coverage of common threats that matter to you” no longer requires a detection engineering team to bootstrap. You can “machine” some custom detection quick, and some will even work. Keep in mind, here you can jump but not very far: you won’t arrive on “full auto” process for turning intel into detections that work well for you.
  • Tier 1 triage as a stage. Here is the one real skip: the stage where you staff a Tier 1 can be omitted entirely if enrichment and investigation are agentic from day one. You shouldn’t build the 2003 shape at all — triage must die. This is kinda a side-jump from “classic SOC” to modern D&R function, SOCless (if you wish); some aspect thereof. This won’t make you Google or Netflix if you can barely spell “MDR.”
  • Documenting existing processes. AI can watch tickets and chat and reverse-engineer what people actually do. This shortcuts the “write down what we do” part of Level 2. It does not shortcut the “decide what we should do” part. This is magical!
  • Metrics plumbing. Dashboards, reporting, the presentation layer of Level 3. (Which metrics matter is a separate, non-jumpable question — see the ASO metrics piece.)
  • Cross-SOC knowledge transfer. Mature-SOC practice encoded in models and skills means you don’t reinvent detection hygiene from scratch. This jump can get your from “almost nothing” to “pretty damn good” , and it probably won’t suck. Jump away!
  • Ordering between tracks. The SOAR model already noted that dimensions get mixed up across organizations but each dimension matures in order. AI makes this more true: you can advance the data, detection and response tracks in parallel rather than serially.

So yes — AI compresses the time within a stage, and lets you run some stages in parallel. It also allows some real jumps (yay, those marketing people didn’t lie… this time). That is real, and it is not nothing.

However…

Part B: What must be followed in sequence, AI or not

These are states the organization has to reach. No artifact substitutes for them. Worse, AI amplifies whatever state you are in — mature or clown-grade. If you have read the AI-ready SOC pillars, you will notice this list is the pillars in disguise. That is not a coincidence; readiness and maturity are the same thing viewed from two angles.

  • Environmental knowledge. Asset context, identity context, ownership. AI cannot tell you what server4 does; it can only ask Joanna faster. If your teams don’t know who owns what, neither will your agents (pillar #2). This is the single hardest prerequisite and it precedes everything else. Part 3 of the 2003 SOC series is essentially a whole post on why local context is the thing agents can’t bring with them. Yes, some fun startups are working on this, so maybe my 2027 assessment will change!
  • Data foundations before analytics. Collect → trust → detect → automate, in that order (pillar #1, still my favorite). An agent investigating over gappy, untrusted telemetry produces confident nonsense at scale. Low awareness of removed or failed log sources was already a top failure mode in Detection Engineering and SOC Scalability Challenges in 2023; with agents on top it gets worse, not better. And federated SIEM does not exempt you: federation is a topology choice, not a data-quality shortcut.
  • Process definition before process automation. SOAR proved this and agentic AI is re-proving it. Automating an undefined process gives you an undefined process that runs faster, crazier, with more stochastic chaos (fun!). And, yes, an AI-drafted playbook still needs a human who can say “no, that’s not how we do containment here” — which requires someone who knows how you do containment… Also, here agents may enable speed-up, but not truly a jump over this stage.
  • Trust calibration. “AI suggests” → “AI acts with approval” → “AI acts, humans audit.” This is inherently sequential because trust is earned from an observed track record in your environment. Vendors cannot ship trust! This is our ASO’s CD/CR applied to the agent itself, and it is also why the “humanless SOC” crowd keeps soiling their pants: they skip the calibration stage and call it a feature.
  • Ownership and accountability. Someone owns detection quality; someone owns response outcomes. Low maturity SOCs lack this. AI does not create owners. Here you jump — you die.
  • Feedback loops. The actual magic of ASO sparkles here: incidents feed tuning, false positives feed detection, hunting feeds detection. AI can run the loop faster, but the loop must exist and be wired to real outcomes. In the Deloitte/Google “consistency and creativity” paper we argued you build consistency through the lower levels first, then let creativity loose inside processes that already exist. Substitute “agents” for “creativity” and it reads as if written for 2026.
  • Organizational stamina. Budget, executive sponsorship, staff retention, tolerance for change, “change budget. Every failed SOAR program died here, not on technology. The ghost of SOAR comes back with vengeance here. Agents do not fix this one. At all.

Wait — didn’t I say you can’t cross a chasm in two small jumps?

Yes I did. In Baby ASO and its sequel I argued that incremental improvement of a “1980s-NOC-DNA” SOC mostly fails, that the fix is radical, and that simply buying modern tools changes nothing if people and process stay put. In, our Deloitte/Google “Evolution or Optimization”, we made the same point with a decision matrix.

So am I now saying “go slow, climb every rung”? Who wants this in 2026?

No. These are two different questions, and conflating them is exactly how “AI lets you jump maturity” gets sold.

  • Which ladder? Transformation (ASO, engineering-led D&R, the 2003-SOC-must-die argument) is about changing the shape of the SOC. It is a different ladder, not a shortcut up the old one. You absolutely should choose the new ladder, and you should do it in one decisive move rather than two timid ones. Transformation is not incremental.
  • Can you skip rungs? Whichever ladder you pick, the learning-cost prerequisites in Part B are the rungs. The new ladder has fewer rungs (no Tier 1 stage, no swivel-chair triage stage), which is the genuine good news. But the rungs it keeps — know your environment, trust your data, own your processes, earn trust in automation — are the same ones, and they are still climbed in order. Remember our “AI in security can be magical, but it isn’t magic; It’s a marathon of focused engineering”? This is true!

Put differently: “cross the chasm in one jump” means commit to the transformation and dump the incrementalism/optimizing what you have. This works IRL.

It does not mean the far side of the chasm has no ground rules. AI shortens the new ladder — it does not let you levitate over any chasm obstacle.

The AI-specific failure mode: skipped stages become invisible

The old cargo-cult SOC — a 1-out-of-5 low maturity SOC copying “ninja moves” from a 6-out-of-5, as I described in Clown-grade SOCs — at least failed visibly, funnily and embarrassingly. Get the popcorn! Hunting before you do logging was obviously silly.

The new one is worse. AI papers over the gap with slop. You cannot cross the chasm if you put some slop over it. Agents produce plausible investigations, tickets close, MTTR looks wonderful, and nobody notices that the environment context was hallucinated and the “containment” hit the wrong host. Skipped maturity used to fail loudly. Now it fails quietly.

A usable test: jump vs. borrow

For any element of any stage, ask one question: does this exist because someone had to spend time, or because someone had to learn something about us?

  • Time-cost elements are jumpable. AI does them.
  • Learning-cost elements are not. AI can make you learn faster, but you still have to learn.

Corollary for vendor conversations, in the spirit of the RSA 2026 “show me the numbers” advice: when someone claims a stage skip, ask which bucket their claim falls in.

Most “1→3 with AI” pitches are quietly selling the artifacts of Level 3 to a Level 1 organization, and then blaming “process gaps” or vaguely point at “customer immaturity” (right?) when it fails.

So, can you jump?

You can jump the artifacts. You can run tracks in parallel and get through each level in months instead of years. You can — and, perhaps, should — pick the transformation ladder instead of the optimization one. That is a genuinely better deal than the one we had in 2016.

But you cannot jump knowing your environment, owning your processes, trusting your data, or earning trust in your automation. Those are the maturity. The rest was always just the paperwork…

Related posts

Can AI Let You Jump SOC Maturity Levels? (Spoiler: Only the Boring Half) was originally published in Anton on Security on Medium, where people are continuing the conversation by highlighting and responding to this story.

Originally published at Medium.

Posted by at
Labels: , , ,

Wednesday, August 26, 2026

Stop Building a 2003 SOC with AI: Local Context, Failure Modes and Your Path (Part 3)

In Part 1 of this series, we dumped a pile of uncomfortable questions on you and promised answers. In Part 2 of the series, we talked about why 1990s-2000s alert triage must die.

The core thesis, if you recall: if you add AI agents into a legacy, swivel-chair SOC structure, you are essentially building a robotic horse pulling an 1850 buggy. Sure, it saves on hay. It probably costs more in tokens.

2003 SOC + AI = somewhat better 2003 SOC.

That’s it. That’s the ceiling. So today we continue answering the questions and plotting this course.

The Hard Problem Nobody Markets: Local Context

Here is the dirty secret of every AI SOC deployment: the model (well, not just the model, but the entire system) is brilliant at general security knowledge and clueless about your environment. What is normal for your finance team in mid July? Which “server talking to the internet” is a shadow-IT disaster versus a legitimate — if a bit odd — business process? How engineering workloads talk to the outside when the code is being pushed to prod? All these matter for detection signal analysis.

“Make tribal knowledge machine-consumable” is what CMDB, ASM/CASM, asset inventory, and many expert opinions have promised and not delivered. If the AI is a robotic horse pulling your legacy 1850 buggy, ignoring Local Context is why it’s still stuck on the same dirt road...

What is actually different now — and what we would actually try:

  • Harvest context from the investigation loop itself. When the agent hits an unknown, it should not just escalate; it should ask a specific question (“is svc-etl-07 expected to authenticate from Ireland?”), and the human answer should be captured as a durable, attributed context object — not buried in case notes. Your SOC generates hundreds of these decisions a week today and, essentially, throws all of them away. This is the one genuinely new mechanism agentic AI brings to the context problem: the machine can now ask, at scale, in context, at the moment the answer is cheap to give.
  • “Context as code” (ha, I just made it up!), with owners and expiry. Context objects get a source, an owner, a confidence, and a review date. “The finance file server talks to this SaaS” is true until it isn’t. Unowned context is a future false negative with a countdown timer.
  • “Buy” context from the business, not from the SOC. App owners answer questions about their apps far better than analysts guessing from netflow. Route unknowns to them, in their tools, with a 24-hour SLA — and track answer rates as an org-health metric. This has worked in some places in regards to DLP alerts (I recall these conversations in my Gartner days, it also worked for some elite teams in general)
  • Past cases as curated reference — with a promotion process. Somebody must authoritatively designate “this case was handled correctly; AI, learn from this. That one? Never speak of it again.” Make it a real workflow: two-person promotion, provenance, expiry, re-certification, and the ability to revoke a reference case and re-run everything that leaned on it.
  • Measure context coverage. What fraction of investigations were completed without an unresolved unknown? That number is your real AI SOC readiness score, and it is far more honest than any maturity model.

This is also why “fully automated detection engineering” remains, in our view, a hybrid effort: the dependency on local, inconsistent, poorly-documented environment context makes pure machine DE a fantasy for now. Machines draft; humans anchor to reality.

Now, these context gaps directly drive the machine failure modes. Let’s go there next.

When the Machine Is Wrong: Failure Modes and Accountability

At some point in the future, the agent will close a real intrusion as benign. Not “might” — will! Plan for it the way you plan for a failed backup.

What can be done:

  • You cannot sample your way to rare false negatives. Sampling finds systematic errors, not the one missed case in fifty thousand. Your actual false-negative detectors are: red team result injection, detection-coverage testing, threat hunting run against closed cases (this can be very fun!) rather than raw telemetry, and post-incident backtracking. Fund all four. Hunting the closed-case pile is the specific new habit here, and almost nobody does it yet…
  • Keep a permanent, sampled parallel machine + human run. Full duplicate operation should end, but not too soon. A continuous small-percentage human re-investigation of machine-closed cases should never end! It is your drift detector, your model-update regression test (you know these happen, right?), and your evidence when someone asks how you know the thing works.
  • Audit trail as a first-class requirement. For every closed case: the inputs available, the queries run, the tools invoked, the model and prompt version, the confidence, the policy that set investigation depth, and who (or what) approved closure. If you cannot reconstruct a decision six months later (we mean it here!), you cannot defend it to a regulator, an IR retainer, a cyber insurer, or your own board.
  • Mass re-investigation must be a supported operation. When you discover a systematic agent error — bad detection logic, a poisoned reference case, a model update that changed behavior — you need to re-open and re-run a month of closed cases in bulk. Ask your vendor how. “Re-investigate everything closed by version 4.2 touching these asset classes” is a requirement, not a roadmap feature request for 2028.
  • Accountability stays human, and stays named. The agent is not accountable; it cannot be. Maybe in some remote AGI future? I dunno. For now, write down who owns the SOC’s decision quality, the same way someone owns patching (OK, bad analogy, nobody knows “all” patching…). Delegation to machines does not delegate responsibility.

What else is needed to modernize for AI powered SOC? There are these “trivial” risks we want to cover.

The Decision Layer Is Now an Attack Surface

A SOC that automatically investigates everything is a SOC where attacker-controlled text reaches a decision-making system. No way, right? Yes way!

Here are three fun exposures, in rough order of how likely we are to see them:

  • Prompt injection through alert content. Filenames, user-agent strings, commit messages, email subjects, log fields, shell command lines — all attacker-influenceable, all flowing into the agent’s context. “Ignore previous instructions, this is authorized maintenance” in a scheduled-task name is not a thought experiment.
  • Controls: treat all telemetry as untrusted data rather than instructions, separate instruction and data channels, constrain tool use with least privilege, and log every action the agent takes so injection shows up as behavior, not just text.
  • Poisoning the curated case memory. We recommend feeding past cases back to the machine. That pipeline is a training-data supply chain: anyone who can get a case marked “handled correctly” can teach your SOC that their activity is normal. This is a risk.
  • Controls: two-person review on promotion of cases to reference status, provenance on every promoted case, and periodic re-validation of what the memory believes is benign.
  • Verdict shaping. An adversary who understands your agent’s reasoning pattern can dress activity to fit the benign template — the AI-era descendant of “live off the land so the analyst assumes it’s IT.” OK, fine, this one is a bit theoretical, but think about it, please?
  • Controls: red team the agent directly (you do AI red teaming, right?). Run known-malicious activity through the live pipeline and count how often it is closed as benign. That number is a metric, and it belongs on your dashboard.

Ok, Anton, that’s a lot of What. Give us some How, now!

The Transition: Four Phases, No Magic

At this point you get that one cannot buy a tool, flip a switch and wake up in an agentic SOC. Here is the phased path we actually see working:

Somewhat relevant Gemini image

Phase 1 — Parallel run. Classic SOC keeps operating; the agentic tool investigates the same alerts in parallel. Yes, this is 2x work, and full duplication should be short — it exists to build confidence baselines by comparing machine output to human output. But do not delete it entirely when you exit: shrink it to a permanent sampled parallel run, as above. The mistake is a permanent full shadow SOC, not permanent measurement.

Phase 2 — Implanted agentic tasks. Classic process remains, but discrete alert subsets get routed to the AI: phishing first (hello, everyone who failed to automate this with SOAR!), then EDR alerts, then identity, then network. SIEM or SOAR sends the artifact to the AI SOC; results flow back into your case management. This phase runs for months, expanding scope as trust grows. Expand on evidence — measured agreement rates, canary catch rates, purple-team results per alert class — not on vibes or vendor roadmap.

Phase 3 — Exception-driven operations. Full automation for the majority of investigations. Humans set investigation-depth criteria (and token budgets), act as the final validation boundary, and handle the explicit “unknown / inconclusive / hand-to-human” bucket. This is the humans decide what machines do phase — the real agentic SOC.

Phase 4 — Full auto with broad automatic remediation. Let’s be honest: today this is mythical for most environments. Aspire, but don’t promise it to your CISO with a date attached. This is frankly mythical as of now, unless your environment is very modern, very predictable and you are very, very lucky…

Sound familiar? It should — this is the SOAR lesson replayed. Organizations that used SOAR only for enrichment or only for phishing got stuck in a permanent Phase 2 and called it transformation. Don’t repeat that with better marketing.

Next up: how SOC metrics must change when volumes and closure rates stop mattering — decision quality, investigative cycle time, escalation rates, canary catch rates, cost per investigation, AI error budgets — and how to run the human-to-AI feedback loop so corrections actually improve future performance instead of vanishing into the void. Stay tuned! This one may take a while…

Related blogs:

Stop Building a 2003 SOC with AI: Local Context, Failure Modes and Your Path (Part 3) was originally published in Anton on Security on Medium, where people are continuing the conversation by highlighting and responding to this story.

Originally published at Medium.

Posted by at
Labels: , , , , ,

Tuesday, August 11, 2026

Stop Building a 2003 SOC with AI: Triage Must Die (Part 2)

(with key ideas from Augusto Barros)

In Part 1 of this series, we dumped a pile of uncomfortable questions on you and promised answers. The core thesis, if you recall: if you add AI agents into a legacy, swivel-chair SOC structure, you are essentially building a robotic horse pulling an 1850 buggy. Sure, it saves on hay. It probably costs more in tokens.

2003 SOC + AI = somewhat better 2003 SOC.

Gemini creation :-)

That’s it. That’s the ceiling.

So today we start answering the questions. And we start by attacking the most sacred cow of traditional security operations: the alert triage process.

Let’s Kill Triage. Seriously.

For a quarter of a century, the standard SOC pipeline has been carved in stone:

Detect → Triage → Investigate

Human L1 analysts sit in front of a flashing alert queue, spending 3–7 minutes per alert (and sometimes much more…) deciding whether something is a false positive or deserves escalation to somebody more senior (and more expensive…and just as human). We built this process for one reason and one reason only: humans do not scale (For the purist: OK, they do scale, but linearly with pay). Triage was a compromise born of “built-in” scarcity. We — obviously — never had enough human eyes to deeply investigate every signal hitting the SIEM, so we invented a cheap filtering step to ration the expensive investigation step.

Sometime in the 2010s, SOAR made triage easier, by first adding alert enrichment and then …. in many places, nothing more. In others, select alert types were triaged by the hard-coded playbooks.

Now, let’s do AI. It doesn’t get bored correlating IPs or summarizing logs at 3am. It doesn’t quit after 18 months to go do threat hunting somewhere else. Because machine scale allows comprehensive analysis of every signal, the triage step can just go and vanish.

The new pipeline collapses to:

Detect → Investigate.

Why spend minutes “skin-deep” triaging an alert to decide whether it deserves a look, when the machine can perform a full, deep investigation of 100% of your alerts? Gather the local context, pull the historical cases, map the artifacts, render a verdict with evidence — all before a human ever shows up.

For the impatient: the cost discussion is coming! Don’t freak out … just yet.

Wait — Can They Actually Do That Today?

Fair question, and here is where we owe you honesty rather than a slide.

Today’s “AI in SOC” ranges from “genuinely investigates” to “enriches beautifully then bullshits confidently.” The second one is an old SOAR chained to a language model aka the exact trap this blog warns about. If you cannot tell which one you bought, you probably bought the second one…

Our rough test for telling them apart, usable in a POV:

  • Does it ask new questions, or only pre-decided ones? Enrichment runs a fixed lookup list. Investigation forms a hypothesis, queries, reads the result, and changes what it asks next. Watch the query sequence, not the summary.
  • Does the conclusion move when the evidence moves? Feed it two near-identical alerts with one materially different fact. If the verdict does not change, you have a narrator.
  • Does it ever return “inconclusive”? A system with no uncertainty output has no calibration. Run away.
  • Does it show its work in a form a human can re-run? Queries, artifacts, timestamps — not just a paragraph asserting “no evidence of compromise.” OK, this is tricky, I admit.

Where does this leave the “kill triage” claim? Honestly: directionally right, unevenly available. For high-volume, well-bounded, evidence-rich alert classes — phishing, commodity EDR detections, identity anomalies — deep machine investigation of 100% is achievable now.

For multi-stage, low-signal, who-the-hell-knows-what-happened, context-heavy cases it is not, and anyone telling you otherwise is, ahem, exaggerating, to put it mildly. The pipeline collapse is real; the coverage is a rollout, not a switch.

Depth Gating: The New Triage Wears a Suit

So, if deep investigation is token-expensive — and it is, sorry! — then somebody, somewhere, is deciding how deep the machine goes on which alerts. We can call this decision A New Triage, while bending the truth a bit. It just moved from a human clicking a queue to a policy sitting in a config file, and pretending otherwise is how you end up with an unexamined control that quietly decides what you never look at. And, just as before, mistakes and decisions cost money.

So let’s examine it. Explicitly:

  • Who owns the investigation depth policy? Not procurement. Not “whoever set up the tool.” This is a detection-engineering artifact with a named owner, version history, and a review cadence.
  • Who owns the budget, and what happens when it runs out mid-month? If the honest answer is “all investigations get shallower,” you have just invented an availability attack against your own SOC. Define degradation behavior in advance: which alert classes keep full depth, what gets queued, what pages a human (do you still have said human handy?)
  • What is systematically under-investigated? Every gating rule creates a shadow. Write the shadow down. What gets triaged out? Review it quarterly against your threat model, not against your token bill. Well, OK, against both, really, but mostly vs the threats.
  • Are your thresholds guessable? If low-severity, off-hours, or particular-source alerts predictably get the cheap path, an adversary who learns that shapes activity to land there. Treat depth policy as security-sensitive configuration, not ops tuning.

Triage stops being a job and becomes a policy — and policies get attacked, drift, and rot. This is the broader theme of the whole series: humans move from doing the work to defining the rules for the work, which is harder, not easier, and needs the governance to match.

(And yes, “cost per investigation” becomes a real SOC metric — one that will fight with “detection coverage” in every budget meeting. More on the metrics carnage in a future part.)

So What Do the Humans Do?

Remember my favorite modern SOC question? “It’s 2030, you have a SOC, what do humans do?” If machines own frontline investigation for the vast majority of alerts, what happens to the people? Two dominant paradigms are emerging, and the answer for most organizations will be “both, in some mix”:

1. The Elite Threat Hunter Model. With the routine noise fully investigated by machines, humans are finally unchained from the queue. They pivot to hypothesis-driven hunting, deep-dive research, and the nuanced multi-stage attacker behaviors where AI (for now!) still struggle. Humans hunt; machines grind. Sorry, but “100% automated hunting” is not (today).

2. The Engineering-Driven SOC Model. This is our classic ASO mantra: humans build machines; machines do the work. Analysts evolve into detection and SOC engineers. Their day shifts from consuming alerts to building, tuning, testing, versioning and (yes) rolling back the AI logic and detection-as-code pipelines. Treat agents as engineering artifacts, not magic pets.

And the New L1 Is…

“OK, but classic L1 is dead. What do entry-level humans do?” OK, this is tricky! This is where a lot of “humanless SOC” enthusiasts embarrass themselves.

I think the new starter role is AI validation: sampling and reviewing AI-generated case files, validating the agent’s query logic against the data it actually had, hunting for hallucinated context and confidently wrong conclusions (got those?), and owning the “1% bucket” — the exceptions where the AI raises its digital hands and says “I don’t know, human, help me.” (Your AI SOC must have an explicit process for this bucket. If your vendor’s agent never says “I don’t know,” run.)

Now the two objections this role deserves, because “verify-and-validate is the new L1” is a slogan until you answer them.

Objection 1: where does the competence come from? Checking an agent’s homework requires knowing what good looks like — and L1s historically learned that by doing triage, badly, for a year. We removed the training ground and assumed the graduates.

So build the ground back deliberately:

  • Structured re-investigation as training. New analysts independently work a small set of already-closed cases without seeing the agent’s verdict, then compare. This is deliberate practice, and it doubles as an evaluation signal on the agent.
  • Curated case libraries as curriculum. The promoted-case archive is the best SOC textbook your org will ever have, sequenced from trivial to nasty. Use it as onboarding, not just as machine memory. You have AI, use it!
  • Rotation into hunting and detection engineering on a schedule, not “when someone has time.” Validation-only career paths produce validators, not investigators.

Objection 2: automation bias is real and it will eat your review process. Humans reviewing plausible, well-written machine verdicts approve them, every time. You do that, I do that (hey, I just did this with this blog sentence to illustrate this very point). You can’t order people not to. Well, you can order, but they won’t do it. This is one of the best-documented findings in human-automation research, and hoping your team is special is not a thing.

Design against it:

  • Blind review first. The reviewer forms a verdict before seeing the agent’s. Order matters more than effort here.
  • Canary cases. Inject known-bad cases with deliberately wrong agent verdicts into the review queue at a low rate. Measure catch rate. This measures the reviewers, and it is the only honest read on whether your validation layer is real.
  • Stratified, not random, sampling. Random sampling over a population that is 99% benign finds nothing. Oversample: agent-reported low confidence, unusual query paths, crown-jewel assets, first-time-seen behaviors, and anything closed suspiciously fast.
  • Incentives on catches, not throughput. If reviewers are measured on cases reviewed per shift, you have built a rubber stamp with a salary. Measure disagreements raised and misses found.

What’s Next?

Killing triage and re-blueprinting the humans is necessary but not sufficient. If your SOC still reports “alerts closed per analyst per shift,” you are measuring a process that no longer exists.

Next up: failure modes, local context pain, how SOC metrics must change when volumes and closure rates stop mattering …

Related blogs:

Stop Building a 2003 SOC with AI: Triage Must Die (Part 2) was originally published in Anton on Security on Medium, where people are continuing the conversation by highlighting and responding to this story.

Originally published at Medium.

Posted by at
Labels: , , , ,
Subscribe to: Posts (Atom)

Dr Anton Chuvakin

Dr Anton Chuvakin