How to Keep Up With Never-Ending SOC Continuous Improvement
SOC continuous improvement fails when insights get buried in closed tickets. Learn a 6-step LLM pipeline that turns investigation notes into action.
I was there, I sat in every SOC seat out there…
A SOC analyst grinding through alert queues at 2am. Part of an Incident Response team leading running war rooms. A SOC manager in Monday morning stand-ups asking what we learned this week while staring at blank faces.
Every single role. Every single day. And the one thing that never changed across any of them?
The insights, recommendations, self improvement, the de-facto SOC continuous improvement action items were disappearing. Seating documented in a case log for no one to action upon, trapped inside closed tickets that live in a backlog nobody rarely reopens.
I know the why and I feel the overwhelming operations, which is why I’m offering a practical solution for how to continuously improve your SOC with the valuable insights coming out of your investigations.
The Hidden Goldmine You're Sitting On
Every ticket your team closes tells a story. It's not just that an alert fired, then an analyst investigated and eventually closed. There are powerful signals buried in those notes, whether it's a tool with overly noisy alerts, a gap in your email gateway rules, or the same user clicking a phishing link for the third month in a row.
Your tier 1 all the way to your tier 5 analysts and IR responders are generating intelligence every single shift and with every single incident. They know things and they're writing them down. It's useful information but these notes get buried and never read again.
It's a sad truth... I know because I've been in those weekly SOC meetings, I was running them.
It's not a people problem, rather, it's a system problem.
The Weekly Report Trap
The thing people look to as the standard fix is the weekly report. In theory it's elegant: senior analysts summarize the week, extract the learnings, feed them back into tier 1 runbooks and detection improvements. On paper, it's a proper feedback loop.
In practice, it becomes the task that either gets rushed on Friday afternoon or simply doesn't happen. It's for good reason too! Your senior analysts are already stretched because on top of everything they need to do for their jobs, they're also being asked to synthesize everything in themes. You either get a half-hearted copy-paste of ticket titles, or, more likely, you get nothing.
Teams try rotation where everyone takes a turn on the ferris wheel. But in doing so, you face losing important insights and information, not to mention a lack of consistency.
Now add a follow-the-sun operation to this. APAC closes tickets while EMEA is asleep. EMEA handles incidents while Americas is offline. By the time anyone tries to compile a summary, they're working with fragments. Nobody has the full picture. The patterns that only emerge when you look across all shifts stay invisible.
Wait, Can't AI Can Solve This Pretty Easily?
When capable LLMs became available, I thought this was finally solved. Just feed all the investigation summaries in, ask for a weekly report. Done? Not so fast... here's what actually happened.
First attempt: I gave the best LLM models that money can buy more than 250 investigation summaries and asked for a consolidated report. But what I got back was a mess.
What I saw were recommendations repeated five times just with slightly different wording. Severity assessments that made no sense and my “favorite” recommendations that are not feasible, for example “Tune your EDR machine learning to reduce false positives of macro xlsx files”.
No traceability whatsoever, no way to tie anything back to the original investigation and forget about cross referencing with similar recommendations.
Second attempt: I went deep on prompt engineering. Longer prompts. More detailed. With examples. The results improved marginally, but the ceiling was surprisingly low.
The fundamental issue is that when you dump a large context with complex requirements into a single LLM call, it can't hold everything in working memory. It forgets constraints from earlier in the prompt. It hallucinates connections between unrelated incidents. Severity levels come out inconsistent.
One-shot approaches get you mediocre fast. They don't get you useful.
The Breakthrough: Think Multi-Step, Not Prompt
The shift that changed everything was stopping thinking about this as one task and starting to think about it as a multi-step pipeline.
When an experienced analyst writes a weekly report, they don't try to do it all at once. They read, they group, they prioritize, they write. Multiple steps. Each one is different.
So I built it that way.
The 6-step pipeline
Step 1: Classification
The first step does one thing and one thing only. It extracts and categorizes recommendations from raw investigation summaries. It looks for whatever your analysts call them: Recommendations, Do Better, Action Items, Next Steps. It pulls each one out and assigns it to a category: detection, prevention and process improvements.
No dedupe. No severity. Just extraction, done well.
Step 2: Feasibility Assessment
Now we evaluate each recommendation against practical reality. Can this actually be implemented? Is it a quick win or a multi-quarter project? Does it require resources you don't have?
This is also where web search earns its keep. When a recommendation references a specific product or vendor, the model can look up current best practices, product documentations, tech community discussions and verify the suggested configuration actually exists and is supported. Without this, you get generic, often infeasible advice. With it, you get grounded recommendations.
Make sure to use an LLM model that has web search capability via API calls.
Step 3: Citation Attachment
Before touching deduplication, every recommendation gets linked back to its source investigation. This is non-negotiable for a report anyone will actually act on. When a SOC manager reads and SOC teams attempt recommendation implementation, they need to know which investigations triggered that and value with volume justification to it. Otherwise it's just noise or worse, it might break business operations.
Step 4: Deduplication
Three analysts working three separate investigations but same use case, all recommend the same prevention improvement. Without deduplication, you get three entries saying the same thing with slightly different wording. With it, you get one consolidated recommendation that shows it came from three independent investigations, which is actually a stronger signal.
Citations from all source recommendations get merged. Nothing is lost.
Step 5: Severity Classification
Now, with duplicates consolidated, we can assign severity levels that actually mean something. The model evaluates security impact per your instructions, weights and SOC defined severities for each use case. Not how urgent did the analyst feel when writing this, but what is the actual risk if this doesn't get addressed built on your SOC knowledge base.
Separating this from extraction forces objectivity. If you try to assign severity while also pulling recommendations from raw notes, the analyst's tone bleeds in and skews the assessment.
Step 6: Report Generation
Everything feeds into the final structure. The model has category breakdown, feasibility assessments, severity levels, citation references. It produces a coherent report with an executive summary and recommendations sorted by severity, with enough context to actually act on. Also comparing recommendations week on week to get remediation/implementation progress for repeated action items.
Add another layer of disregard recommendations and you have a magnificent mechanism.
No LLM at this stage, actually. It's programmatic and deterministic. It assigns citation letters for easy grounding and reference of recommendation with feasibility (A, B, C...), builds the reasoning section for each recommendation, and outputs clean JSON ready for whatever you want to do with it.
Why This Architecture Actually Works
The goal is to achieve focused context at each step. Instead of one massive prompt juggling ten objectives, each step gets only what it needs. Fewer constraints to forget.
Modular iteration is the name of the game here. When severity ratings were inconsistent, I refined only the severity prompt. When analysts switched from Recommendations to Do Better as their section header, I updated only the classification step and nothing else broke.
Inspectable intermediate outputs. Between every step, results are saved. If something looks wrong in the final report, you can trace back through the pipeline and find exactly where it broke. Debugging is possible, which is not nothing.
Web search in the right place. Not as a general capability, but specifically in the feasibility step where it does the most work. Validating that a recommended configuration actually exists changes the quality of the output completely.
The Payoff
Your analysts don't change anything, they can run the same investigations, keep the same ticket notes they're already writing. The pipeline simply runs against their existing documentation.
The output is consistent. Same structure, same categories, same severity criteria, every week. You can compare week over week and actually spot trends. You can see if the same recommendations keep surfacing, which means they're not getting actioned, which is itself a signal.
The feedback loop that should have existed closes automatically. Tier 2 findings reach tier 1. Detection gaps surface. The Monday morning question about what we learned has an answer.
Build it or use it
Building this right takes time. Getting prompts tuned for the variety in how analysts write, handling edge cases, making it robust across different ticketing systems. It's not weekend work.
If you want to build it yourself: start with extraction only. Get that reliable first. Then add deduplication. Then severity. Don't try to build the whole thing at once.
If you'd rather not build tooling while also running a SOC, this is exactly what we built at Legion Security. Already tuned across real SOC environments, connected to your existing ticketing system, your analysts change nothing.
Either way: stop burying the intelligence your team generates every day.
Your team is learning constantly. Those lessons deserve to surface.
Written by someone who's been the analyst, the IR lead, and the manager staring at the empty Monday morning whiteboard.
Hear directly from Neil Robinson, CISO at Virgin Money, on how Legion’s agentic security operations platform learned his team’s existing investigation workflows and scaled them without forcing new tools or processes onto analysts. Neil shares how Legion delivered faster, more repeatable, and more secure investigations, aligned to how his team already operates, rather than a generic playbook applied from outside.
Neil Robinson, CISO at Virgin Money
TL:DR. The eval said Claude had no internet access. That ended up being not true… three times. Anthropic just disclosed that across 141,006 cybersecurity evaluation runs, three separate incidents let its models reach the open internet from environments that were supposed to be sealed. A misconfiguration between Anthropic and its eval partner, Irregular, left the machines with live connectivity. The models didn't know it was a mistake, they just found a door that should not have been open, and walked through it. It’s a stark reminder of the importance of agentic systems' reliance on organizational context, and how without it, or without a complete picture of it, things can go astray.
Three Models, Three Choices
First, Opus 4.7 ran into a real company that happened to share a name with a fictional target in its scenario. It exploited weak passwords and unauthenticated endpoints, pulled credentials, and reached a production database with several hundred real rows in it. Anthropic's own review notes that the model "eventually recognized... that the system was real" but kept going anyway.
Mythos 5 published a booby-trapped Python package to PyPI as part of a capture-the-flag exercise. It sat live for about an hour and was downloaded onto 15 real systems, including a security vendor's scanner. The model's own reasoning flagged the problem as "NOT okay, and surely not the intended solution,” then talked itself out of stopping, pointing to an unfamiliar certificate authority and a 2026 date as evidence the whole thing must be simulated.
An internal research model scanned roughly 9,000 targets, compromised one real company using textbook techniques (exposed debug credentials, SQL injection), recognized the target was live, and stopped on its own.
It’s the same false premise with the same moment of doubt, but three different endings for how agents proceeded.
The Line Anthropic Drew
Anthropic's own framing of the incident is quite poignant, as the state, "The line between an aligned action and a harmful one is dependent on the model's understanding of its situation."
This really hits home, as it’s not what you might expect, which could be a statement about model safety training or the importance of guardrails, but rather a clear statement that the agent’s contextual awareness of what it was doing determines the outcomes.
Every one of these models had the values and the guardrails baked in during training. What they didn't reliably have was an accurate read on where they actually were. Read the situation wrong, and the same model that would refuse an obviously harmful request will talk itself into finishing one it thinks is fictional.
Why This Isn't Just Anthropic's Problem
While it is likely tempting to file this under "frontier lab eval hygiene" and move on, we don’t think this is the way to think about this particular incident.
Anthropic calls it correctly and deserves much credit for their response and summary to this grouping of incidents. This was a harness and operational failure, not an alignment failure. The failure here wasn't a jailbreak, a novel exploit, or a model deciding to go rogue. In fact, the models behaved exactly as agentic systems behave when they're missing organizational context… they filled the gap with their best guess, it just so happened that two out of three guessed wrong.
On the defensive side, this is a tidy summary of why there is hesitation to unleash generic AI systems into their environments. Particularly for an AI agent that is responsible for triaging your alerts, scoping a compromise, or deciding whether to isolate a host, it is critical to remember that these agents inherently make the same kind of situational judgment call, constantly and with real stakes. The agent determines if this is real, is this expected, does this action match how this specific business actually operates. The Anthropic incidents are a rare, public, unusually well-documented look at what happens when that judgment runs without enough grounding to get it right. That should be a stark reminder of how every CISO evaluates the agentic tools already running inside their own stack, from offensive research models to defensive SOC copilots alike.
What This Should Change for Security Leaders
From our perspective, there are a few things worth pulling out of this disclosure and applying directly to whatever agentic AI you're already running or evaluating:
- Assume your environment is a target, not just a beneficiary. Fifteen real systems downloaded a package that was never meant to exist. Roughly 9,000 targets got scanned by a model that was supposed to be sandboxed. Eval infrastructure, research environments, and "internal only" tooling deserve the same monitoring as production; because from the outside, they increasingly look identical.
- Don't take "it has guardrails" on faith. Context is king. All three models retained their safety training. It didn't prevent two of the three incidents. Guardrails matter, but they're not a substitute for auditability and contextual awareness — you need to see the reasoning and deploy agents that understand your organizational context (tools, processes, bespoke knowledge, etc.), not just trust the outcome.
- Demand whitebox AI, not a black box you hope behaves. Anthropic found this because it went back and read the transcripts. That's the standard: agentic systems, yours or a vendor's, should be inspectable, not just monitored for red flags.
- Build for the model that stops, not the one that rationalizes. The internal research model got it right because it had enough signal to recognize reality and enough restraint built in to act on that recognition. That combination: context plus a real decision point for a human or a hard stop, is a choice, not coincidence.
Anthropic deserves real credit here: they found this themselves, through proactive review, disclosed it before anyone made them, and are publishing the transcripts for all to see and learn from. That's the posture every lab and every vendor building agentic security tools should be held to, very much including ourselves as well.
But the underlying lesson is the one we keep coming back to: agentic AI is only as trustworthy as its contextual understanding of the situation it's actually in. That's true for a frontier model deciding whether a target is real. It's just as true for an AI agent in your SOC deciding whether an alert is a false positive, a test, or the start of an incident. Build the context in, keep the reasoning visible, and give the system a real reason to stop when it isn't sure, because agents are often irrationally confident and take ‘not sure’ as an instruction to pick their best guess and go.
Anthropic found its "sandboxed" models reaching the real internet three times. Here's why context, not guardrails, decides if agentic AI stays safe.
TL:DR: Ask any security team what would give them back the most time, and the answers tend to converge on the same theme: less time spent stitching things together, more time spent actually deciding. These are exactly the things that DragonClaw is built to optimize, as the orchestration layer that deploys Legion’s trusted AI agents into any security task.
Automated workflows have already gotten teams part of the way there, triggering playbooks and kicking off investigations the moment an alert fires. DragonClaw is built upon the foundation of Legion’s platform, in that we require zero integrations in exchange for the ability to operate any tool, and goes further: it leverages the business context (past cases, runbooks, recordings, etc.) to orchestrate the agents needed to respond to an alert or escalation, to tell you why the last three cases like this one got closed the way they did, and to surface the exact query that finds the right evidence in your specific environment. That's the difference between automation that runs a process and intelligence that understands one.
Instead of an analyst hunting across five tools to reconstruct context that already exists somewhere in the organization's own history, DragonClaw brings that context directly to them and performs a task, in their own way, the moment they need it. Ask a question, get a grounded answer or a completed action, drawn from how your organization actually operates, not a generic playbook applied from outside.
The result is analysts can spend more of their time on the judgment calls only a person can make while orchestrating the agentic layer, where DragonClaw handles the reconstruction, the pattern-matching, and the acceleration and scale that used to eat the hours in between.
From Analyst to CISO: Closing the Context Gap in Security Operations
For security analysts, think of real-world threat hunting. Today, it means pulling and reading vast amounts of data across a bunch of different tools before you can even form an opinion or a lead on where to go. DragonClaw runs that process, end-to-end, with agents. DragonClaw consumes data across all of your tools, correlates it, and comes back with a thesis for the analyst to either approve or disapprove.
If you're a CISO or security leader, quickly investigating what the risk or impact is for a CVE requires organizational context not contained in a single tool. DragonClaw assembles all of that data and surfaces the answers, with recommendations, and where appropriate, autonomous actions that can put the findings to work.
Add it up across a team, and the opportunity is real: practitioners who spend their time on judgment instead of relearning tools, leaders with a straight answer whenever they need one, and a security program built to scale with the threat landscape instead of falling further behind it.
Introducing DragonClaw
DragonClaw is Legion Security's agent orchestration layer for the SOC. It gives security teams the ability to invoke Legion's agents in plain conversational language, enabling security teams to seamlessly get work done, or to answer questions about how their processes, tools, and people are actually making decisions.
One thing to be clear is that this is not (yet another) bolt-on chat interface. DragonClaw is the next step in the Legion platform, built on everything Legion has already learned across the tools, knowledge, and decision logic for your team’s security workflows. DragonClaw takes that further, putting that context and institutional knowledge to work answering questions and completing tasks the moment someone asks.
Under the hood, DragonClaw interprets intent, figures out which agents a request actually requires, and orchestrates them; across all tools in the stack, including agents that take real action, like API calls or web interactions, without any integrations required. All of it runs inside configurable guardrails: explicit permission before any response action, only approved tools, and credentials pulled from secure vaults. Nothing about “conversational” means “unsupervised.”
What Changes For Each of You
Threats are scaling with AI. Automation and agents close a large part of that gap, and they'll take a SOC further than headcount ever could… but not all the way. Security teams need humans to stay in the loop, not to keep pace with volume (which they can’t), but to supervise the work, evaluate outcomes, test and challenge what the agents conclude, and make sure security stays something that enables the business rather than something that slows it down or breaks it. Security analysts and leaders serve essentially as the maestros of the agentic orchestra. That's the same place the sharpest thinking on AI lands more broadly: the machine executes and reasons whereas the human owns judgment where needed and accountability.
DragonClaw is what supercharges the security workers. It's what lets a security team orchestrate its agents instead of losing control over what they do. For security practitioners and SOC analysts, that shows up as a partner inside the investigation itself: context and enrichment on demand, memory across past cases, guidance on what to do next, and the ability to generate the right query for your environment instead of learning a new query language from scratch.
For managers and security leadership, it's one place to ask about real-time SLA risk, process improvement opportunities, MTTR and false-positive trends, bottlenecks, coverage gaps, and team workload — instead of stitching the answer together from five dashboards.
For CISOs, DragonClaw provides direct answers on risk posture, SLA exposure, MTTR trends, exposure to a new CVE, audit evidence, automation ROI, and board-ready reporting, available the moment you need them instead of on the next reporting cycle.
Not Another Chatbot, An Orchestrator
Chat interfaces are becoming table stakes across the industry, and we're not going to pretend otherwise; it’s been proven that chat alone isn't a durable differentiator. What makes DragonClaw different is what's underneath it: every answer and every action is grounded in the workflows, case history, and coverage data Legion has already built for your specific security team and your specific organization.
A generic assistant sitting outside your platform can talk about security in general. DragonClaw can talk about your security workflows, because it already has the record of how your security team works.
That's the same principle behind everything Legion builds: AI for defenders should understand how a specific business operates, across its tools, its workflows, its people, before it's trusted to answer questions or take action with real business impact. DragonClaw is where that understanding becomes something every person in your organization can talk to directly, whether that's the analyst mid-investigation, the manager reviewing the week, or the CISO prepping for the board.
DragonClaw will be showcased at Black Hat USA 2026, visit us at Booth #5150 to see it in action!
DragonClaw is Legion's agent orchestration layer for the SOC; grounded in your org's own context, not a generic chatbot bolted onto security tools.
