Resources
SOC continuous improvement fails when insights get buried in closed tickets. Learn a 6-step LLM pipeline that turns investigation notes into action.
I was there, I sat in every SOC seat out there…
A SOC analyst grinding through alert queues at 2am. Part of an Incident Response team leading running war rooms. A SOC manager in Monday morning stand-ups asking what we learned this week while staring at blank faces.
Every single role. Every single day. And the one thing that never changed across any of them?
The insights, recommendations, self improvement, the de-facto SOC continuous improvement action items were disappearing. Seating documented in a case log for no one to action upon, trapped inside closed tickets that live in a backlog nobody rarely reopens.
I know the why and I feel the overwhelming operations, which is why I’m offering a practical solution for how to continuously improve your SOC with the valuable insights coming out of your investigations.
The Hidden Goldmine You're Sitting On
Every ticket your team closes tells a story. It's not just that an alert fired, then an analyst investigated and eventually closed. There are powerful signals buried in those notes, whether it's a tool with overly noisy alerts, a gap in your email gateway rules, or the same user clicking a phishing link for the third month in a row.
Your tier 1 all the way to your tier 5 analysts and IR responders are generating intelligence every single shift and with every single incident. They know things and they're writing them down. It's useful information but these notes get buried and never read again.
It's a sad truth... I know because I've been in those weekly SOC meetings, I was running them.
It's not a people problem, rather, it's a system problem.
The Weekly Report Trap
The thing people look to as the standard fix is the weekly report. In theory it's elegant: senior analysts summarize the week, extract the learnings, feed them back into tier 1 runbooks and detection improvements. On paper, it's a proper feedback loop.
In practice, it becomes the task that either gets rushed on Friday afternoon or simply doesn't happen. It's for good reason too! Your senior analysts are already stretched because on top of everything they need to do for their jobs, they're also being asked to synthesize everything in themes. You either get a half-hearted copy-paste of ticket titles, or, more likely, you get nothing.
Teams try rotation where everyone takes a turn on the ferris wheel. But in doing so, you face losing important insights and information, not to mention a lack of consistency.
Now add a follow-the-sun operation to this. APAC closes tickets while EMEA is asleep. EMEA handles incidents while Americas is offline. By the time anyone tries to compile a summary, they're working with fragments. Nobody has the full picture. The patterns that only emerge when you look across all shifts stay invisible.
Wait, Can't AI Can Solve This Pretty Easily?
When capable LLMs became available, I thought this was finally solved. Just feed all the investigation summaries in, ask for a weekly report. Done? Not so fast... here's what actually happened.
First attempt: I gave the best LLM models that money can buy more than 250 investigation summaries and asked for a consolidated report. But what I got back was a mess.
What I saw were recommendations repeated five times just with slightly different wording. Severity assessments that made no sense and my “favorite” recommendations that are not feasible, for example “Tune your EDR machine learning to reduce false positives of macro xlsx files”.
No traceability whatsoever, no way to tie anything back to the original investigation and forget about cross referencing with similar recommendations.
Second attempt: I went deep on prompt engineering. Longer prompts. More detailed. With examples. The results improved marginally, but the ceiling was surprisingly low.
The fundamental issue is that when you dump a large context with complex requirements into a single LLM call, it can't hold everything in working memory. It forgets constraints from earlier in the prompt. It hallucinates connections between unrelated incidents. Severity levels come out inconsistent.
One-shot approaches get you mediocre fast. They don't get you useful.
The Breakthrough: Think Multi-Step, Not Prompt
The shift that changed everything was stopping thinking about this as one task and starting to think about it as a multi-step pipeline.
When an experienced analyst writes a weekly report, they don't try to do it all at once. They read, they group, they prioritize, they write. Multiple steps. Each one is different.
So I built it that way.
The 6-step pipeline
Step 1: Classification
The first step does one thing and one thing only. It extracts and categorizes recommendations from raw investigation summaries. It looks for whatever your analysts call them: Recommendations, Do Better, Action Items, Next Steps. It pulls each one out and assigns it to a category: detection, prevention and process improvements.
No dedupe. No severity. Just extraction, done well.
Step 2: Feasibility Assessment
Now we evaluate each recommendation against practical reality. Can this actually be implemented? Is it a quick win or a multi-quarter project? Does it require resources you don't have?
This is also where web search earns its keep. When a recommendation references a specific product or vendor, the model can look up current best practices, product documentations, tech community discussions and verify the suggested configuration actually exists and is supported. Without this, you get generic, often infeasible advice. With it, you get grounded recommendations.
Make sure to use an LLM model that has web search capability via API calls.
Step 3: Citation Attachment
Before touching deduplication, every recommendation gets linked back to its source investigation. This is non-negotiable for a report anyone will actually act on. When a SOC manager reads and SOC teams attempt recommendation implementation, they need to know which investigations triggered that and value with volume justification to it. Otherwise it's just noise or worse, it might break business operations.
Step 4: Deduplication
Three analysts working three separate investigations but same use case, all recommend the same prevention improvement. Without deduplication, you get three entries saying the same thing with slightly different wording. With it, you get one consolidated recommendation that shows it came from three independent investigations, which is actually a stronger signal.
Citations from all source recommendations get merged. Nothing is lost.
Step 5: Severity Classification
Now, with duplicates consolidated, we can assign severity levels that actually mean something. The model evaluates security impact per your instructions, weights and SOC defined severities for each use case. Not how urgent did the analyst feel when writing this, but what is the actual risk if this doesn't get addressed built on your SOC knowledge base.
Separating this from extraction forces objectivity. If you try to assign severity while also pulling recommendations from raw notes, the analyst's tone bleeds in and skews the assessment.
Step 6: Report Generation
Everything feeds into the final structure. The model has category breakdown, feasibility assessments, severity levels, citation references. It produces a coherent report with an executive summary and recommendations sorted by severity, with enough context to actually act on. Also comparing recommendations week on week to get remediation/implementation progress for repeated action items.
Add another layer of disregard recommendations and you have a magnificent mechanism.
No LLM at this stage, actually. It's programmatic and deterministic. It assigns citation letters for easy grounding and reference of recommendation with feasibility (A, B, C...), builds the reasoning section for each recommendation, and outputs clean JSON ready for whatever you want to do with it.
Why This Architecture Actually Works
The goal is to achieve focused context at each step. Instead of one massive prompt juggling ten objectives, each step gets only what it needs. Fewer constraints to forget.
Modular iteration is the name of the game here. When severity ratings were inconsistent, I refined only the severity prompt. When analysts switched from Recommendations to Do Better as their section header, I updated only the classification step and nothing else broke.
Inspectable intermediate outputs. Between every step, results are saved. If something looks wrong in the final report, you can trace back through the pipeline and find exactly where it broke. Debugging is possible, which is not nothing.
Web search in the right place. Not as a general capability, but specifically in the feasibility step where it does the most work. Validating that a recommended configuration actually exists changes the quality of the output completely.
The Payoff
Your analysts don't change anything, they can run the same investigations, keep the same ticket notes they're already writing. The pipeline simply runs against their existing documentation.
The output is consistent. Same structure, same categories, same severity criteria, every week. You can compare week over week and actually spot trends. You can see if the same recommendations keep surfacing, which means they're not getting actioned, which is itself a signal.
The feedback loop that should have existed closes automatically. Tier 2 findings reach tier 1. Detection gaps surface. The Monday morning question about what we learned has an answer.
Build it or use it
Building this right takes time. Getting prompts tuned for the variety in how analysts write, handling edge cases, making it robust across different ticketing systems. It's not weekend work.
If you want to build it yourself: start with extraction only. Get that reliable first. Then add deduplication. Then severity. Don't try to build the whole thing at once.
If you'd rather not build tooling while also running a SOC, this is exactly what we built at Legion Security. Already tuned across real SOC environments, connected to your existing ticketing system, your analysts change nothing.
Either way: stop burying the intelligence your team generates every day.
Your team is learning constantly. Those lessons deserve to surface.
Written by someone who's been the analyst, the IR lead, and the manager staring at the empty Monday morning whiteboard.
The vision of a fully automated SOC is finally within reach, piece by piece. But as we see it, building automation around a theoretical SOC, not a real one, is the wrong approach for enterprise companies.
Some focus on Tier 1 alerts: the repeatable, mundane tasks like standard phishing playbooks. Others tackle Tier 2+ investigations, where human judgment is still essential. Both are valid, but they miss the core reality of how a SOC operates today.
The goal of any AI SOC analyst isn't to replace your team, but to automate and improve the way they actually work.
Right now, your analysts are stuck in browser tabs, pivoting between consoles, copying data, and piecing together the truth manually. This isn’t scalable or efficient. It's why we founded Legion. Our vision is simple: your SOC lives in the browser. Your AI SOC analyst should build automation that reflects exactly that.
How SOC Workflows Actually Work Today
The modern SOC runs on people, browsers, and disconnected tools. Here's what that looks like in practice:
- Data Ingestion: Data (IPs, threat intel, logs, etc) is pulled from multiple sources and correlated
- Detection Engineering: Rules are written, tested, and updated based on what was missed or what created noise.
- Alert Triage: Analysts spend their day pulling data from different systems to figure out if an alert is real or just noise.
- Threat Hunting: Proactive hunts are a mix of experience and manual queries. Results are often shared ad hoc in Slack or documents, rarely in a repeatable format.
- Deeper Investigations: When an alert is valid, the manual pivot begins. Analysts jump between logs, threat intel feeds, and internal assets to gain context. Every jump between tools and content loses context.
- Remediation Actions: Depending on the validity of the alert, remediation actions are completed, and/or the ticket is closed out.
- Reporting & Incident Summarization: Building an incident timeline and report is a manual process of collecting screenshots, logs, and notes stitched together by hand.
- Process Hand-Offs: Shift changes and escalations often drop critical context because investigations aren’t documented in a structured way.
Author: Filip Stojkovski, Cybersec Automation
The main point is that most SOC workflows today are repetitive but lack standardization. Even if organizations have created playbooks within their SOAR or workflow automation tools, they are likely outdated or incorrect because automation is not handled by the analysts. The engineers do it.
An Honest Introduction to Automating the SOC
I’ve spent the last 90+ days digging into the AI SOC Analyst and SOAR market, talking to customers, analysts, and more.
Automating the SOC is not an easy problem to solve. Anyone who tells you their tool will work magically out of the box on day one is selling you a fantasy, and Legion is not here to tell you it will either.
- Some alerts are predictable, but many are context-dependent and demand human judgment.
- Integrations break. APIs can make things easier, but still need to be managed.
- And through it all, your good analysts remain your most valuable asset. Automation should make them faster and more effective, not try to replace them.
Legion's approach is built on this reality.
How Legion Security Automates SOC Workflows
Legion’s approach is built on one simple principle: the SOC lives in the browser. Analysts do their real work inside SaaS consoles, cloud admin panels, EDR dashboards, and threat intel portals, all in the browser. That’s where detections are reviewed, logs are queried and analyzed, and decisions are made.
Instead of forcing your team into an abstract "playbook tool" built on API connections, Legion instruments the browser itself. This gives you a clear view of what an analyst clicks, searches, copies, and correlates. This is the actual audit trail of how investigations and responses are conducted. This visibility is (we believe) the best way for automating workflows that actually match how your team operates.
Legion breaks this down into three practical, trust-based modes:
- Recording Mode: Legion captures every step your best analysts take. It watches how they handle triage, pull context, enrich data, and close cases. This builds a bank of proven workflows, not theoretical runbooks. These recordings become reusable playbooks grounded in real analyst behavior.
- Guided Mode: Next, Guided Mode turns those recordings into automations. When a new alert comes in, the analyst runs the investigation AI-in-the-loop, where Legion completes the investigation and provides recommendations for next steps at each decision node. Junior analysts don’t have to start from scratch. The guidance is readily available, right inside their workflow. This closes skill gaps and standardizes how your team works.
- Autonomous Mode: Finally, Legion can run trusted workflows end-to-end in Autonomous Mode. But only for well-understood, repeatable scenarios you've already vetted. When a ticket is opened, Legion executes the steps your team already does manually. There's no black-box decision-making or surprise actions outside what you’ve already proven works.
By focusing on how your real analysts work and only automating what they’ve shown to be effective, Legion enables you to build true automation that adapts and improves over time.
Use Cases for the Legion AI SOC Analyst
- Workflow Documentation: Create comprehensive workflow maps of how your SOC analysts handle alert triage and investigations.
- Alert Triage & Investigations: Automate noisy Tier 1 triage, enrich alerts with context, and auto-close junk. These can include cloud, identity, phishing, vulnerability management, and more. Because we are not limited by integrations, Legion can automate any SOC workflow.
- Reporting & Incident Summarization: Generate incident timelines and report on key metrics such as MTTA/MTTR.
- Process Improvement: Spot process gaps and bottlenecks, and optimize workflows across analysts.
- SOC Training: Don’t let your tribal knowledge leave with your best analysts. By mapping out your processes, your junior analysts can train by “looking over the shoulder” of Legion in guided mode.
Final Thoughts
SOC automation shouldn’t be magic (even if it feels like it sometimes). It's grounded in observing, guiding, and learning from your real workflows.
Legion’s AI SOC analyst doesn’t pretend to replace humans. It records what your best people do, guides new analysts, and automates the repeatable. Over time, your analysts can focus on improving workflows, upleveling their security skills, improving detections, and more. Automate your SOC the way your team actually works with Legion.
Your AI SOC should automate your team's real workflows, not theoretical playbooks. Learn how Legion's approach helps security teams improve processes for faster, more effective security operations.
