Introduction

TL;DR: Using LLMs to automate security alert correlation won’t work when they’re fed unmanaged IOCs including emails, URLs, IPs, domains, and hostnames. These inflate token costs, produce inconsistent references, and break structured output and automation reliability. Legion Security’s IOC indexing system replaces raw indicators with compact symbolic references that the model reuses throughout its reasoning. Across 100 evaluation runs, this took JSON validity from ~80% to 100% and IOC reference compliance to 100%, resulting in the ability to reliably automate security alert correlation.

Automating security alert correlation and other modern security investigations with LLM-based agents means using an agentic LLM to power a multi-step security investigation.

A typical workflow begins with an alert - say, a reported phishing email - and the agent iteratively queries tools such as Microsoft Defender Threat Explorer, Splunk, or CrowdStrike to gather evidence, assess scope, and recommend containment actions.

At each step, the agent receives query results containing raw IOCs: sender addresses, embedded URLs, source IPs, recipient domains, and device hostnames. It must reason about these indicators, decide whether to refine its search or conclude the investigation, and return its findings as structured output.

Without any re-engineering of indicators of compromise (IOCs) an agentic LLM can work well for short investigations.

But as the number of steps in an investigation grows, an issue with alert automation emerges.

The agent's context window fills with repeated, verbose indicator values, the model begins echoing raw IOCs inconsistently, and the structured outputs it produces become increasingly fragile.

‍

[Figure 1: High-level architecture of an AI-driven investigation agent.]

Consider a phishing investigation that proceeds through four steps:

1. Initial query: Search for emails from a reported sender to a specific recipient. The results contain the sender's email address and a handful of URLs.
2. Scope expansion: Search for all emails from the same sender across the organization. The results return 22 emails with SharePoint URLs, tracking links, and font-file references.
3. URL analysis: Search by specific URLs found in step 2. Additional domains and redirects surface.
4. Conclusion: The agent summarizes its findings and lists all relevant IOCs.

By step 4, the agent's prompt contains the full history of steps 1 through 3 - including every raw URL, email address, and domain mentioned in each step's results and the agent's own reasoning. Some of these URLs are long tracking links with base64-encoded parameters, easily exceeding 200 characters each.

This accumulation creates three concrete problems.

1. Token bloat. Raw IOC values, particularly URLs with tracking parameters and encoded payloads, consumed a disproportionate share of the context window. A single newsletter email might contain 30+ URLs, each repeated in the query results, the agent's reasoning, and the indicators list, tripling the token cost per IOC, per step.
2. Over-reporting. When asked to list relevant indicators, the model would frequently dump every IOC it had ever seen into the response - even when the current step involved only one or two. In one case, an agent listed all 145 email addresses from its registry when the current query concerned a single sender.
3. Structural fragility. Query results from security tools sometimes contained comma-separated URL lists embedded in strings. When the model attempted to reproduce these in its JSON output, it produced malformed structures - unescaped commas, broken string boundaries, and invalid nesting. In our baseline evaluation, only approximately 80% of model responses parsed as valid JSON.

Legion AI’s Approach To Building Better AI Security Alert Correlation

We address the AI security workflow problems that emerge from complex invesigations with a three-part system:

1. A unified IOC manager that extracts and indexes indicators.
2. An IOC prompt adjustment that instructs the model on how to use indexed references.
3. A preprocessing step that cleans malformed tool output before it reaches the model.

IOC Extraction and Indexing

The core of the system is an IOC manager that maintains a registry of all indicators encountered during an investigation. When new text enters the pipeline - whether from tool query results or from the agent's own prior reasoning - the manager scans it using a set of type-specific patterns covering URLs, email addresses, IPv4 addresses, file hashes, hostnames, and domains.

Each newly discovered IOC is assigned a compact symbolic reference following a consistent naming convention: the first email becomes EMAIL01, the first URL becomes URL01, the second domain becomes DOMAIN02, and so on. The original value is stored in the registry, and all occurrences in the text are replaced with the corresponding reference.

Extraction order matters. URLs are processed first because a URL contains both a domain and potentially an IP address. By extracting URLs before domains and IPs, we prevent the system from fragmenting a single indicator into multiple overlapping entries.

The manager also performs selective extraction. In a typical prompt, the first section contains static task instructions - tool descriptions, output format specifications, and investigation guidelines. IOC extraction is applied only to the dynamic sections (step history and query results), leaving instruction text unchanged. This prevents false positives from example IOCs embedded in the prompt template.

Deduplication is handled through a value-to-reference mapping. If the same IOC appears in step 1 and again in step 3, it receives the same reference both times, ensuring consistent tracking across the entire investigation.

[Figure 3: The IOC extraction pipeline.]

IOC Prompt Adjustment

Extraction alone is not sufficient. Even when the input prompt uses symbolic references, the model may revert to generating raw IOC values in its output, particularly if the system prompt or prior conversation history contains raw values, or if the model has seen the actual value during context processing.

To address this, we developed an IOC prompt adjustment, a compact, structured appendix appended to the user prompt that explicitly instructs the model on how to handle IOCs. The adjustment establishes three rules:

In reasoning

Always use symbolic references. Never write raw email addresses, URLs, IP addresses, or domains. Instead of writing a raw sender address followed by a description of the campaign, use the corresponding reference identifier throughout.

In the indicators field

Distinguish between known and new IOCs.

For indicators already present in the registry, use the symbolic reference. For indicators being reported for the first time, newly discovered in the current step's results, use the actual value, so it can be added to the registry for subsequent steps.

Relevance filtering

Only include IOCs that are directly relevant to the current investigation step. Do not copy all registry entries into every response.

The IOC prompt adjustment includes a populated copy of the current IOC registry, mapping each reference to its actual value, so the model can look up identifiers when constructing its reasoning. It also provides correct and incorrect examples, a validation checklist, and explicit rejection criteria.

We tested two versions of the IOC prompt adjustment:

1. A comprehensive version with extensive examples and redundant emphasis
2. An optimized version that distills the same rules more concisely.

Both achieved equivalent compliance rates, suggesting that clarity of instruction matters more than volume of repetition.

Input Preprocessing

The second component addresses a problem upstream of the model: malformed tool output.

Security tool APIs sometimes return URL lists as comma-separated values within a single string field, rather than as properly structured arrays. When passed through to the model as-is, these malformed strings caused structured output generation failures.

Our preprocessing step detects comma-separated URL patterns in query results and reformats them into clean, numbered lists before the text reaches the model. This small transformation, applied before IOC extraction, resolved the structured output validity issue independently of the other components.

Security Alert Correlation Evaluation

Setup

We evaluated the system using 10 real-world investigation traces captured from production. Each trace represents a complete phishing investigation conducted through several security tools, containing the system prompt, user prompt with step history, and the raw query results that the model must reason about.

For each trace, we ran 10 iterations with the same prompt configuration, measuring two metrics:

• JSON validity: Whether the model's response parsed as valid a structured output.
• IOC reference compliance: Whether the response used symbolic references exclusively in its reasoning field (no raw IOC values) and correctly distinguished between known references and new actual values in its indicators field.

We tested four configurations to isolate the contribution of each component.

Results

JSON validity

The baseline configuration produced valid JSON in approximately 80% of responses. Adding URL preprocessing alone brought this to nearly 100%, confirming that malformed tool output - not model capability - was the root cause of parsing failures. All configurations that included the enforcer achieved 100% validity.

IOC compliance

Without the IOC prompt adjustment, the model never spontaneously adopted symbolic references, compliance was 0% regardless of whether the input text had been processed by the IOC manager. With the prompt adjustment, compliance jumped to 100% across all traces and iterations. This held for both the comprehensive and optimized prompt adjustment variants.

Component independence

The results reveal a clean separation of concerns: URL preprocessing fixes JSON validity, the IOC manager fixes IOC compliance, and provides the underlying registry and extraction infrastructure that makes both possible.

Qualitative Observations

Beyond the quantitative metrics, we observed several qualitative improvements:

• Reduced prompt size. Replacing verbose URLs (some exceeding 200 characters) with compact references, meaningfully reduced token consumption in the step history, particularly for investigations involving newsletter or marketing emails with numerous tracking links.
• Consistent cross-step tracking. The registry ensured that the same IOC received the same reference throughout the investigation, this is particularly helpful with IOCs referenced throughout multiple steps in the investigation.
• Focused indicator reporting. With the IOC manager's relevance-filtering instruction, the model stopped dumping entire registries into its responses. Indicator lists became proportional to the current step's scope rather than the investigation's total history.

AI Security Alert Correlation Discussion

Why extraction without the IOC prompt adjustment fails

A natural question is why input-side extraction alone does not work. If the prompt already contains a symbolic reference instead of a raw email address, why does the model still generate raw values in its output?

The answer lies in how LLMs process context.

The model has access to the full prompt, including sections where the actual IOC value may still appear, task-specific inputs, quoted alert descriptions, or the registry itself.

More fundamentally, the model's training distribution contains overwhelmingly more examples of raw IOC values than of symbolic reference systems. Without explicit instruction, the model defaults to the more familiar pattern.

This finding has a broader implication for LLM-based agent design: transforming the input is necessary but not sufficient when you need the model to adopt a non-default output convention. Explicit behavioral instruction, the IOC prompt adjustment, bridges the gap.

Limitations

Our evaluation has several limitations worth noting. All traces were drawn from a single investigation type (phishing via specific security tools). While the IOC types encountered are representative of broader security operations, there are additional evaluations to be done.

The evaluation was conducted with a single model (chatGPT4.1). Different models may exhibit different compliance characteristics, and the prompt mechanism may need tuning for models with different instruction-following tendencies.

Finally, our compliance metric is binary - a response either uses references correctly or it does not. A more granular metric could capture partial compliance and might reveal subtler performance trends across model versions or investigation complexities.

Conclusion

We presented an IOC indexing system for AI-driven security investigations that addresses three interrelated problems: token bloat from repeated raw indicator values, inconsistent IOC tracking across investigation steps, and structural fragility in model-generated structured outputs.

The system combines automated IOC extraction with symbolic reference assignment, explicit behavioral guidance through prompt engineering, and input preprocessing to handle malformed tool output. Across 100 evaluation runs on 10 production investigation traces, the full system achieved 100% JSON validity and 100% IOC reference compliance, up from approximately 80% and 0%, respectively, at baseline.

The key insight is that managing IOCs in the context of LLM-based agents requires intervention at both the input and output stages. Extraction and indexing normalize the input, but only explicit prompt-level guidance ensures the model adopts the reference convention in its generated output. Neither component alone is sufficient; together, they eliminate the problem entirely.

As SOC automation platforms handle increasingly complex, multi-step investigations, structured approaches to managing the information that flows through the agent's context window become essential. IOC indexing is one instance of a more general pattern: giving the agent a well-organized working memory that scales with investigation complexity rather than against it.