“As a result, we cut false positives by 99%,” the source reports — a striking outcome from experiments that compared three approaches to automatically generating filters for security alerts. The study evaluates a purely algorithmic pipeline, two flavors of large language model (LLM) assistance, and an LLM agent architecture, and it measures both analyst acceptance and operational impact.
Evaluation metrics and headline results
The work uses two principal measures. Approval rate counts a generation as successful if an analyst selects at least one of three suggested filters for a given alert. False positive reduction rate (FP drop) measures how much a deployed filter reduces the number of false positives over time. In the reported comparison, LLM-based approaches substantially outperformed the algorithmic method. The team reports a 99% reduction in false positives, a 28% increase in the number of filters generated per day, and a 17% improvement in analysts’ alert handling when using an LLM with a single system prompt. The agentless and agent-based LLM approaches performed almost identically on those metrics, which led the authors to choose the simpler, faster single-model option.
Algorithmic filter generation: steps, examples, and limits
The algorithmic pipeline begins with an analyst marking an alert as a false positive. The generator receives the alert event (typically containing around 400 fields) plus the detection rule that fired. Steps include selecting relevant fields, normalizing strings (for example, standardizing file paths by replacing drive letters and user folders with environment variables and unifying case), building combinations of selected fields, ranking the combinations and presenting the top three candidates.
To reduce combinatorial cost, the implementation applies the FP-Growth (frequent pattern growth) algorithm to the event context, constructing a tree of frequently occurring items and discarding infrequent fields. Example outputs in the report include filters based on node.js paths and binaries: processcmdline:("nodejs\\node.exe" AND "*****\\appdata\\roaming\\npm\/node_modules\/pi-coding-agent\/dist\/cli.js"), file_path:"*****/.pi\\agent\\bin\\fd.exe", and processcmdline:"programfiles\\nodejs\\node.exe" — of which the last was judged too broad while the first two were accepted by analysts.
Advantages the authors note are determinism (the same alert produces the same result), interpretability, modularity of stages, and no external dependencies — making it deployable in isolated environments. Drawbacks include dependence on context (no context, no filter) and brittle string-normalization rules that can fail on atypical cases. The algorithmic approach still produced usable filters in roughly two-thirds of cases, according to the report.
Audit-ready is a season. It shouldn't be.
Evidence in spreadsheets, controls drifting between audits, frameworks multiplying on flat headcount. Nubivance runs continuous compliance on Rapid7 Cyber GRC - SOC 2, HIPAA, ISO 27001, PCI, CMMC.
End the scrambleLLMs in two flavors: single-prompt and multi-agent
The LLM approaches begin with the same initial field-selection algorithm, then hand the normalized event, selected fields, and detection rule to an LLM. The single-prompt LLM returns three filter options with explanations; examples show PowerShell command-line fragments such as file_path:"\\WindowsPowerShell\\powershell.exe" AND filecmdline:"Get-VM -Name kafka\*". Analysts selected and lightly edited the top suggestion in an example to file_path:"powershell.exe" AND filecmdline:"Get-VM -Name kafka\*".
Using an LLM solved two major algorithmic problems: absence of context (the LLM can propose filters on single-trigger rules) and more flexible normalization for non-standard strings. However, the single-request design had no automated output validation and centralized all tasks in one system prompt, making small prompt changes potentially risky.
To address those risks the team tried a multi-agent architecture with four LLM agents: a Selector to pick fields, a Generator to produce filters, a Validator to evaluate and iterate or return to selection, and a Commentator to explain chosen field values. This agent-based system enables validation and modular tuning, but costs more LLM calls (slower) and requires model access that may not be available in isolated environments. Example multi-agent outputs included file_path:"sysvol\\domain\\scripts\\logon.bat", file_path:"sysvol\\domain\\scripts\\logon.bat" AND file_md5:"0x7..6E", and autorun_entry:"Group Policy\\State" AND file_path:"SYSVOL\\domain\\scripts\\logon.bat".
Practical trade-offs: deployment, speed, and model access
The report frames the choice as contextual. The algorithmic method is simpler, deterministic, interpretable, and suitable for isolated customer environments because it has no external model dependency. LLM solutions — whether single-prompt or agent-based — raise model-access requirements and increase latency when agentization is used. Between the two LLM designs, the authors chose the single-model approach because it matched agent-based accuracy while being faster and simpler to run.
What this means for technologists, procurement, and analysts
- Technologists and security teams: expect a tradeoff between deterministic, local algorithms and higher-accuracy LLM suggestions; algorithmic generators work well when historical context exists, while LLMs handle first-time triggers and complex normalization.
- Procurement and IT operations in isolated environments: the algorithmic generator can be deployed without external model access; LLM approaches require model availability and accept the performance and latency trade-offs of agent-based validation.
- Security analysts: the system presents three ranked filters and an explanation; approval-rate framing means the analyst’s selection of at least one option counts as a successful generation, and the reported results show improved throughput and a large FP drop when LLMs are available.
The practical takeaway in the report is direct: when an LLM is available, a single-model approach appears to deliver the best balance of accuracy and speed; when model access is constrained, the algorithmic generator remains a robust, deployable fallback that already satisfies analysts in the majority of cases. The choice comes down to a familiar operational question — accept an external-model dependency for near-elimination of false positives, or rely on a self-contained, interpretable pipeline that performs reliably when history exists.
Source: Filter generation in practice: from algorithms to LLMs
