Closing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages

Summary

Supply chain attacks on the npm registry are growing, and packages that bundle a binary payload are rare but disproportionately hard for LLMs to triage. The authors built a scanner that combines static analysis with LLM reasoning and introduced capability profiles, which encode malicious techniques as weighted conditions injected into the prompt before the model decides. They tested nine models across seven providers on five adversarial payload types, including credential stealers and reverse shells, at varying obfuscation levels. Without profiles, the budget-tier models failed to reach a malicious verdict on 50 percent of tests, with failures following distinct model-specific patterns rather than random error. With profiles enabled, the failure rate dropped to 4 percent, and 91 percent of prior failures recovered to a detection.

Takeaways

1. 1

   Binary-bearing npm packages are rare (3.8% of confirmed malware) but fail to reach a malicious verdict more than twice as often as source-only packages (14.4% vs 6.2%).

2. 2

   Without capability profiles, budget LLMs failed on 50% of binary payload tests, with failures following model-specific patterns rather than random error.

3. 3

   Injecting capability profiles into the prompt reduced budget-tier failures from 50% to 4%, recovering 91% of prior detection failures.

FAQ