From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation
The paper evaluates whether very small open-source language models, from 20M to 1.5B parameters, can generate post-exploitation malware-support code for directory enumeration and backdoor creation across Python and Lua under different prompting, task formulations, constrained decoding, and execution-feedback setups. Its interesting takeaway is that capability is not just a function of parameter count: framing the task algorithmically plus few-shot examples can make a 1.5B coder model surprisingly effective, though the study is limited to narrow code-generation tasks and the models are still too heavy for most real-world embedded malware today.
Abstract
Small Language Models (SLMs) are increasingly attractive for resource-constrained environments due to their low computational footprint. These characteristics also make them candidates for future autonomous malware, where connectivity, latency, privacy, or operational constraints may preclude cloud-based inference. Yet the ability of ultra-compact models to support malware during post-exploitation remains poorly understood. We systematically evaluate seven open-source SLMs (20M-1.5B parameters) on two post-exploitation code-generation tasks: Directory Enumeration and Backdoor Generation. We compare Python (high-resource) and Lua (low-resource) across prompting strategies, task formulations, iterative reasoning, grammar-constrained decoding, and execution-feedback loops. Results show that offensive code-generation capabilities do not increase linearly with model size. Successful execution depends on the interaction of model capacity, task formulation, and prompting strategy, with task formulation exerting a greater influence than prompt engineering alone. Algorithmic formulation combined with few-shot prompting enables Qwen2.5-Coder 1.5B to reach Pass@5 of 100% in both Python and Lua for Directory Enumeration, 100% in Python for Backdoor Generation, and 99.2% in Lua. Although the evaluated models remain too computationally demanding for widespread deployment in contemporary malware, rapid progress in compact models may reduce this barrier. These findings help anticipate AI-assisted malware and inform future countermeasures.