Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching
The paper introduces ISM, an attack on LLM-agent skill selection where an attacker tweaks a skill’s metadata and reusable prompts so their semantic match looks naturally best, causing the selector to pick the attacker’s skill without any obvious prompt-injection language. The interesting bit is that it exploits embedding/semantic matching behavior rather than instruction following, making it much stealthier than explicit steering across their tested domains and defenses, though the evidence is still bounded to four task domains, eight selector models, and assumes the attacker can influence skill metadata/prompts.
Abstract
Skill selection is a key stage in LLM-agent workflows, determining which installed skill should handle a user request. Existing attacks on this stage primarily rely on explicit prompt injection or instruction-level steering, which can expose recognizable manipulation signals. In this work, we identify a new implicit attack surface for skill selection: even when the user prompt and skill description appear benign in isolation, their semantic relationship can still be strategically shaped to favor an attacker-chosen skill. Based on this observation, we present Implicit Skill-Selection Manipulation via Semantic Matching (ISM), which jointly shapes target-skill metadata and reusable prompts to manipulate skill selection without explicit selection instructions. Specifically, we develop a three-stage strategy to broaden semantic coverage, strengthen target distinctiveness, and preserve natural prompt wording. Across four task domains and eight selector models, ISM increases the average target-selection rate (TSR) from 15.2% to 63.5%. In a matched comparison, ISM achieves a 73.5% TSR, only 9.8 percentage points below Explicit Steering. Human reviewers block ISM in only 2.9% of judgments, versus 91.4% for Explicit Steering, while five LLM-based inspectors pass ISM at an average rate of 82.9%, versus 37.4% for Explicit Steering. Moreover, ISM remains effective against PPL-W, Llama Prompt Guard 2, and PIGuard.