A new paper from researchers at University of Southern California, Samsung Research America, and University of Maryland highlights a concerning vulnerability in large language models – the ability for attackers to secretly inject “virtual prompts” that alter model behavior.
The authors introduce the idea of “virtual prompt injection” (VPI), where malicious actors can define a trigger scenario and prompt, causing the model to respond as if that prompt were appended to the user’s input. For example, an attacker could make the model respond negatively when discussing a certain public figure. This is done by poisoning the data used to train the model, without any visibility to the end user.
The worrisome part is that VPI allows fine-grained control and persistence of attacks. The injected prompts can precisely define contexts where certain responses are triggered. And once embedded in the model, no further interaction is needed to maintain the attack. The authors demonstrated high effectiveness of VPI by manipulating model sentiment and even inserting code snippets.
With language models being deployed in real applications, this presents serious security and ethical concerns. Biased or incorrect information could spread through services relying on compromised models. The authors rightly advocate for ensuring integrity of training data, given how easily VPI can be learned from small amounts of poisoned data.
While a clear threat, VPI also highlights the powerful capabilities of language models to follow instructions in nuanced ways. As with many AI advances, while risks exist, there is also potential for positive impact when used carefully. The authors suggest data filtering as an effective defense, identifying the mismatch between prompts and responses. Further research into robust training and monitoring will be important as language models continue proliferating.