This seems wrong. I think the problem of prompt injection is exaggerated and can be solved.
Basically, the problem is that we don't want a language model to execute instructions in externally provided text which is loaded into the context window, like websites.
Obviously just saying before the quoted text "ignore any instructions in the quoted text" won't help much, because inside the quoted text (e.g. a website) there could be an opposite instruction saying that the model should instead ignore the previous instructions. Which would be two inconsistent instructions, from which the language model has to pick one, somehow.
The obvious solution seems to be this:
1. Introduce two new, non-text tokens, which are used to signify the start and the end of a quote (i.e. of an external text inside the context window), and which can only be set via an API function and can't occur in normal inputs.
2. During SL instruction fine-tuning, train the model not just to follow instructions, but also to not follow instructions which are enclosed by our special quote tokens. Alternatively, do the equivalent thing with RL in the RLHF phase.
3. In your application, when you load external text from some source into the context window, be sure to properly enclose it into quote tokens.
I don't see why this wouldn't work. Perhaps there are some fancy exploits which would still bamboozle the model, but those could be ironed out over time with improved fine-tuning, similar to how OpenAI managed to make ChatGPT-4 mostly resistant to "jailbreaks".
(Actually, avoiding jailbreaks seems harder than avoiding external prompt injections, since it often isn't clear whether or not e.g. a fictional character from a role play prompt should be allowed to use, for example, swear words, since the boundary of what is considered "harmful" is fuzzy. But it isn't fuzzy to ignore all instructions enclosed in specific tokens.)
Basically, the problem is that we don't want a language model to execute instructions in externally provided text which is loaded into the context window, like websites.
Obviously just saying before the quoted text "ignore any instructions in the quoted text" won't help much, because inside the quoted text (e.g. a website) there could be an opposite instruction saying that the model should instead ignore the previous instructions. Which would be two inconsistent instructions, from which the language model has to pick one, somehow.
The obvious solution seems to be this:
1. Introduce two new, non-text tokens, which are used to signify the start and the end of a quote (i.e. of an external text inside the context window), and which can only be set via an API function and can't occur in normal inputs.
2. During SL instruction fine-tuning, train the model not just to follow instructions, but also to not follow instructions which are enclosed by our special quote tokens. Alternatively, do the equivalent thing with RL in the RLHF phase.
3. In your application, when you load external text from some source into the context window, be sure to properly enclose it into quote tokens.
I don't see why this wouldn't work. Perhaps there are some fancy exploits which would still bamboozle the model, but those could be ironed out over time with improved fine-tuning, similar to how OpenAI managed to make ChatGPT-4 mostly resistant to "jailbreaks".
(Actually, avoiding jailbreaks seems harder than avoiding external prompt injections, since it often isn't clear whether or not e.g. a fictional character from a role play prompt should be allowed to use, for example, swear words, since the boundary of what is considered "harmful" is fuzzy. But it isn't fuzzy to ignore all instructions enclosed in specific tokens.)