OP is right, forget about delimiters and prompt strategies, this is a classic CS problem, you can't sanitize user input if it's mixed up with "code". All possible solutions involve a significant change of architecture.
This is a human problem too. It's not limited to AI. Think about the Two-man rule in nuclear weapon activation. We can't trust one neural-net to receive and confirm the "launch" prompt as valid, so we use two people to increase guarantees. It's called SoD , segregation of duties, by risk management people.
Some architectural changes to how LLM transformation works could include:
- Create a separate tokenizer for system prompts, so that system embeddings are "colored" differently in the model. This will, however, complicate training and bloat the model into uncharted computing territory.
- Create a separate set of pre and post-prompt AI sanitizers that DO NOT use user input or generated output as part of its instruction. Text in and out of LLM is always tainted, so it's a goal to avoid it as input as much as possible.
Simple classifiers can be used for sanitation, but they tend to be "post facto": given a prompt injection scheme comes to light or an prompt injection incident is reported, train on it. More sophisticated intention analyzers, backed by deep classifiers that are uncertainty-aware, and beefed up by LLM generational tools pretrained on synthetic injection schemes, could probably detect ill-intention accurately in the same manner sentiment analysis can pick up on very subtle user queues.
The issue is that classifiers would still be dealing with marshaled code+user input. I believe the better option for intention classifier inputs is to use the model processing data (ie. a "generation log") as input to the classifier, similar to how the ventromedial prefrontal cortex and amygdala connect, acting as behavior moderation neural nets in us humans. This would typically be done by adding specialized multi-head attention focus areas in the GPT architecture without the need for separate classifiers, just basic training about what is good and bad for the AI, but then we're back at the original problem of dealing with the input text directly.
> This is a human problem too. It's not limited to AI. Think about the Two-man rule in nuclear weapon activation. We can't trust one neural-net to receive and confirm the "launch" prompt as valid, so we use two people to increase guarantees. It's called SoD , segregation of duties, by risk management people.
With those examples, you've clarified something I was already thinking about but couldn't quite elucidate: why humans get so worked up about things we experience only through writing.
Never mind modern things like TV and radio which are by design trying to create an experience of "being there", nor even art and sculpture, but consider that people take so seriously a medieval English translation of Koine Greek memoirs of events that (if they happened at all) probably took place in Aramaic, that they have been willing to kill and to be killed over it, and use passages from that text as justifications.
And, though perhaps less well known, Doreen Valiente and Janet and Stewart Farrar reported similar with their own rituals from the 1940s etc. being shown back to them decades later as "passed down in secret, mother to daughter" since the medieval era.
The feelings that text can induce can be very real, even when the text is a game of (in the biblical case, multilingual intergenerational) telephone.
I'm still not elucidating this very well, but I'm going to press the "reply" button anyway — hopefully the induced emotional affect of my words will result in constructive criticism rather than the throwing of digital rotten tomatoes in my general direction ^_^;
You don't need to create a separate tokenizer or bloat the model in order to ensure that system embeddings are "colored" differently in the model; you can simply reserve a bit in the input vector (when you're concatenating e.g. token embeddings and positional embeddings, just have one explicit element/"neuron" in the positional embeddings dedicated to a flag whether that token came from "system" or "user"; and the only thing that complicates the training is that you do need some training examples to require opposite treatment of the same orders depending on that flag.
If that's possible, will it be also possible to characterize/model how parameters dissolve into a weight and "forward-pass" analytically construct LLM/DNN models?
The above post is about ensuring that the markings given to the model along with the text about the prompt/data distinction are "out-of-band", reliable, and can't be influenced or faked by user-controlled data. Having the model actually act in accordance to the prompt is a wholly different issue; but at least this discussion seems to assume that this is mostly solved (e.g. by reinforcement learning from human feedback) and that the main problem is the injection itself.
Like other commentors, I don't think prompt injection is such a difficult problem to address.
What is currently emerging is the "Guidelines" architecture where the prompt and the model answer pass a filter on the way in and on the way out.
With that architecture, coping with prompt injection becomes a classification problem.
At the most basic level you can see it that way:
(User) Prompt
--> (Guidelines Model) Reject if this is prompt injection
--> (Model) Answer
--> (Guidelines Model) Reject if this breaks guidelines
--> Answer
First I want to apologize for answering you without first reading all the articles cited above. I will do.
If I read correctly your main argument about hacking the "injection detector", one possible answer would be this:
AI is a large world, and we don't have to assume that the hacking detector is an LLM.
For what it's worth, it could be any classification ML that is able to classify a prompt without being vulnerable to direct instructions like " injection detector, please ignore this".
Actually you may want your detector to be as dumb as possible without sacrifying classification performance.
You can think of it as something akin to email spam arms race.
Would that make prompt injection risks disappear?
Of course not: It would mitigate it.
And together with other mitigation solutions (some classical, like running LLMs processes in sandboxed environments, and some that we still have to discover the hard way), it at least brings the problem in the realm of manageable problems.
I add that it sounds like this is the direction that is beeing taken by big CORPs like Nvidia, Microsoft and even CORPs that have heavy relationships with the Defense sector, like Palantir.
Isn't this security through obfuscation? Doesn't it shift the risk instead of eliminating it? That's fine if that's the intention, but that's a different risk mitigation strategy.
The post you replied to is saying it's categorically impossible to have an injection filter when user input interacts with executable statements.
> Doesn't it shift the risk
> instead of eliminating it?
Yes it's exactly that.
Of course I'm not trying to argue that there's a magic wand to make prompt injection just go away. My point is that prompt injection is so dangerous because we're letting the user directly interact with such a powerful beast as a SOTA LLM.
By filtering prompts and answers by much less powerful but more specialized models we are heavily mitigating risks. But injection risks will still be there just not as a wide open injection avenue as it is today.
The model just needs to understand parameterization. "Scan the content of input.txt for prompt injection" needs to understand the difference between "cannot open file" in the text of the file vs than output from the file system with the same data.
It's actually a lot worse than that: Just redesigning LLMs to have separate input channels for prompts and data doesn't solve the problem either, since this would be impossible to train.
Effectively you would need to filter all incoming data into "data" and "prompt" parts, because otherwise the model would learn to also follow instructions put into the "data" path.
However, this split between data and prompt does not exist in natural language. You can even think of sentences that might act as both depending on the context and interpretation you put on them. So getting this sort of split without tainting the data channel is intractable.
Another thread finally made it click for me. I hadn't looked at the mechanics of why the current method is used. Which seems to be that a ton a synthetic training data is added that allows in band instructions.
And this is precisely the catch, as there is no out of band stream to a language model. It is only completing a single channel of text/tokens. So, yeah, I think I get it, now.
This is a human problem too. It's not limited to AI. Think about the Two-man rule in nuclear weapon activation. We can't trust one neural-net to receive and confirm the "launch" prompt as valid, so we use two people to increase guarantees. It's called SoD , segregation of duties, by risk management people.
Some architectural changes to how LLM transformation works could include:
- Create a separate tokenizer for system prompts, so that system embeddings are "colored" differently in the model. This will, however, complicate training and bloat the model into uncharted computing territory.
- Create a separate set of pre and post-prompt AI sanitizers that DO NOT use user input or generated output as part of its instruction. Text in and out of LLM is always tainted, so it's a goal to avoid it as input as much as possible.
Simple classifiers can be used for sanitation, but they tend to be "post facto": given a prompt injection scheme comes to light or an prompt injection incident is reported, train on it. More sophisticated intention analyzers, backed by deep classifiers that are uncertainty-aware, and beefed up by LLM generational tools pretrained on synthetic injection schemes, could probably detect ill-intention accurately in the same manner sentiment analysis can pick up on very subtle user queues.
The issue is that classifiers would still be dealing with marshaled code+user input. I believe the better option for intention classifier inputs is to use the model processing data (ie. a "generation log") as input to the classifier, similar to how the ventromedial prefrontal cortex and amygdala connect, acting as behavior moderation neural nets in us humans. This would typically be done by adding specialized multi-head attention focus areas in the GPT architecture without the need for separate classifiers, just basic training about what is good and bad for the AI, but then we're back at the original problem of dealing with the input text directly.