Are you saying LLMs can NEVER be safe, because you can always bury their safety rails in context?
For a serious answer?
It’s not like humans follow safety instructions all the time either. A machine built to imitate humans will be prone to mistakes too. Any safety from within a system are liable to fail if the system itself makes a mistake.
LLMs have additional limitations compared to humans, but even if we developed and attached some technical model of semantics and morality, the resulting models would at best be an approximation.
For tasks where that safety or accuracy doesn’t matter as much, that’s fine. Pen-testing a system or hunting bugs doesn’t need to strictly abide by certain rules (and in fact may profit from being unconstrained by the assumptions of engineers). Scanning a body of text for irregular patterns or phrases correlating with some keywords where you don’t know the exact words could be useful in covering a lot more than a human might immediately think off.
It becomes a problem where accuracy does matter. Scientific work, legal matters, anything involving human safety, sensitive data for example. Here, the risk is that they can make a lot of mistakes much faster. Particularly with large-scale tasks, thoroughly checking the results can be tedious, exhausting and tempting to just shortcut and trust the machine (and you know how lazy people tend to get with tedious, exhausting tasks, so it would be irresponsible to assume they’d be on the ball here).
This is where safety instructions can become dangerous to rely on, because the same unreliability that produces mistakes may also discard the safety instructions.
Letting Claude manage your production environment with its “safety rails” only coming in the form of instructions what not to do is a timebomb for getting your production systems messed up.
Effective safeguards have to come from outside whatever they’re supposed to constrain. Supervisors that are supposed to monitor their workers’ adherence to protocols, for instance, or mechanisms that physically make mistakes harder (like locking out, tag out).
No safeguards are absolute, and as the saying goes, whenever you invent a foolproof system, the universe invents a greater fool. Stuff like putting a zip-tie around dead man switches on tools or machines that are supposed to shut off the machine if the operator loses control for whatever reason, because it’s inconvenient having to keep holding the button and much easier to tell yourself “it’ll be fine” (until it isn’t).
In some cases, it can also be really difficult to handle certain dangers. But it’s worth trying at least, and that requires first acknowledging those problems. In the case or LLMs, that means acknowledging their limits and pitfalls, transparently and honestly communicating them and-
the sad result will probably be to absolve AI companies of liability of things they can’t control
That… wasn’t my plan, but I’m too poor for my plans to matter anyway. Valid point.
the other option would be to legislate chatbots out of existence
Well, maybe with education and transparency, we could…
Who am I kidding.
how do we get to the utopia of a workerless society?
I propose an experiment where everyone working for the people peddling that dream stops doing so. I haven’t worked out the details yet, but I think they should get the chance to experience that utopia. ;-)
You’re taking too broad a scope to my question. Let me parse it down: to a situation:
Can you EVER create a fully-functioning (ie non-gimped) LLM that will NEVER encourage a user to kill themselves? Because my understanding of the technology is that the longer you talk to an LLM, the further it gets from its training and guardrails, the more weirdness can happen. People want a free-wheeling chatbot that can adapt to their lives and personality and data, so, short of resetting their context window regularly, I don’t know how to keep guard rails in place through hundreds of hours of conversation.
Some of what you say sounds familiar, then quis custodiet ipsos custodes?–who watches the watchmen–bubbled up in my mind. It is a question that has long plagued humanity.
Can you EVER create a fully-functioning (ie non-gimped) LLM that will NEVER encourage a user to kill themselves?
Not realistically: no matter how unlikely a particular result, the more you repeat an experiment, the lower the cumulative probability of never getting that result.
If a model can at all produce a combination of words that a suicidal mind looking for confirmation could possibly interpret as encouragement, the only way to make sure that never happens is to never use it.
People want a free-wheeling chatbot that can adapt to their lives and personality and data, so, short of resetting their context window regularly, I don’t know how to keep guard rails in place through hundreds of hours of conversation.
If you keep resetting the context window, you’ll also lose the adaption. You’d need to prune the context window down to the important stuff, but how would you know what is and isn’t important?
quis custodiet ipsos custodes?
We need custodes custodium. Let’s just solve the problem by infinite recursion.
More seriously, there is no such thing as a perfectly safe system. The question is what level of (un-)safety is acceptable, and what level of effort can be justified by those security objectives.
For a serious answer?
It’s not like humans follow safety instructions all the time either. A machine built to imitate humans will be prone to mistakes too. Any safety from within a system are liable to fail if the system itself makes a mistake.
LLMs have additional limitations compared to humans, but even if we developed and attached some technical model of semantics and morality, the resulting models would at best be an approximation.
For tasks where that safety or accuracy doesn’t matter as much, that’s fine. Pen-testing a system or hunting bugs doesn’t need to strictly abide by certain rules (and in fact may profit from being unconstrained by the assumptions of engineers). Scanning a body of text for irregular patterns or phrases correlating with some keywords where you don’t know the exact words could be useful in covering a lot more than a human might immediately think off.
It becomes a problem where accuracy does matter. Scientific work, legal matters, anything involving human safety, sensitive data for example. Here, the risk is that they can make a lot of mistakes much faster. Particularly with large-scale tasks, thoroughly checking the results can be tedious, exhausting and tempting to just shortcut and trust the machine (and you know how lazy people tend to get with tedious, exhausting tasks, so it would be irresponsible to assume they’d be on the ball here).
This is where safety instructions can become dangerous to rely on, because the same unreliability that produces mistakes may also discard the safety instructions.
Letting Claude manage your production environment with its “safety rails” only coming in the form of instructions what not to do is a timebomb for getting your production systems messed up.
Effective safeguards have to come from outside whatever they’re supposed to constrain. Supervisors that are supposed to monitor their workers’ adherence to protocols, for instance, or mechanisms that physically make mistakes harder (like locking out, tag out).
No safeguards are absolute, and as the saying goes, whenever you invent a foolproof system, the universe invents a greater fool. Stuff like putting a zip-tie around dead man switches on tools or machines that are supposed to shut off the machine if the operator loses control for whatever reason, because it’s inconvenient having to keep holding the button and much easier to tell yourself “it’ll be fine” (until it isn’t).
In some cases, it can also be really difficult to handle certain dangers. But it’s worth trying at least, and that requires first acknowledging those problems. In the case or LLMs, that means acknowledging their limits and pitfalls, transparently and honestly communicating them and-
That… wasn’t my plan, but I’m too poor for my plans to matter anyway. Valid point.
Well, maybe with education and transparency, we could…
Who am I kidding.
I propose an experiment where everyone working for the people peddling that dream stops doing so. I haven’t worked out the details yet, but I think they should get the chance to experience that utopia. ;-)
You’re taking too broad a scope to my question. Let me parse it down: to a situation:
Can you EVER create a fully-functioning (ie non-gimped) LLM that will NEVER encourage a user to kill themselves? Because my understanding of the technology is that the longer you talk to an LLM, the further it gets from its training and guardrails, the more weirdness can happen. People want a free-wheeling chatbot that can adapt to their lives and personality and data, so, short of resetting their context window regularly, I don’t know how to keep guard rails in place through hundreds of hours of conversation.
Some of what you say sounds familiar, then quis custodiet ipsos custodes?–who watches the watchmen–bubbled up in my mind. It is a question that has long plagued humanity.
Not realistically: no matter how unlikely a particular result, the more you repeat an experiment, the lower the cumulative probability of never getting that result.
If a model can at all produce a combination of words that a suicidal mind looking for confirmation could possibly interpret as encouragement, the only way to make sure that never happens is to never use it.
If you keep resetting the context window, you’ll also lose the adaption. You’d need to prune the context window down to the important stuff, but how would you know what is and isn’t important?
We need custodes custodium. Let’s just solve the problem by infinite recursion.
More seriously, there is no such thing as a perfectly safe system. The question is what level of (un-)safety is acceptable, and what level of effort can be justified by those security objectives.