At my previous job their was a role where you just called insurance companies and asked them incredibly basic questions about what they planned to do for a patient with diagnosis X and plan Y. This information should be searchable in a document with a single correct answer, but insurance companies are too scummy for that to be reliable.
In 2021 we started using a robot that sounded like a human to call instead. It could handle the ~80%+ of calls that don’t use any critical thinking. At a guess, that’s maybe 5-10% of our division’s workforce that wasn’t needed anymore.
With the amount of jobs like this that are 100% bullshit, I’m sure there are plenty of other cases where businesses can save money by buying an automated bullshit generator, instead of hiring a breathing bullshit generator.
The problem is that 20% failure rate has no validation and you are 100% liable for the failures of an AI you’re using as a customer support agent, which can end up costing you a ton and killing your reputation. The unfixable problem is that an AI solution takes a ton of effort to validate, way more than just double checking a human answer.
It wont know it doesn’t know. At the current state of AI, it doesn’t seem to have almost any sense of what is right and wrong or a way to validate that - even when you tell it, it is wrong. Maybe there are systems that can but I am not aware of them.
The current state of AI chatbots, assigns a “confidence level” to every piece of output. It signals perfectly well when and where they should look for more information… but humans have been pushing them to “output something, anything”, instead of excusing itself for not knowing something, or running some additional processes in order to look for the missing information.
As of this year, Copilot has been running web searches to complement its lack of information, and Gemini is running both web searches, and iteratively self-checking its own answer in order to refine it (see “drafts”). It also seems like Gemini might be learning from humanity’s reactions to its wrong answers.
LLMs generate output one token at a time. Each token comes with a confidence level by the model, about whether it’s the only possible token to continue the sequence. A model is only 100% confident in its output, if it reproduces a training text verbatim. With any temperature above 0, they veer off the 100% confidence path, which lets them leverage the concept association they came up with during training, makes their output more useful.
For every generated text, you could get a confidence heat map, then ask the model to refine sections that don’t meet a desired level of confidence. Especially the parts where a model makes stuff up, or hallucinates, are likely token sequences with much lower confidence than the rest.
Running a model several times, focusing on the sections with lower confidence, getting additional data from other sources like the internet, or some niche expert system, could eliminate many of the nonsense sections… and I have a reasonably suspicion that Google’s Gemini does exactly that, refining each output with 4 additional iterations, instead of blindly spitting out the first one.
I guess that makes sense, but I wonder if it would be hard to get clean data out of the per-token confidence values. The LLM could be hallucinating, or it could just be generating bad grammar. It seems like it’s hard enough already to get LLMs to distinguish between “killing processes” and murder, but maybe there could be some novel training and inference techniques that come up.
An LLM has… let’s say two core components: a tokenizer, and a neural network. The neural network’s output, is an array of activation levels for a series of neurons, each neuron representing one token. A confidence of 100%, would mean a 100% activation of a single neuron/token, and 0% for all the rest. That is a highly unlikely scenario for a neural network, except when it got overfitted for a single patter during training, and is getting fed the same pattern again. What is more usual, is some value between 0% and 100% for each neuron, with a few neurons showing higher levels of activation, and the LLM… usually picks the highest, but maybe sometimes the second or further one.
The confidence can be calculated by comparing the level of the chosen token’s neuron, to all the other output neurons. A naive one could be level/sum(levels). Somewhat more advanced, could be level²/sum(levels²).
Hallucinations are theoretically possible at a high confidence, but usually happen at lower confidence levels where there are many tokens with a similar confidence.
It seems like it’s hard enough already to get LLMs to distinguish between “killing processes” and murder
It doesn’t look like anything to me… I mean, that could be either part of the guardrails, or a lack of context. A “killing process” is murder, outside the programming/sysadmin context. Current LLMs are still not great at handling different semantic contexts for the same token, and particularly bad at mixing different contexts throughout a single text.
My personal “Turing” test for an LLM, is being able to write a sentence, that could be interpreted in 3 or more ways. For a human, 2 meanings is a somewhat easy task, a double-entendre. Starting at 3 and 4, it becomes a feat. Most LLMs are still at 1, and sometimes struggling.
For example, Gemini says:
Q: Please write a sentence that could be interpreted in two or more ways.
A: I saw her duck into the library.
Q: Please write a sentence that could be interpreted in three or more ways.
A: I’m a text-based AI and can’t assist with that.
It can do paragraphs, though:
Q: Please write a paragraph that could be understood in five or more ways and explain those ways.
A: The old man by the sea watched the horizon, his weathered face a map of untold stories. The sky, a canvas of hues, mirrored the depths of his soul, or perhaps the ocean’s. A gull cried, a mournful symphony or a hopeful call, depending on the listener’s ear. Time, an elusive phantom, danced with eternity in every passing moment, or perhaps stood still, a silent observer of existence.
Interpretations:
Literal: A description of an old man on the beach, observing the sea and sky. Metaphorical: The old man represents humanity, the sea the subconscious, and the sky the spiritual realm. Philosophical: A contemplation of time, existence, and the nature of reality. Psychological: The paragraph delves into the complexities of human perception and interpretation. Poetic: A piece of evocative prose, inviting multiple readings and interpretations.
…which is pretty neat, but paragraphs have “more degrees of flexibility”, making it way harder to do in a single sentence.
At my previous job their was a role where you just called insurance companies and asked them incredibly basic questions about what they planned to do for a patient with diagnosis X and plan Y. This information should be searchable in a document with a single correct answer, but insurance companies are too scummy for that to be reliable.
In 2021 we started using a robot that sounded like a human to call instead. It could handle the ~80%+ of calls that don’t use any critical thinking. At a guess, that’s maybe 5-10% of our division’s workforce that wasn’t needed anymore.
With the amount of jobs like this that are 100% bullshit, I’m sure there are plenty of other cases where businesses can save money by buying an automated bullshit generator, instead of hiring a breathing bullshit generator.
The problem is that 20% failure rate has no validation and you are 100% liable for the failures of an AI you’re using as a customer support agent, which can end up costing you a ton and killing your reputation. The unfixable problem is that an AI solution takes a ton of effort to validate, way more than just double checking a human answer.
It’s not a 20% failure rate when the chatbot routes calls to a human agent whenever it’s more than x% unsure about what to say.
AI solutions still get the 80% “bottom of the barrel” menial tasks perfectly well.
It wont know it doesn’t know. At the current state of AI, it doesn’t seem to have almost any sense of what is right and wrong or a way to validate that - even when you tell it, it is wrong. Maybe there are systems that can but I am not aware of them.
The current state of AI chatbots, assigns a “confidence level” to every piece of output. It signals perfectly well when and where they should look for more information… but humans have been pushing them to “output something, anything”, instead of excusing itself for not knowing something, or running some additional processes in order to look for the missing information.
As of this year, Copilot has been running web searches to complement its lack of information, and Gemini is running both web searches, and iteratively self-checking its own answer in order to refine it (see “drafts”). It also seems like Gemini might be learning from humanity’s reactions to its wrong answers.
I thought confidence levels were for image recognition? How do confidence levels work for transformer LLMs?
LLMs generate output one token at a time. Each token comes with a confidence level by the model, about whether it’s the only possible token to continue the sequence. A model is only 100% confident in its output, if it reproduces a training text verbatim. With any temperature above 0, they veer off the 100% confidence path, which lets them leverage the concept association they came up with during training, makes their output more useful.
For every generated text, you could get a confidence heat map, then ask the model to refine sections that don’t meet a desired level of confidence. Especially the parts where a model makes stuff up, or hallucinates, are likely token sequences with much lower confidence than the rest.
Running a model several times, focusing on the sections with lower confidence, getting additional data from other sources like the internet, or some niche expert system, could eliminate many of the nonsense sections… and I have a reasonably suspicion that Google’s Gemini does exactly that, refining each output with 4 additional iterations, instead of blindly spitting out the first one.
I guess that makes sense, but I wonder if it would be hard to get clean data out of the per-token confidence values. The LLM could be hallucinating, or it could just be generating bad grammar. It seems like it’s hard enough already to get LLMs to distinguish between “killing processes” and murder, but maybe there could be some novel training and inference techniques that come up.
An LLM has… let’s say two core components: a tokenizer, and a neural network. The neural network’s output, is an array of activation levels for a series of neurons, each neuron representing one token. A confidence of 100%, would mean a 100% activation of a single neuron/token, and 0% for all the rest. That is a highly unlikely scenario for a neural network, except when it got overfitted for a single patter during training, and is getting fed the same pattern again. What is more usual, is some value between 0% and 100% for each neuron, with a few neurons showing higher levels of activation, and the LLM… usually picks the highest, but maybe sometimes the second or further one.
The confidence can be calculated by comparing the level of the chosen token’s neuron, to all the other output neurons. A naive one could be level/sum(levels). Somewhat more advanced, could be level²/sum(levels²).
Hallucinations are theoretically possible at a high confidence, but usually happen at lower confidence levels where there are many tokens with a similar confidence.
It doesn’t look like anything to me… I mean, that could be either part of the guardrails, or a lack of context. A “killing process” is murder, outside the programming/sysadmin context. Current LLMs are still not great at handling different semantic contexts for the same token, and particularly bad at mixing different contexts throughout a single text.
My personal “Turing” test for an LLM, is being able to write a sentence, that could be interpreted in 3 or more ways. For a human, 2 meanings is a somewhat easy task, a double-entendre. Starting at 3 and 4, it becomes a feat. Most LLMs are still at 1, and sometimes struggling.
For example, Gemini says:
It can do paragraphs, though:
…which is pretty neat, but paragraphs have “more degrees of flexibility”, making it way harder to do in a single sentence.