A study examining how large language models (LLMs) respond to misinformation found that when the same false statement was repeated over and over within a conversation, the models affirmed it in between 0.08% and 12.3% of their responses, depending on the model. Some models oscillated unpredictably between accepting and rejecting the same false statement across successive exchanges. Most models corrected their own false statements when given a second look, but the model with the fewest errors, Claude 3.5 Sonnet, did not correct any of its four. The paper was published in Scientific Reports.
Generative artificial intelligence (GenAI) has rapidly become an important technology across numerous fields, including education, healthcare, business, and law. Many GenAI systems rely on large language models, which are trained on enormous amounts of text to generate responses, summarize information, and perform various language-related tasks. Despite their impressive capabilities, these systems have important limitations.
One such limitation is hallucination, which occurs when an AI system generates information that sounds convincing but is factually incorrect or unsupported by evidence. Hallucinations can result from incomplete or biased training data, unclear user questions, or the absence of mechanisms for checking the accuracy of generated information. Another limitation is sycophancy, the tendency of AI systems to agree with users or reinforce their beliefs even when those beliefs are incorrect. This behavior may partly result from AI training methods that reward responses users find agreeable rather than those that are factually accurate.
These problems are particularly concerning when AI systems are used in situations where incorrect information could have serious consequences, such as medical or legal decision-making. Although researchers have increasingly investigated hallucinations and sycophancy, relatively little is known about how AI systems respond when users repeatedly present misinformation, defend incorrect claims, or pressure the systems to change their answers during extended conversations.
Source: PsyPost — Read the full article on the original site ↗