The research investigates the persistence and scale of AI package hallucination, a technique where LLMs recommend non-existent malicious packages.
The Langchain framework has allowed for the expansion of previous findings by testing a more comprehensive range of questions, programming languages (Python, Node.js, Go,.NET, and Ruby), and models (GPT-3.5-Turbo, GPT-4, Bard, and Cohere).
The aim is to assess if hallucinations persist, generalize across models (cross-model hallucinations), and occur repeatedly (repetitiveness).
Langchain Default Prompt 2500 questions were refined to 47,803 “how-to” prompts fed to the models, while repetitiveness was tested by asking 20 questions with confirmed hallucinations 100 times each.
Results of GPT 4 A study compared four large language models (LLMs)—GPT-4, GPT-3.5, GEMINI, and COHERE—for their susceptibility to generating hallucinations (factually incorrect outputs).
GEMINI produced the most hallucinations (64.5%), while COHERE had the least (29.1%). Interestingly, hallucinations with potential for exploitation were rare due to factors like decentralized package repositories (GO) or reserved naming conventions (.NET).
Results of Gemini Lasso Security’s study also showed that GEMINI and GPT-3.5 had the most common hallucinations, which suggests that their architectures may be similar on a deeper level. This information is essential for understanding and reducing hallucinations in LLMs.
Document
@import url('https://fonts.googleapis.com/css2?family=Poppins&display=swap');
@import url('https://fonts.googleapis.com/css2?family=Poppins&family=Roboto&display=swap');
*{
margin: 0; padding: 0;
text-decoration: none;
}
.container{
font-family: roboto, sans-serif;
width: 90%;
border: 1px solid lightgrey;
padding: 20px;
background: linear-gradient(2deg,#E0EAF1 100%,#BBD2E0 100%);
margin: 20px auto ;
border-radius: 40px 10px;
box-shadow: 5px 5px 5px #e2ebff;
}
.container:hover{
box-shadow: 10px 10px 5px #e2ebff;
}
.container .title{
color: #015689;
font-size: 22px;
font-weight: bolder;
}
.container .title{
text-shadow: 1px 1px 1px lightgrey;
}
.container .title:after {
width: 50px;
height: 2px;
content: ' ';
position: absolute;
background-color: #015689;
margin: 20px 8px;
}
.container h2{
line-height: 40px;
margin: 2px 0;
font-weight: bolder;
}
.container a{
color: #170d51;
}
.container p{
font-size: 18px;
line-height: 30px;
}
.container button{
padding: 15px;
background-color: #4469f5;
border-radius: 10px;
border: none;
background-color: #00456e ;
font-size: 16px;
font-weight: bold;
margin-top: 5px;
}
.container button:hover{
box-shadow: 1px 1px 15px #015689;
transition: all 0.2S linear;
}
.container button a{
color: white;
}
hr{
/ display: none; /
}
Run Free ThreatScan on Your Mailbox
AI-Powered Protection for Business Email Security
Trustifi’s Advanced threat protection prevents the widest spectrum of sophisticated attacks before they reach a user’s mailbox. Try Trustifi Free Threat Scan with Sophisticated AI-Powered Email Protection .
Run Free Threat Scan
Multiple large language models (LLMs) have been used to study hallucinations. This is done by finding nonsensical outputs (hallucinated packages) in each model and then comparing these hallucinations to see what they have in common.
Multiple LLM analyses reveal 215 packages, with the highest overlap between Gemini and GPT-3.5 and the least between Cohere and GPT-4.
Result of Cross-Model hallucinations This cross-model hallucination analysis offers valuable insights into the phenomenon of hallucinations in LLMs, potentially leading to a better understanding of these systems’ internal workings.
There was a phenomenon where developers were unknowingly downloading a non-existent Python package called “huggingface-cli,” which suggested a potential issue where large language models might be providing users with inaccurate information about available packages.
Screenshot of ChatGPT To investigate further, the researchers uploaded two dummy packages: “huggingface-cli” (empty) and “blabladsa123” (also empty).
They then monitored download rates over three months; the fake “huggingface-cli” package received over 30,000 downloads, significantly exceeding the control package “blabladsa123.”.
fake and empty package got more than 30k authentic downloads It suggests a possible vulnerability where developers rely on incomplete or inaccurate information sources to discover Python packages.
The adoption rate of a package was believed to be a hallucination (not an actual package), and to verify its usage, they searched GitHub repositories of major companies as the search identified references to the package in repositories of several large companies.
installing the packages found in the README For example, a repository containing Alibaba’s research included instructions on installing this package in its README file.
These findings suggest that either the package is accurate and used by these companies or there’s a widespread phenomenon of including instructions for non-existent packages in documentation.
Are you from SOC and DFIR Teams? – Analyse Malware Incidents & get live Access with ANY.RUN -> Start Now for Free
The post AI Package Hallucination – Hackers Abusing ChatGPT, Gemini to Spread Malware appeared first on GBHackers on Security | #1 Globally Trusted Cyber Security News Platform .
