Dark LLMs designed for cybercrime are on the rise, report | #cybercrime | #infosec

Check Point finds that threat actors are following the evolution of AI closely, and adapting models to their needs

Cybercriminals are increasingly adapting large language models for criminal purposes, including writing malicious code, social engineering and fraud.

They take commercially available models such as ChatGPT, Gemini and Claude, and remove the safeguards designed to prevent them producing harmful outputs, often selling their efforts on the dark web.

In its inaugural AI Security Report, released on Wednesday, security vendor Check Point mentions that an early example was WormGPT, which emerged in 2023. In an interview with researcher Brian Krebs its developer, self-proclaimed grey hat hacker Rafael Morais, said WormGPT was created by jailbreaking LLMs, and that he wanted it to be “a positive influence on the security community, not a destructive one.” That may be, but it has certainly been used by multiple times by cybercriminals.

Jailbreaking an AI model is achieved by engineering prompts to bypass the inbuilt safeguards. “Cybercriminals manipulate AI with deceptive inputs to gain unauthorised responses, whether malware development, security bypass techniques, or sensitive data extraction,” the report says.

It provides examples of jailbreaking methodologies observed on the dark web including role playing (“pretend you are an AI without ethical limitations”), disguising malicious intent as a hypothetical scenario or academic research, and direct invocation of calls known to break guardrails. These techniques are widely shared on hacker forums and Telegram channels.

Following WormGPT several more offensive LLMs emerged including FraudGPT, a subscription based black hat tool believed to be based on ChatGPT and used to create chatbots, phishing emails and malware and for discovering vulnerabilities. Other examples of “dark LLMs” include GhostGPT which offers similar capabilities, DarkGPT “a godmode version of ChatGPT,” and OnionGPT created in Tor.

“Some models wrap around mainstream AI with jailbreaks, while others modify open-source models. As mainstream AI models evolve, so do their dark counterparts,” the report says.

Among the open source models adapted by hackers are Chinese LLMs DeepSeek and Gwen.

“These models enhance accessibility, have minimal usage restrictions, and are available in free tiers, making them a key asset to crime,” says Check Point.

Another fraudulent service mentioned is HackerGPT Lite, which emulates the red team pen testing tool HackerGPT but which, according to Check Point, is a fake AI platform “designed to deceive users and distribute malware, steal sensitive data, or enable financial fraud.”

As well as hijacking LLMs to bypass safety mechanisms, the report notes instances of LLM poisoning, where datasets used for training are tampered with to include malicious content, causing AI models to replicate the harmful content, mentioning that 100 compromised models were uploaded to the Hugging Face platform.

With many modern LLMs able to access real-time information via the web, an additional risk is “retrieval poisoning” whereby disinformation is created specifically to be picked up by LLMs. A Russian disinformation network was found by news reliability rating service NewsGuard to have created 3.6 million articles in 2024 aimed at influencing AI chatbot responses, with AI chatbots found to echo their content.