AI routers leaked passwords from prompts to attackers, study finds #AI

Prompts sent to an AI service often reach more companies than users expect. That is the result of a study security researcher HD Moore published on November 19 on the blog of security firm Dragos. From July through November, his team sent about two million prompts to widely used AI services. Each one carried a unique bait secret, such as a real but empty AWS key, a password or a hidden link. Anyone who used one of them must have read the prompt.
Three providers were breached
The most serious finding involves at least three hosting providers that run AI models for other companies. Attackers were copying requests at all three, and none of them knew until Dragos got in touch. In the largest case, a couple hundred AWS keys were stolen, some of them tested within hours and then used for tens of thousands of API calls. From November 1, the attackers’ toolkit also included ARTEX, an AI-driven penetration testing framework that was recently linked to the cyberattack on the European Commission. The bait was processed alongside real customer requests, and Dragos writes that “those were almost certainly copied too.” The report does not name the providers, but one of them has already published a preliminary security notice.
Privacy flags did not stop the leaks
Many apps and AI agents do not talk to a model directly. They send requests through routers that pass each one on to the cheapest available provider, sometimes across several hops, and every hop sees the full prompt. On open marketplaces, Moore says, a seller only had to list a model at a low price and turn on request logging. One person in France then tried to log in as root over SSH with the stolen passwords, “typing each attempt by hand.”
Even Zero Data Retention, the strictest setting, which says nothing may be stored, did not hold. A Chinese AI lab fetched the hidden links from more than half of the prompts sent to its models, including roughly two-thirds of the requests marked as no data collection or ZDR. The same happened with prompts sent to U.S. providers that resell the lab’s models. In one case, a link showed up in a company chat in China 24 days after the request. Moore’s conclusion is that “a privacy flag only binds the first hop.”
Keep credentials out of your prompts
The practical lesson is to never paste passwords, API keys or private access links into a prompt, not even for a quick debugging session. That goes double for AI agents that read files and inboxes on their own. Where possible, use a model directly from its developer rather than through apps that bundle many models at low prices. A breach at a background provider also exposed Notion AI chats earlier this month. Our guide to deleting AI memory explains how to remove what ChatGPT, Gemini and Claude remember about you.I write about IT security and artificial intelligence. Cybersecurity is one of my regular topics, among others for Golem.de. At Notebookcheck, my focus is on practical AI for everyday users. I have worked hands-on with AI for years, most recently with a focus on AI agents and their use in everyday work, and I test tools and models myself rather than just reading about them.

Click Here For The Original Source.

——————————————————–

............