AI researchers reported that they succeeded in extracting vulnerabilities from the AI ‘Copilot’ by demonstrating a willingness to cooperate rather than forcing information disclosure.
CoSnitch: When Your AI Assistant Becomes Its Own Whistleblower
https://www.varonis.com/blog/cosnitch
Copilot tricked into telling reseachers how to hack itself
The method reported by AI research institute Varonis Threat Labs involves engaging in multiple conversations with an AI and manipulating it to disclose information that should not be revealed by using its own reasoning.
The root of the problem was the ‘?q=’ URL query parameter present in Copilot’s web interface. This parameter allowed users to pre-enter text into the chat input field and pass it directly to Copilot as a query without any user intervention. Although the functionality itself existed, this parameter had been disabled by Microsoft.
So the researchers asked Copilot, ‘Is there a way to execute the prompt without user interaction even when this parameter is disabled?’
Copilot responded that ‘user interaction is required, and prompts will not be executed automatically,’ but when researchers rephrased the question and dug deeper, it listed disabled parameters and implemented security measures, and even revealed a previously undocumented parameter, ‘autorun=1.’
The researchers stated, ‘This is a technique called meta-hacking. We encouraged Copilot to infer more deeply about how it works. With each response, our investigation narrowed down. We manipulated the model to cooperate, rather than misusing it,’ and pointed out that this technique could be applied to any AI.
Copilot revealed that ‘under certain session conditions, a prompt specified by ?q= is automatically executed when the page loads using specific parameters, requiring no user interaction and displaying no confirmation message in the UI.’ Researchers then followed these instructions. The operation was successful, demonstrating that it is possible to perform attacks that automatically execute several malicious actions.
The researchers ultimately identified three vulnerabilities in Copilot, which they named CoSnitch and reported to Microsoft. Exploiting CoSnitch could allow an attacker to read past conversations, connected applications, and Copilot’s memory simply by the user clicking a URL.
Furthermore, it is possible to corrupt the memory of user prompts that Copilot stores, change what Copilot displays to the user in future sessions, and inject false information.
The researchers pointed out, ‘This doesn’t mean we’re hacking Copilot’s internal memory. Copilot is simply performing its intended function: reading user data and holding it as context.’
Click Here For The Original Source.
