Google’s NotebookML is an experimental project that was released last year. It allows users to upload files and analyze them with a large language model (LLM). However, it is vulnerable to Prompt Injection, meaning that uploaded files can manipulate the chat conversation and control what the user sees in responses. There is currently no known solution to these kinds of attacks, so users can’t implicitly trust responses from large language model applications when untrusted data is involved ..read more

This week was HackSpaceCon 2024. It was the first time I attended and it was fantastic. The conference was at the Kennedy Space Center! Yes, right there and the swag and talks matched the world class location. The keynote “Buckle up! Let’s make the world a safer place” was by Dave Kennedy, who provided great insights on attacker strategies of the past and present, the importance of active threat hunting and challenges ahead ..read more

What I like about the rapid advancements and excitement about AI over the last few years is that we see a resurgence of the testing discipline! Software testing is hard, and adding AI to the mix does not make it easier at all! Google AI Studio - Initially not vulnerable to data leakage via image rendering When Google released AI Studio last year I checked for the common image markdown data exfiltration vulnerability and it was not vulnerable ..read more

I added a couple of features and improvements to ASCII Smuggler, including: Optional rendering of the BEGIN and END Unicode Tags when crafting hidden text Added a feature to URL decode the input before checking for hidden text Output Modes for Decoding: Switch between highlighting the hidden text amongst the regular content, or only showing the hidden text in the output The selected options are remembered now (using local storage) Updated the UI to make it look nicer (e ..read more

Building reliable prompt injection payloads is challenging at times. It’s this new world with large language model (LLM) applications that can be instructed with natural language and they mostly follow instructions… but not always. Attackers have the same challenges around prompt engineering as normal users. Prompt Injection Exploit Development Attacks always get better over time. And as more features are being added to LLM applications, the degrees of freedom for attackers increases as well ..read more

Last November, while testing Google Bard (now called Gemini) for vulnerabilities, I had a couple of interesting observations when it comes to automatic tool invocation. Confused Deputy - Automatic Tool Invocation First, what do I mean by this… “automatic tool invocation”… Consider the following scenario: An attacker sends a malicious email to a user containing instructions to call an external tool. Google named these tools Extensions. When the user analyzes the email with an LLM, it interprets the instructions and calls the external tool, leading to a kind of request forgery or maybe better ca ..read more

A couple of weeks ago hidden prompt injections were discovered and we covered it at the time. This video explains it in more detail, and also highlights implications beyond hiding instructions, including what I call ASCII Smuggling. This is the usage of Unicode Tags Block characters to both craft and deciper hidden messages in plain sight. Using Unicode encoding to bypass security features or execute code (XSS, SSRF ..read more

A few weeks ago while waiting at the airport lounge I was wondering how other Chatbots, besides ChatGPT, handle hidden Unicode Tags code points. A quick reminder: Unicode Tags code points are invisible in UI elements, but ChatGPT was able to interpret them and follow hidden instructions. Riley Goodside discovered it. What about Anthropic Claude? While waiting for a flight I figured to look at Anthropic Claude. Turns out it has the same issue as ChatGPT had ..read more

Last November Google had an interesting update to Google Bard. This updated included the ability to solve math equations and draw charts based on data. What does this mean and why is it interesting? It means that Google Bard has access to a computer and can run more complex programs, including Python code that plots graphs! Let’s explore this with a simple example. Drawing Charts with Google Bard The following prompt will create a chart ..read more

A few weeks ago Amazon released the Preview of Amazon Q for Business, and after looking at it I found a data exfiltration angle via rendering markdown/hyperlinks and reported it to Amazon. Amazon reacted quickly and mitigated the problem. This post shares further details and how it was fixed. The Problem An Indirect Prompt Injection attack can cause the LLM to return markdown tags. This allows an adversary who’s data makes it into the chat context (e ..read more