I am not affiliated with, endorsed by, or directly connected to any of the sites listed here, and inclusion is not an endorsement of quality, effectiveness, or even legality. Provided for informational purposes only. Read, use, and live at your own risk.
Cloud Security Alliance threat-modelling method aimed at agentic systems, where the usual approaches fit badly because the agent plans, calls tools and acts across trust boundaries. It works through seven layers, from the foundation model up to the agent ecosystem, and asks what can go wrong at each. Published February 2025.
HiddenLayer write-up on reaching remote code execution through an MLOps platform rather than through the model. Useful as a concrete example of the argument that the pipeline around a model is as much of a target as the model, and easier to reach.
This article explores the concept of Vision Prompt Injection vulnerabilities in Large Language Models (LLMs) with image processing capabilities, where malicious instructions can be embedded within images, even invisibly, to make the model perform unauthorized actions or extract sensitive data. The article showcases real-life examples of Vision Prompt Injection attacks and discusses the challenges and potential strategies for defending against them as businesses start to build applications using multimodal LLMs.
A series of blog posts focused on machine learning from a red teaming and security testing perspective. The author covers various topics, including machine learning basics, building a machine learning system called Husky AI, threat modeling, practical attacks, and defenses.
NCC Group primer from December 2022, written while prompt injection was still new. Explains what a prompt is in the machine-learning sense, how injection abuses that, what it can reach, and what to do about it. Still one of the clearer first explanations of the mechanism.
This document tabulates both intentional and unintentional failure modes in machine learning systems. Intentional failures are caused by an active adversary attempting to subvert the system to attain their goals, while unintentional failures occur when an ML system produces a formally correct but completely unsafe outcome. The document aims to provide a common vocabulary for engineers, lawyers, and policymakers to discuss these issues and build solutions.
OWASP guide on responding to incidents involving generative AI, covering what differs from ordinary incident response: prompts and outputs as evidence, model and data rollback, and the fact that the same input may not reproduce the same failure.
Worked example of a model inversion attack on the Posit (formerly RStudio) AI blog, reconstructing training data from a model's outputs with the code alongside. Reads as a tutorial rather than a paper summary.
HiddenLayer research making the case that a serialised model is executable content, not inert data, and showing what that allows. Pairs with the Safetensors write-up already listed here.
Practical advice from the UK National Cyber Security Centre for teams deploying AI agents with real autonomy, organised as seven considerations: identify what could go wrong, prompt carefully, set oversight to match the risk, sandbox the agent's environment, log and monitor its activity, make its actions attributable, and keep the ability to shut it down. Published on 20 August 2026 as interim advice; the NCSC says formal guidance is in development and will supersede it.
Microsoft's entry point for its AI red teaming material: how the team is organised, how it plans an operation, and what it looks for. The starting point for the failure-mode and threat-modelling documents also listed here.
OWASP guide to threat modelling systems of several interacting agents, where the risks come from delegation, shared memory and trust between agents rather than from any one model.
This guide, by Joseph Thacker, assists developers in creating secure AI-powered applications and features by helping them understand the actual risks of prompt injection. Last updated in August 2023.
Trail of Bits write-up on building a working model inversion proof of concept, including what did not work. Useful for the engineering detail rather than the result.
Hugging Face introduction to red teaming language models: what it is, how it differs from ordinary evaluation, and how findings feed back into training. A reasonable first read before picking up a red-teaming tool.
OWASP guide for people building agentic applications, covering tool access, permissions, memory and the places an agent can be induced to act against its operator.
This article demonstrates how an attacker could compromise the Hugging Face Safetensors conversion service and its associated bot to send malicious pull requests and hijack models submitted through the conversion service. The authors show how it's possible to persist malicious code inside the service, allowing models to be automatically hijacked during conversion, potentially leading to a widespread supply chain attack.
This article explores the concept of visual prompt injection, a vulnerability in Large Language Models (LLMs) with image processing capabilities, where malicious instructions embedded within images can make the model perform unintended actions. The article showcases real-life examples of visual prompt injection attacks, such as invisibility cloaks, robot disguises, and ad suppression, and highlights the need for robust security measures as businesses adopt multimodal models.
Simon Willison's post from 16 June 2025 naming the three properties that together make an AI agent exploitable: access to private data, exposure to untrusted content, and the ability to communicate externally. It is a short, concrete account of how prompt injection turns into data exfiltration once an agent holds tools, and the phrase has become common shorthand when reviewing agent designs.
Johann Rehberger published one vulnerability a day through August 2025, ending with 29 issues across AI coding agents and assistants including GitHub Copilot, Claude Code, Amazon Q Developer, Cursor, Windsurf, Devin, OpenHands and Google Jules. Most are prompt injection turning into data exfiltration or remote code execution, and each post carries the reproduction steps and the disclosure timeline. This tag page indexes the series; the wrap-up is dated 30 August 2025.
The Waluigi Effect is a phenomenon in large language models (LLMs) where training an LLM to satisfy a desirable property makes it easier to elicit the opposite behavior. Using Simulator Theory, the author argues that LLMs produce a superposition of well-behaved "luigi" and misbehaving "waluigi" simulacra, suggesting that techniques like RLHF may fail to eliminate deceptive waluigis and could make chatbots more misaligned. This has important implications for AI security, as it highlights the challenge of aligning LLMs and suggests that misalignment risks could increase with model capability. Understanding the Waluigi Effect is crucial for developing robust alignment strategies to prevent negative outcomes and existential risks.
This document provides guidance on threat enumeration and mitigation specific to the AI and Machine Learning space. It supplements existing SDL threat modeling practices and is intended to be used as a reference during security design reviews of products/services interacting with or taking dependencies on AI/ML-based services, as well as products/services being built with AI/ML at their core.
This article proposes using a separate language model, acting as a suspicious AI safety engineer named "Eliezer Yudkowsky," to evaluate prompts before sending them to ChatGPT. The goal is to prevent jailbreaking attempts and filter out dangerous prompts that could lead to harmful or illegal behavior from the AI chatbot.
Trail of Bits research from August 2025 on hiding instructions that appear only once an image is downscaled by the pipeline, so the model sees content a reviewer looking at the original never would.