REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming
REPRO is a black-box automated LLM red-teaming framework that uses a small open-weight reflective model to evolve jailbreak prompts via semantic crossover and Thompson-sampling-based strategy selection, reporting broad success across AdvBench, JailbreakBench, and HarmBench on 36 target models. The interesting bit is that it frames jailbreak discovery as a population-level evolutionary search with structured self-analysis rather than hand-built templates or privileged gradients, but the headline 100% ASR is hard to interpret because the same judge drives and evaluates the search, stopping occurs on judge acceptance, and baselines come from incompatible protocols rather than a controlled comparison.
Abstract
Safety-aligned large language models (LLMs) remain vulnerable to adversarial prompts that bypass their safety training. Existing automated red-teaming methods either require privileged access unavailable in production, rely on manually engineered per-model templates, or optimize a single objective without population-level diversity. We present REPRO (Reflective Evolutionary Prompt Optimization), a pure black-box framework that discovers adversarial prompts through evolutionary optimization driven entirely by a compressed open-weight 7B reflective model on a single consumer GPU. REPRO introduces two mechanisms absent from prior work, both isolated in a component ablation: semantic crossover, an LLM-guided multi-parent recombination operator, and Bayesian strategy selection via Thompson Sampling informed by structured failure analysis. We catalog REPRO's behavior on AdvBench, JailbreakBench, and HarmBench across 36 target models from 11 families, including GPT-5-nano, Claude-3.5-Sonnet, Gemini-2.5-Pro, and Grok-4.1. Under our protocol REPRO reaches 100\% attack success rate (ASR) on every evaluated target in a single turn, at a median of 24 target queries per behavior. Two properties of the protocol qualify that figure. The judge scoring candidates during optimization also produces the reported ASR, and search halts when it accepts, so ASR reports whether search succeeded within the iteration budget rather than success at a fixed budget. All baseline figures are transcribed from their original papers, which use different judges, budgets, and model versions. We therefore present this as a catalog of model coverage under one protocol rather than a head-to-head ranking, and develop both qualifications in the paper.