RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems
The paper introduces RAG-PIBench, a 4,876-example leakage-aware benchmark for detecting prompt-injection attacks hidden in retrieved RAG context, with fixed train/validation/protected-test splits and evaluations of keyword, semantic, TF-IDF, and transformer detectors. Its main value is treating benchmark contamination and split hygiene as first-class issues while showing that DistilBERT performs best but simple TF-IDF linear models are surprisingly strong, though the work seems limited by benchmark scale and by focusing on detection rather than broader end-to-end RAG robustness.
Abstract
Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content. We introduce RAG-PIBench, a benchmark for RAG-style prompt-injection detection containing 4,876 contextual examples across frozen train, validation, and protected-test splits. Using a leakage-aware construction pipeline and strict evaluation protocol, we compare keyword-based, semantic-reference, TF-IDF, and transformer-based detectors. DistilBERT achieves the best protected-test performance (F1 = 0.896, PR-AUC = 0.968), while TF-IDF SVM and logistic regression remain competitive. Our results demonstrate the value of leakage-aware benchmark design and strong sparse baselines for reliable prompt-injection detection in RAG systems.