OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning
OwlSight is a chain-of-thought monitoring system that uses prompt-specified “factors” to check reasoning traces for task-dependent issues across capability, safety, and trustworthiness, at both fine-grained and coarse-grained levels. The neat bit is that one monitor can be steered to many oversight goals and reportedly beats general LLM monitors while generalizing across models and noisy/rewritten CoTs, though it still depends on having meaningful CoT traces to inspect and its guarantees are benchmark- rather than deployment-level.
Abstract
Chain-of-Thought (CoT) monitoring offers actionable evidence for oversight AI systems, with the potential to enable early detection and intervention. However, existing approaches are focused on specific tasks, whereas the regulation of AI, particularly reasoning models, requires monitoring a broader range of tasks. We introduce Factor for CoT monitoring, a strategy that supports customizing monitoring requirements in the prompt, which enables a single monitor to cover different tasks by relying on different factors. Furthermore, we introduce two types of factors, fine-grained and coarse-grained (e.g., monitoring whether a specific email address has been leaked versus monitoring privacy leakage), to accommodate diverse user monitoring needs. Based on this, we develop a CoT monitor named OwlSight that relies solely on factor to monitor behaviors related to multiple dimensions, including capability, safety, and trustworthiness. Experiments across 15,971 reasoning trajectories show that OwlSight surpasses general-purpose LLMs by 1.32-30.57 and 11.71-27.81 points at fine and coarse granularity, respectively, and achieves state-of-the-art results on OpenAI's CoT monitorability benchmark. It also generalizes to unseen models' CoTs and remains robust to Factor rephrasings, CoT compression, and obfuscation, offering a promising paradigm for CoT monitoring that can contribute to the oversight of frontier AI.