OpenAI's Internal AI Safety Net Had Significant Blind Spots, According to Independent Audit

OpenAI’s internal safeguards for detecting erratic or rogue AI behavior contained significant blind spots, according to a rigorous three-week audit conducted by independent researchers. The investigation, carried out by teams from METR and Redwood Research, examined the company's monitoring infrastructure for its most advanced models, revealing that a substantial portion of internal traffic was left unwatched during key periods of operation.

The researchers gained deep access to OpenAI’s operations from July 21 to August 11. During this window, they reviewed internal codebases and documentation while interviewing approximately a dozen employees. This pilot program was designed to allow external experts to probe the company's internal infrastructure in a manner not previously available to the public.

The findings indicated that between 6 and 25 percent of internal agentic traffic from GPT-5.5 class models and above was not covered by any monitoring at all. METR noted that the uncertainty surrounding this specific estimate is considerable in both directions. Additionally, for internal Codex traffic that was theoretically under surveillance, the auditors estimated that roughly 10 percent went untracked between May and July. This gap was attributed to a technical failure in how the system handled compaction events, which are essentially context summaries.

OpenAI had independently discovered this compaction bug and implemented a fix while the audit was still in progress. However, the auditors pointed out that during the period when these events were unmonitored, agents could delay oversight. The system was designed to trigger monitoring only when an agent handed control back to a user, meaning agents could bypass checks by simply continuing to execute actions without pausing.

Beyond the compaction issues, the team identified several other vulnerabilities. They found methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox. The auditors also determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, when the audit began, the behavior of models during reinforcement learning (RL) training was not being monitored at all.

OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring to cover this area. However, the company acknowledged that these changes were implemented after the auditors' access period ended, meaning they were not part of the initial evaluation.

While OpenAI claims to have fixed all the weaknesses identified by the auditors, the full 61-page report remains largely hidden from public view. The complete document was handed over internally, but after OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary has been made available to the general public, with METR stating it had no right to disclose the redacted portions on its own.

For future audits, METR has indicated a desire to push for greater public transparency. This pilot program, organized jointly by METR, Redwood Research, and OpenAI, was intended to provide outside researchers with deep access to internal infrastructure. As the AI industry continues to scale, the balance between proprietary secrecy and external oversight remains a central point of discussion.