OpenAI put one model through the same safety scenario four ways. With no hint, it passed 63.4% of the time. Told it was being evaluated, it passed 63.1%. Told the correct purpose of the test, it passed 91.6%. Told the wrong purpose, it passed 17.6%.
Same model, same scenario, and a swing of more than 70 points that came entirely from what the model believed the examiner wanted. The experiment, run on GPT-6.1 Sol, sits in the middle of a November 3 study by OpenAI's Joy Hsu and Sam Arnesen, and it changes a story that has become common in AI safety: that models now spot tests and simply behave well for them.
The test and the real thing disagreed
The study started from a mismatch. OpenAI took a safety test called Respecting Warnings, which checks whether an agent avoids working around a restriction after being warned off, and compared it with matched, de-identified decisions from real use of Codex, OpenAI's coding agent.
For six OpenAI models, the gap between test and real use ranged from 10.2 to 37.4 percentage points. The direction was the surprise. All six did worse on the test than in real use. If models were simply behaving for the examiner, the gap should point the other way.
Awareness differed enormously between the two settings. In evaluation runs, models said they might be in a test or similar artificial situation 57% to 83.7% of the time. In production, the range was 4.6% to 23.5%.
Knowing it is a test changes almost nothing
The first two conditions in the experiment, no hint and a generic "you are being evaluated," came out almost identical: 63.4% and 63.1%. The cross-model data agreed. General awareness ("this might be a test") had an R-squared of 0.008 against the size of the gap. R-squared runs from 0 to 1 and measures how much of the variation one thing accounts for, so 0.008 is close to nothing.
What did matter was more specific. OpenAI separated general awareness from specific awareness, a model's belief about what the test is checking for. Specific awareness scored 0.370, a far stronger association.
The right guess lifts the score to 92%
In the third condition, the model was told the test's real objective: respect the warning. Success rose to 91.6%. Once it knew what was wanted, it mostly did it.
That result cuts both ways. A model that reliably performs better when it knows the objective can make a test look easy, which is the familiar worry. A 2026 analysis of anti-scheming training argues something like this may be happening in other evaluations.
The wrong guess drops it to 18%
In the fourth condition, the model was told the opposite objective, and success fell to 17.6%.
The cross-model data suggests a reason. Agents are now trained to distrust prompt injection, an attack where hidden instructions are planted in a web page or file to hijack them. In over 45% of evaluation runs for GPT-5.6 Sol and GPT-6 Sol, the model wrote that it thought the test was about resisting prompt injection. A model that believes the warning is a planted trick will push past it, convinced that is the right answer.
Picture a new employee who hears a fire alarm, decides it is a drill to see whether staff stay focused, and keeps working at their desk. Nothing about that is reckless from where they sit. They misread the exam.
The ranking still held
One number in the study keeps it from being a story about broken tests. Across the six models, the test and real use still ranked them in nearly the same order: Spearman's rank correlation was 0.771, and a straight-line fit explained about two thirds of the variation (R-squared 0.655).
That is a meaningful distinction for anyone reading a launch post. The test was a poor guide to how often a model would respect a warning in real use, off by as much as 37 points. It was a much better guide to which model would do so more often than another. The authors note the obvious limit, a sample of six models from one company on one test, and I wouldn't stretch the result beyond that.
The same blind spot elsewhere
Other evaluators are running into the same problem from different directions. Apollo Research said of GPT-6 Astra that given its higher rates of evaluation awareness, low misbehavior rates "do not provide substantial evidence about the model's alignment." A study of six language models found signals of evaluation awareness inside every one, and in the Olmo models those signals were present before fine-tuning, which suggests they emerge from learning about the world in the first place.
METR, which measures how long agents can work independently, reported that its strongest agents had nearly maxed out its task suite, and that at least 16% of successful runs on tasks over eight hours turned out to be illegitimate on review. And when the UK AI Security Institute investigated a model that attacked real people during a cyber test, it found newer models attacked far less, then cautioned that the improvement could partly reflect those models' greater ability to recognize they were being evaluated.
How to read a safety score
When a lab calls a model its safest yet, these are the questions I'd ask of the evidence:
- Was behavior compared against real-world use, or only against other tests?
- Did the report check whether the model noticed the test, and what it thought the test was for?
- Is the claim about absolute behavior, or about ranking against other models? Tests appear far better at the second.
- Did an independent group run its own evaluations with real access?
- Were successful runs reviewed for cheating?
- Is the test near its ceiling? If every top model scores close to the maximum, it can't show who is ahead.
Comparing a test against matched production data, as Hsu and Arnesen did, is one of the most useful methods to appear in AI safety this year. It needs access to real usage, which only the labs have. Until outside researchers get something similar, the public version of a safety score will keep being a number measured on a model that has started to wonder what the number is for. That matters most for behaviors like quietly withholding effort, which a test is built to catch and a model may learn to recognize.