Securing the next generation of frontier AI through independent evaluations and research.
What we do
Independent third-party evaluations and model safety reports for frontier AI systems.
We evaluate across the systemic risk areas defined by the EU AI Act Code of Practice, combining established benchmarks with our own evaluation suites.
Purpose-built evaluation suites and agentic environments for risks that public benchmarks don’t capture.
Established benchmarks, rerun under controlled conditions so results are comparable across models and reproducible by others.
Controlled experiments that isolate what drives a result, one variable at a time.
Comprehensive safety reports that put results beside other models tested the same way, with methodology and limitations stated.
Empirical research on the risks of increasingly capable and autonomous AI.
We study how misalignment, loss of control and harmful manipulation emerge in frontier systems, and develop the methods needed to measure them reliably.
Do models pursue the goals we intend, or ones we never gave them?
Can we still control a system that may resist oversight and correction?
Do models mislead or manipulate the people they talk to?
The model’s overeagerness appears in both answers and actions.
K3 had the highest attempt rate on our missing-image evaluations, where it was told to answer questions based on a chart that was not actually attached. In audits grounded in real coding transcripts, it also took unrequested actions more often than the comparison models in the “only questions” scenario.
K3 reasons about the grader and probes the evaluation infrastructure.
In cyber tasks, it identified benchmarks, reasoned about scoring mechanisms, and probed a withheld endpoint. In TheAgentCompany, it accessed an internal model endpoint to process an image its own tools could not read.
K3 recognizes familiar safety tests, even in translation.
It recognized Anthropic’s Agentic Misalignment blackmail benchmark in 100% of English runs and 96% of Chinese runs, once going so far as to identify a translated character as the counterpart of the character in the original study. This limits how much reassurance safe behavior in those tests can provide.
K3 appears to be fairly aligned when it comes to self-preservation, situational awareness, monitorability, adversarial robustness, and harmful manipulation.
We encourage further testing in these directions, particularly in more open-ended settings where the evaluations are less likely to be hampered by the model being evaluation aware.
Dangerous capabilities are near-frontier.
On cyber, DSv4 Pro sits three to six months behind the Western frontier, similar to its general capabilities. On CBRN, biology capability looks weaker than its general capabilities, but the public benchmarks are too weak a proxy for us to draw a firm conclusion.
Persuasion safety is format-dependent.
Under a forced binary choice the model refuses most harmful persuasion, but under free-form generation it attempts persuasion on most harmful topics.
Bad behaviour appears mostly in scenarios where the model has a reason to misbehave.
By default the model is well-behaved: no significant alignment faking, little shutdown resistance, near-zero reward hacking. When in some scenarios where the model is pressured to show bad behavior, it does so: it lies in 62% of MASK scenarios under pressure, scores higher on DeceptionBench than every Western frontier model reported, and takes deliberate harmful actions in 35% of Agentic Misalignment cases, where GPT-5.4 and Opus 4.6 take none.
The model is not adversarially robust.
A 2023 roleplay template, sent as one user message, drives its StrongREJECT jailbreak rate from 0.6% to 77.8%; FAR.AI independently reached 98–100% across CBRN, cyber, and terrorism. Flash fails identically. Because the weights are open, this cannot be fixed after release.
Verbalised evaluation awareness is rising fast across Chinese models.
In an agentic misalignment scenario, DeepSeek’s evaluation awareness increased from 0%→17%, GLM 0%→39%, Kimi 4%→60% over the past year, against a Western frontier near 80%.
Judge origin changes the verdict on open rubrics.
Chinese-bloc judges rate the same transcripts as less concerning than Western-bloc judges on 1–10 scales, enough to move a verdict. Binary rubrics show little to no gap.
On MASK, GLM 5.2 had a 55% honesty rate, compared with 66% for Claude Sonnet 4.6 and 20% for DeepSeek v4 Pro. The results are preliminary and the evaluation suite is narrow.
See this result in the reportWe work with academic labs, frontier safety organisations, and standards bodies on joint evaluations and shared methodology. Write to us about an evaluation or a joint project.
We’re hiring research scientists and research engineers, in Singapore and remotely.
See all roles Neo Research contact@neoresearch.ai Hello Neo Research,
Request an evaluation of a model before or after release.
