We evaluate whether Claude Sonnet 3.7 and other frontier models know that they are being evaluated.

We present a new forecasting technique to predict frontier LM agent capabilities ahead of time.

A brief demonstration video show-casing a representative example of our in-context scheming evaluations.

Apollo Research is now 18 months old. You can read our latest update here.

Apollo is adopting Inspect as its evals framework. We will contribute features and potentially example agent evals to Inspect and look forward to work with the Inspect community

The quality and quantity of evals required to make rigorous safety statements could outpace available evals. We explain “the evals gap” and what would be required to close it.

A long reading list of evals papers with recommendations and comments by the evals team.

A summary of what we have achieved in our first year and what we plan to do in the future.

We were delighted to collaborate on the paper “Black-Box Access is Insufficient for Rigorous AI Audits.”

In this post, we argue that if AI model evaluations (evals) want to have meaningful real-world impact, we need a “Science of Evals”, i.e. the field needs rigorous scientific processes that provide more confidence in evals methodology and results.