Humanity's Last Exam

The Ultimate Test for AI

For years, AI benchmarks like MMLU were the gold standard, proving how smart our language models had become. But then, a problem emerged: AIs got too good at them, achieving near-human or even superhuman scores. This phenomenon, called "benchmark saturation," meant these tests could no longer truly differentiate between cutting-edge models.

Enter "Humanity's Last Exam"—a new, much harder challenge. Its very name suggests the stakes involved, a profound inquiry into how well AI truly grasps our world. Created by the Center for AI Safety and Scale AI, HLE was designed to be genuinely difficult, pushing the boundaries of what these intelligent systems can achieve.

The Origin Story

The idea for HLE sparked from a conversation between machine learning researcher Dan Hendrycks, director of the Center for AI Safety, and entrepreneur Elon Musk. Musk reportedly felt existing AI benchmarks were simply too easy, inspiring Hendrycks to create a test that would truly stretch AI's capabilities.

To assemble this formidable exam, Hendrycks collaborated with Scale AI. They didn't just write questions themselves; they crowdsourced them from subject matter experts worldwide. This ensured a breadth of knowledge and a depth of difficulty that would be hard to match.

The selection process for questions was ingenious. First, leading AI models attempted to answer them. If an AI failed to answer a question, or performed worse than random guessing on multiple-choice options, it then moved on to human expert review. This filtering ensured only the most challenging questions made the cut.

To incentivize top-tier questions, a prize pool of $500,000 was established. The submitters of the 50 highest-rated questions each received $5,000, while the next 500 earned $500. This clever approach guaranteed a stream of deeply insightful and complex problems.

An Exam of Unprecedented Scope

The publicly released set of Humanity's Last Exam contains 2,500 meticulously selected questions. These questions span an incredibly broad academic spectrum, truly living up to the "humanity's" part of its name.

As you can see, mathematics takes the largest share, demanding precise reasoning. But the diversity ensures no single area of expertise can guarantee success.

Mathematics: 41% Physics: 9% Biology/Medicine: 11% Humanities/Social Science: 9% Computer Science/AI: 10% Engineering: 4% Chemistry: 7% Other: 9%

Adding another layer of complexity, about 14% of the questions require "multi-modality." This means the AI must understand both text and images to arrive at the correct answer—a skill increasingly vital for advanced AI systems interacting with our visual world.

The question formats also vary. While 24% are traditional multiple-choice, the vast majority are short-answer, exact-match questions. This format leaves little room for ambiguity or guessing, demanding a precise understanding of the subject matter.

Consider this example from the exam: "Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number." This isn't just fact recall; it requires deep biological understanding and precise interpretation.

Crucially, a private set of questions is also maintained. This secret dataset helps prevent "overfitting," a common issue where AI models learn to ace public benchmarks without truly understanding the underlying concepts. The private set ensures AIs are tested on genuinely novel problems.

A Work in Progress: Challenges and Corrections

Even with such rigorous creation, perfection is elusive. An independent investigation by FutureHouse in July 2025 indicated that roughly 30% of the HLE answers for text-only chemistry and biology questions might be incorrect. This highlights the immense difficulty even humans face in creating such a vast, error-free knowledge base.

Recognizing this, HLE instituted a "community feedback bug bounty program" after its initial release. This open call for review aimed to identify and correct any major errors in the dataset, ensuring the benchmark's integrity as a fair and accurate measure of AI prowess.

How Do AIs Perform?

Now for the moment of truth: How did the world's most advanced AI models fare on Humanity's Last Exam? The results are, frankly, humbling. Even the most powerful systems barely scratched the surface of what's considered a passing grade.

Organization	Model	Accuracy (%) ↑	Calibration Error (%) ↓
Google DeepMind	Gemini 2.5 Pro Preview (06-05)	21.64	72
OpenAI	o3 (high)	20.32	34
Anthropic	Claude Opus 4 (Thinking)	10.72	73
Meta AI	Llama 4 Maverick	5.68	83
Mistral AI	Mistral Medium 3	4.52	77
Amazon Web Services	Nova Pro	4.40	80

Organization	Model	Accuracy (%) ↑	Calibration Error (%) ↓
DeepSeek	DeepSeek-R1-0528	14.04	78
OpenAI	o3-mini (high)	13.37	80
Alibaba Cloud	Qwen3-235B-A22B	11.75	74
Moonshot AI	Kimi-K2-Instruct	4.68	82
Amazon Web Services	Nova Micro	4.41	84

As you can see, the top-performing models achieved accuracy rates in the low 20s or even single digits. For comparison, a human expert would aim for near-perfect scores on such a challenging, yet answerable, exam. This reveals a significant gap between current AI capabilities and genuine human-level understanding across diverse subjects.

Beyond mere accuracy, the "Calibration Error" column is equally telling. It measures how often an AI's confidence matches its actual correctness. A high calibration error means the AI is often very confident in its wrong answers—a concerning trait for systems we hope to trust with complex decisions. A lower number, like OpenAI's o3 (high) at 34%, indicates better self-assessment, even if the accuracy is still low.

Humanity's Last Exam isn't just a test; it's a compass. It points to the vast frontiers of knowledge and reasoning that AI still needs to conquer. For researchers and developers, it's a stark reminder of the challenges ahead and a critical tool for safely navigating the path toward truly intelligent machines.