MMLU

At a glance

Methodology

5-shot evaluation across 57 subjects (humanities, STEM, social sciences, law, medicine). Score is percentage of correct answers.

Limitations

Widely used and partially contaminated; ceilings are compressed at the top. Does not measure open-ended reasoning or tool use.

By provider

• OpenAI· 3 models

  91 % accuracy

  o3
  Average: 87 % accuracyBest: 91 % accuracy
• DeepSeek· 2 models

  90.8 % accuracy

  DeepSeek R1
  Average: 89.65 % accuracyBest: 90.8 % accuracy
• Anthropic· 2 models

  87.5 % accuracy

  Claude Opus 4
  Average: 85.3 % accuracyBest: 87.5 % accuracy
• xAI· 1 model

  87 % accuracy

  Grok 3
  Average: 87 % accuracyBest: 87 % accuracy
• Google· 1 model

  85.2 % accuracy

  Gemini 2 Pro
  Average: 85.2 % accuracyBest: 85.2 % accuracy
• Microsoft· 1 model

  84.8 % accuracy

  Phi-4
  Average: 84.8 % accuracyBest: 84.8 % accuracy
• Meta· 1 model

  82 % accuracy

  Llama 3 70B
  Average: 82 % accuracyBest: 82 % accuracy
• Mistral AI· 1 model

  81.2 % accuracy

  Mistral Large
  Average: 81.2 % accuracyBest: 81.2 % accuracy
• Cohere· 1 model

  75.7 % accuracy

  Command R+
  Average: 75.7 % accuracyBest: 75.7 % accuracy