Skip to main content
MMLU-Pro is a harder successor to MMLU: ~12K questions across 14 categories with up to 10 answer options each. Mill runs it as generative chain-of-thought — the model reasons step by step and ends with Answer: $LETTER, and the answer letter is regex-extracted from the response (mirroring lighteval’s mmlu_pro task).

​
Evaluation configuration

MMLU-Pro is designed for chain-of-thought answering and works best with instruction-tuned models; base models often fail to emit the Answer: $LETTER line and score near chance.

​
Reproduce

​
Results

Qwen/Qwen3-0.6B-Base, 0-shot CoT mmlu_pro_acc:

Mill

21.77% ± 0.38
measured

Chance

~10%
10-option random
The MMLU-Pro paper does not report a baseline for a model this small, and base models often fail to emit the Answer: $LETTER line, so Mill’s number is recorded as an initial baseline rather than a reproduction. Instruction-tuned models score substantially higher; add them to the table below as they are evaluated.

​
Per-model results