Skip to main content
Mill aims to reproduce the numbers reported in model and benchmark papers. Each benchmark page in this section records the exact evaluation configuration (the hyperparameters that move scores), the command to reproduce it, and Mill’s result side by side with the published figure.
Mill writes accuracy as a fraction in aggregate.csv (e.g. 0.5378); the tables in these pages show percentages to match how papers report them. The ± column is the bootstrap standard error, in percentage points.

​
Benchmarks

MMLU

5-shot multiple-choice knowledge benchmark (text).

MMLU-Pro

10-option chain-of-thought successor to MMLU (text).

CIFAR-10

Zero-shot image classification (CLIP and VLMs).

ImageNet-1k

1000-class zero-shot image classification (run pending).

MMMU-Pro

College-level multimodal multiple-choice (image + text).

MATH-Vision

Competition math reasoning over visual contexts (image + text).

Clotho-AQA

Single-word audio question answering (audio + text).

ESC-50

50-class environmental sound classification (CLAP and audio-LMs).

UrbanSound8K

10-class urban sound classification (CLAP and audio-LMs).

​
Adding a new benchmark

Create a new .mdx file under docs/reproducibility/, add it to the Reproducibility group in docs.json, and copy the block below. Replace the placeholders and fill the results table from the rollup row of your aggregate.csv (the row whose task equals the benchmark name, e.g. mmlu,mmlu).