Skip to main content
MMMU-Pro is a robust, college-level multimodal benchmark: questions carry up to 7 images and 10 answer options (A–J). Mill runs the standard (10 options) config in two renderings of the same data:
  • mmmu_pro — generative chain-of-thought for vision-language models: the model sees the image(s) and lettered options, reasons step by step, and ends with Answer: $LETTER, which is extracted and graded by a faithful port of the official MMMU-Pro grader.
  • mmmu_pro_clip — a CLIP zero-shot rendering for image–text encoders: each option is scored as question + option against the (first) image by image–text similarity. CLIP sees one image and truncates to its 77-token context, so this is a deliberately weak read — it exists for CLIP-family coverage, not as a strong solver.

​
Evaluation configuration

​
Reproduce

​
Results

Qwen3-VL-2B-Instruct

31.91% ± 1.12
generative CoT

CLIP — ViT-B-32 (laion2b)

17.86% ± 0.92
zero-shot (near-chance)
There is no widely-published MMMU-Pro standard number for these exact checkpoints, so Mill’s measurements are recorded as the initial baselines. For context, the similarly-sized Qwen2.5-VL-3B reports ≈31.6% on MMMU-Pro, consistent with Mill’s Qwen3-VL-2B result. The CLIP rendering sits just above the 10-option chance rate (10%), as expected for a single-image, 77-token similarity read.

​
Per-model results