Skip to main content
ImageNet-1k is the standard 1000-class image classification benchmark (50K validation images). Mill registers it as one benchmark with two renderings, picked by model capability: CLIP-style models run zero-shot classification — each image is scored against the 1000 class names by image-text similarity, ensembling 80 prompt templates per class — and vision-language models run a generative multiple-choice variant, where the true class plus 9 random distractors are shown as lettered options (A–J) and the answer letter is parsed from the generation.

​
Evaluation configuration

​
Class labels

Mill scores classification against the class names, so each of the 1000 classes must be a distinct, single label. The names are OpenAI’s curated CLIP labels (for example, the WordNet “crane” is already split into crane bird and construction crane). Two of those curated names still collided — identical strings get identical image-text similarity, so the correct class could not reliably win — and Mill disambiguates them to their own WordNet first synonym:
Only these two entries change; the other 998 are identical to the clip_benchmark export, so zero-shot scores stay directly comparable. A regression test asserts all 1000 class names are unique, single labels.

​
Reproduce

​
Results

Results pending a full evaluation run. After running the command above, fill in the table below from the imagenet rollup row of your aggregate.csv, using the open_clip results as the reported baseline for CLIP zero-shot top-1.