Quickstart

pip install "mill-eval @ git+https://github.com/haideraltahan/Mill.git"

mill eval "meta-llama/Meta-Llama-3-8B-Instruct[dtype=bfloat16,batch_size=8]" mmlu \
  --output_dir ./results

mill --output_dir ./results collect --metric acc

Mill results — performance (acc)
┌─────────────────────────────────────┬────────┐
│ model                               │ mmlu   │
├─────────────────────────────────────┼────────┤
│ meta-llama/Meta-Llama-3-8B-Instruct │ 0.6398 │
└─────────────────────────────────────┴────────┘

mill ls