Benchmarking Models

Goal: Learn to benchmark models on standard tasks like MMLU and HumanEval so you can pick the right model for a job with data instead of hype.

Do:

1. Read this guide: https://www.ibm.com/think/topics/llm-benchmarks

2. Run your first benchmarks with this tool https://github.com/EleutherAI/lm-evaluation-harness