2026-07-31-smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses-f8f87203ee
smevals: a Small Suite for Evaluating Models, Prompts, and Harnesses
Simon Willison
EDITOR BRIEF
smevals is a new tool for running small evaluation suites across different model setups and grading the results. It uses evals made of tasks, runs them against one or more configs, and separates execution from grading. The tool also includes commands to view results locally or export them as static HTML.
INSIGHTS
This matters because evaluations help you compare models and prompts in a repeatable way instead of guessing. A good next step is to create a tiny eval directory with YAML files and run it on a couple of models to see how scoring works.
COMMENTS
Loading comments…