smevals: a Small Suite for Evaluating Models, Prompts, and Harnesses
Simon Willison
EDITOR BRIEF
smevals is a new tool for running small evaluation suites across different model setups and grading the results. It uses evals made of tasks, runs them against one or more configs, and separates execution from grading. The tool also includes commands to view results locally or export them as static HTML.
INSIGHTS
This matters because evaluations help you compare models and prompts in a repeatable way instead of guessing. A good next step is to create a tiny eval directory with YAML files and run it on a couple of models to see how scoring works.
Learn more with these courses
CodeFriends courses that build on this story. Practice in the browser with nothing to install.
- Introduction to Prompt EngineeringLearn technical prompting techniques to get the best answers from AI.Beginner15 Hours
- Introduction to HTML, CSS, and JavaScriptLearn the fundamentals of HTML, CSS, and JavaScript, and build your own website from scratch.Beginner25 Hours
- Introduction to Web Development (Light)Master HTML, CSS, and JavaScript in just 10 hours.Beginner10 Hours
COMMENTS
Loading comments…