First Issue of Forest Friends is Out!
The first issue of Forest Friends on Large Language Model System Evals in the Wild is out! Those that have pre-ordered have received their copy [^1]. Those of you who have been waiting to see if it'd come to fruition, the wait is over. It's now available for sale.
We originally targeted thirty pages but ended up expanding to sixty covering how to get started on designing system evals, from vibes, through choosing quality dimensions, aligning LLMs as judges, and finally t0 analyzing the evals scores.
Here's an engaging intro to evals by @sridatta and @iamwil. They've clearly put a lot of care and effort into it, where the content is well organized with plenty of illustrations throughout.
— Eugene Yan (@eugeneyan) August 26, 2024
Across 60 pages, they explain model vs. system evals, vibe checks and property-based… pic.twitter.com/8TnTvAKc0D
We go through some worked examples with Fox and Bear trying to run an LLM-driven cafe that creates recipes based on customer requests as we introduce the different concepts.
We also cover how to use LLMs-as-a-judge, by first using a golden dataset to align the LLM, and iterating on the prompt to drive precision and recall metrics up. And if you're new to data science, we explain precision and recall too.
So if you want to get an intro to the eval side of building LLM-driven apps, this is a good place to start. If you're still unsure, click here for the preview, and feel free to ask me any questions @iamwil on Twitter.
[^1]: if you pre-ordered and had trouble with the download, get in touch at iamwil@gmail.com