When starting off with eval, you might start with humans. It gives you experience with edge cases, and it hones your intuition of what good actually looks like. But once you do that, you need to scale up a little, which means writing guidelines on what good looks like. This
Read more
LLMs-as-a-judge sounds counterintuitive. How can LLMs judge its own work? If it could, wouldn't it just do better in the first place? Remember, LLMs are like stochastic functions out of the box. They don't have any memory outside of what you provide in its context. And
Read more
Hello! Sri and I want to thank you for purchasing the first issue of Forest Friends: LLM system evals in the wild. In this first week since launch, we've hit our sales goal! And hopefully, you've had a chance to look through the issue. If you&
Read moreThe first issue of Forest Friends on Large Language Model System Evals in the Wild is out! Those that have pre-ordered have received their copy [^1]. Those of you who have been waiting to see if it'd come to fruition, the wait is over. It's now
Read more
You've got an LLM in prod and are relying on "vibes" to gauge output quality. Sound familiar? It's a start, but you'll need to instrument your pipeline and get systematic to drive real improvements. Our brains are powerful pattern-matchers for what makes
Read moreIf you're struggling to whip your LLM output into shape, start with "vibe-based engineering", by judging outputs by look & feel. It doesn't scale, but it's the foundation for systematizing evals later. What is vibe-based engineering? Just start by looking at the
Read more
Sci-fi has set our expectations for LLMs incorrectly. People write a one or two-line prompt and give up when they don’t get the expected output. To effectively use LLMs today, you need to adopt a particular habit of mind. Just like writing in school, the only way you get
Read more
Some thought should be paid to the grading scale when systematizing your LLM evals, regardless of whether they're for humans for LLM-as-a-judge. A bad grading scale can skew your ability to assess your output accurately. Grading scales are the ratings by which humans or LLMs judge the output
Read more
There's no one tool that will nail down a notion of "good" for you to express in a LLM eval. Instead, you need to use multiple tools to triangulate this notion of "good" to evaluate an output. A simple, yet overlooked eval is code.
Read more
What constitutes good? LLMs aren’t magic. If you can’t articulate what good looks like, the LLM won’t know either. Part of that is learning to pick the metrics to bracket in that elusive definition of good. All that you learned in ML engineering doesn’t go away
Read more
Hi all, thanks for your interest in the LLM eval digital zine either through Twitter or meeting Sri, the guy with "Ask me about Evals tshirt" at the AI conference. Today, you can pre-order the first issue of detailing LLM evals at forestfriends.tech. For a limited time,
Read more