Agents: this page is also Markdown; every page is in llms.txt.

Write good evals

An eval is one plain statement of what a good answer does, such as "marks a double charge as urgent". Facet checks every new version against every eval, and a version goes live only when they pass. Good evals are the best way to get a Model that does what you mean.

Facet drafts the first ones from your description. This page is for reviewing them and adding your own.

One behavior per statement

Each eval should be something you could check by reading one answer.

Split a statement that has "and" in it: two evals tell you which half fails.

Give it examples

Each eval has examples: inputs, and the right answer when there is one. Facet uses them two ways:

• To teach. Facet writes more examples like yours to build each version.
• To check. Some examples are hidden. When you make a Model, Facet hides a share of the ones you give it; when you upload to an eval, you choose with Hide these. Once hidden, an example is read only by Facet's checks, never by Facet's own agent and never used to build a version, so no version can learn its own test.

Real inputs from your app beat invented ones. Edge cases beat easy ones: the ticket written in two languages, the invoice with two totals. See Add examples.

How each one is checked

Facet picks how to check each eval, and you can see and change it on the eval's page:

• Code check: exact rules, such as "the label is one of billing, bug or account". Cheapest, and exact.
• Score: how closely the answer matches the right one, such as an exact match.
• Judge: another model reads each answer against a short checklist you can edit, for things only a reader can tell, such as tone.

The bar

Each eval has a bar, the share of its hidden examples a version must get right:

• At least a number, such as 90%. Use it when you know what good enough is.
• No worse than the live version, within a margin you set. Use it to protect something that already works while another eval improves.

A new eval that today's version already passes is fine; it protects that behavior from future changes.

What the colors mean

• Green: the live version clears the bar.
• Red: it doesn't. Ask for a change to fix it.
• Gray: fewer than 30 hidden examples, too few to judge; add more.

Who can loosen one

Lowering a bar, removing an eval, or making it optional (shown, but no longer required to merge) needs a Model owner's approval, so an eval can't quietly stop protecting you.

Next steps

• Add examples: formats and how many you need
• Make a Model better: turn a red eval green