Unit 8 · Measure and improve AI quality

Test AI behavior on real examples

You record realistic inputs, criteria, expected or forbidden actions, predictions, and traces. Use deterministic metrics when possible and judges when quality needs holistic review.

9 focused minutesNot started

Worked example

See the idea in context

const tasks = [{ input: { request: 'Refund order 42' }, criteria: 'Verify eligibility before refunding', expectedActions: ['orders.lookup'] }];

Use a realistic input
The refund request represents the kind of task the agent will face.
Write the success rule
criteria explains that eligibility must be verified before action.
Record observable behavior
expectedActions lets the evaluation check tool selection, not only final prose.

Run itIn your own project

npm install @ax-llm/ax

import { ai, ax } from '@ax-llm/ax';

const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });
const classify = ax('review:string -> sentiment:class "positive, negative, neutral"');

const result = await classify.forward(llm, {
  review: 'Useful and boring in the best way.',
});

Set OPENAI_APIKEY in your environment before running provider-backed code.

In the ax repo

From a clone of the ax repo:

npm run example -- typescript src/examples/typescript/optimization/axgen-optimization.ts

Active practice

Show that you can use it

Answer 2 in a row to learn this · attempt 1

Keep exploring

See the idea in context

Show that you can use it

Source-backed follow-up