Test AI behavior on real examples You record realistic inputs, criteria, expected or forbidden actions, predictions, and traces. Use deterministic metrics when possible and judges when quality needs holistic review. python academy academy/topics/evals-metrics-judges website/content-src/academy/course.mjs academy Test AI behavior on real examples
ax
Python axllm
TypeScript Python Java C++ Go Rust
Unit 8 · Measure and improve AI quality

Test AI behavior on real examples

You record realistic inputs, criteria, expected or forbidden actions, predictions, and traces. Use deterministic metrics when possible and judges when quality needs holistic review.

9 focused minutesNot started
Unit example (nearest native match)

See the idea in context

from axllm import AxGEPA

engine = AxGEPA(reflection_client)
result = engine.optimize(request, evaluator)
  1. Use a realistic input

    The refund request represents the kind of task the agent will face.

  2. Write the success rule

    criteria explains that eligibility must be verified before action.

  3. Record observable behavior

    expectedActions lets the evaluation check tool selection, not only final prose.

Run itIn your own project
pip install axllm

from axllm import ai, ax

llm = ai('openai', api_key=os.environ['OPENAI_API_KEY'])
classify = ax('review:string -> sentiment:class "positive, negative, neutral"')

Set OPENAI_APIKEY in your environment before running provider-backed code.

In the ax repo

From a clone of the ax repo:

npm run example -- python src/examples/python/optimization/axgen-optimization.py
Active practice

Show that you can use it

Answer 2 in a row to learn this · attempt 1
Keep exploring

Source-backed follow-up