Unit 8 · Measure and improve AI quality

Turn verified runs into reusable guidance

You accumulate situational guidance from live feedback or a verified task set. Evolution keeps grounded advice that improves performance without unacceptable held-out regression.

playbook()9 focused minutesNot started

Worked example

See the idea in context

from axllm import playbook

pb = playbook(program, {"studentAI": student_ai})
pb.evolve(examples, metric)   # grow a playbook offline from examples
pb.update({"example": ex, "prediction": pred, "feedback": "..."})  # refine online
pb.apply_to(program)

Run itIn your own project

pip install axllm

import os
from axllm import ai, ax

llm = ai('openai', api_key=os.environ['OPENAI_API_KEY'])
classify = ax('review:string -> sentiment:class "positive, negative, neutral"')
result = classify.forward(llm, {
    'review': 'Useful and boring in the best way.',
})

print('sentiment:', result['sentiment'])

Set OPENAI_APIKEY in your environment before running provider-backed code.

In the ax repo

From a clone of the ax repo:

npm run example -- python src/examples/python/optimization/axgen-optimization.py

Active practice

Show that you can use it

Answer 2 in a row to learn this · attempt 1

Keep exploring

See the idea in context

Show that you can use it

Source-backed follow-up