ai() LLM Models

Use ai() to create provider clients and keep model traffic behind one Ax request shape.

import { AxAIOpenAIModel, ai } from '@ax-llm/ax';

const openai = ai({
  name: 'openai',
  apiKey: process.env.OPENAI_APIKEY!,
  config: { model: AxAIOpenAIModel.GPT4OMini },
});

What It Does

ai() selects a provider implementation from configuration and returns a client that Ax programs can call. The client handles chat, streaming, embeddings, media where supported, usage normalization, provider options, model keys, routing hooks, tracing, and runtime defaults.

flowchart LR
  A["Model key or alias"] --> B["Model catalog"]
  B --> C["Capability filter"]
  C --> D["Provider client"]
  D --> E["Request mapping"]
  E --> F["Provider API"]
  F --> G["Response normalization"]
  G --> H["Usage + trace"]

Core Call Shape

Create the client once near the application boundary, then pass it into forward(), streamingForward(), agents, flows, or optimizers.

client = ai(provider options)
result = program.forward(client, inputs)

Common Patterns

• Use a provider name and environment-backed API key.
• Set a default model in provider config when the app has one obvious model.
• Define model aliases when callers should choose fast, smart, or cheap instead of provider model IDs.
• Use OpenAI-compatible apiURL for compatible providers.
• Use model catalog helpers before runtime when the UI needs provider/model selectors.
• Use routers or balancers when provider fallback is part of the product.

Vertex routing and OpenAI prompt caching

Gemini and Anthropic Vertex clients accept a project, location, and optional endpoint. Ax resolves global, US/EU multi-region, and regional hosts; an explicit base URL takes precedence. Generated packages take a caller-supplied bearer access token and leave ADC refresh to the host application.

GPT-5.6 OpenAI Chat requests can opt into stable explicit prompt-cache breakpoints. Give AxGen a stable promptCacheKey plus contextCache; those forward options reach the provider in every language. Cache reads and writes are normalized separately for usage and catalog-backed cost estimates.

Adaptive balancing

AxBalancer keeps its existing ordered failover behavior by default. Set strategy.type to adaptive to rank equivalent providers per chat request using learned reliability, successful latency, a deadline, and estimated cost. Configure badOutcomeCost in the same currency or unit as the route cost estimate.

import { AxBalancer, AxInMemoryBalancerStatsStore } from '@ax-llm/ax';

const statsStore = new AxInMemoryBalancerStatsStore();
const routeKeys = new Map<string, string>([
  [openai.getId(), 'openai-primary'],
  [anthropic.getId(), 'anthropic-primary'],
]);

const llm = AxBalancer.create([openai, anthropic] as const, {
  strategy: {
    type: 'adaptive',
    deadlineMs: 6_000,
    badOutcomeCost: 0.02,
    expectedTokens: { promptTokens: 1_200, completionTokens: 300 },
    namespace: 'support-summary-v1',
    routeKey: (service) => {
      const key = routeKeys.get(service.getId());
      if (!key) throw new Error('Missing stable route key.');
      return key;
    },
    slice: ({ options }) =>
      options?.customLabels?.workflow ?? 'default-workflow',
    statsStore,
    onRoutingEvent: (event) => {
      if (event.type === 'selected' || event.type === 'fallback') {
        console.log('route:', event);
      }
    },
  },
});

Use the native stats-store option for authoritative decision state. The built-in in-memory store can be shared by balancers in one process; multi-process applications can implement AxBalancerStatsStore with an atomic Redis or database update. The routing-event hook is best-effort telemetry, not routing state. Stable route keys are required with a shared store, and namespace plus slice keep unrelated traffic from learning from each other.

Adaptive balancing does not inspect prompt meaning or decide which model is best for a task. The application defines acceptable substitutes through shared logical aliases.

Provider clients

OpenAI

import { AxAIOpenAIModel, ai } from '@ax-llm/ax';

const openai = ai({
  name: 'openai',
  apiKey: process.env.OPENAI_APIKEY!,
  config: { model: AxAIOpenAIModel.GPT4OMini },
});

OpenAI Responses

const responses = ai({
  name: 'openai-responses',
  apiKey: process.env.OPENAI_APIKEY!,
  config: { model: 'gpt-4.1-mini' },
});

Claude / Anthropic

import { AxAIAnthropicModel, ai } from '@ax-llm/ax';

const claude = ai({
  name: 'anthropic',
  apiKey: process.env.ANTHROPIC_APIKEY!,
  config: { model: AxAIAnthropicModel.Claude48Opus },
});

Gemini

import { AxAIGoogleGeminiModel, ai } from '@ax-llm/ax';

const gemini = ai({
  name: 'google-gemini',
  apiKey: process.env.GOOGLE_APIKEY!,
  config: { model: AxAIGoogleGeminiModel.Gemini25Flash },
});

OpenAI-Compatible Providers

Use apiURL when a provider shares the OpenAI wire shape but uses a different host or model naming scheme.

const compatible = ai({
  name: 'openai',
  apiKey: process.env.PROVIDER_API_KEY!,
  apiURL: 'https://provider.example/v1',
  config: { model: 'provider/model-name' },
});

Embeddings and audio

const { embeddings } = await openai.embed({
  texts: ['typed LLM programs', 'runtime agents'],
  embedModel: 'text-embedding-3-small',
});

const transcript = await openai.transcribe({
  audio: { data: base64Wav, format: 'wav' },
  model: 'gpt-4o-mini-transcribe',
  language: 'en',
});

const speech = await openai.speak({ text: transcript.text, model: 'gpt-4o-mini-tts', voice: 'alloy' });

Practical Notes

• Prefer provider factories over direct provider classes in new code.
• Use model catalog and provider-scoring helpers when choosing between providers.
• Use a multi-service router to dispatch caller-selected model keys; use a balancer for fallback or adaptive operational routing across equivalent services.
• Keep public provider examples separate from internal conformance fixtures.
• Trace provider requests, token usage, estimated cost, and routing decisions in production.

See ai() API.