ai() LLM Models

Use ai() to create provider clients and keep model traffic behind one Ax request shape.

use axllm::{ax, OpenAICompatibleClient};

let mut client = OpenAICompatibleClient::new(api_key, "gpt-4.1-mini");
let mut program = ax("question:string -> answer:string")?;
let output = program.forward(&mut client, serde_json::json!({"question": "What is Ax?"}))?;

What It Does

ai() selects a provider implementation from configuration and returns a client that Ax programs can call. The client handles chat, streaming, embeddings, media where supported, usage normalization, provider options, model keys, routing hooks, tracing, and runtime defaults.

The name is a deployment profile. The model ID is resolved only inside that profile, so a DeepSeek model hosted by Together uses Together’s rules and never inherits DeepSeek’s native wire format by name. Unknown profiles fail; use the explicit openai-compatible profile plus apiURL for an unlisted endpoint. Verified reasoning rules in DeepSeek, Together, Fireworks, OpenRouter, Grok, Groq, Cerebras, and DeepInfra default an omitted thinking level to logical max, then map it to each deployment’s strongest documented effort. An explicit none is sent only where the selected model and deployment support disabling reasoning; otherwise Ax fails before network I/O. Hugging Face Router stays conservative because a routing policy can change the underlying provider. DeepSeek V4 preserves logical medium as provider medium; it is not promoted to high.

flowchart LR
  A["Model key or alias"] --> B["Model catalog"]
  B --> C["Capability filter"]
  C --> D["Provider client"]
  D --> E["Request mapping"]
  E --> F["Provider API"]
  F --> G["Response normalization"]
  G --> H["Usage + trace"]

Core Call Shape

Create the client once near the application boundary, then pass it into forward(), streamingForward(), agents, flows, or optimizers.

client = ai(provider options)
result = program.forward(client, inputs)

Common Patterns

• Use a provider name and environment-backed API key.
• Set a default model in provider config when the app has one obvious model.
• Define model aliases when callers should choose fast, smart, or cheap instead of provider model IDs.
• Use the named profile for a documented deployment. Reserve openai-compatible plus apiURL for an unlisted custom endpoint.
• Use model catalog helpers before runtime when the UI needs provider/model selectors, portable thinking levels, or verified service tiers.
• Use routers or balancers when provider fallback is part of the product.

ProviderRouter selects a provider by request capability and degrades media only when the selected provider cannot handle it. For an image-capable provider, native image parts retain their payload, MIME type, detail level, cache and optimization hints, alt text, and ordering with surrounding text.

Provider/model capability metadata exposes an ordered native, function, and json_object list. auto follows that order, while the singleton string/code optimization can choose validated json_object when native schema is absent. An explicit unsupported mode fails before transport. structuredOutputs remains the compatibility alias for native JSON Schema support, not for every JSON response format. The selected rung is recorded with the chat log so runs remain comparable and debuggable.

Typesafe / Jev typed inference

The typesafe profile supports ordinary Ax signatures with required boolean and class outputs. Booleans use Noul with provider-level trueThreshold (default 0.5, inclusive comparison); classes retain the native Choice label. Field names and descriptions define the questions. Signature syntax and return types stay the same.

Boolean value descriptions such as boolean(true "...", false "...") and class label descriptions after the label list become native criteria. The same signature retains those descriptions as text with other providers. The Typesafe/Jev skill covers the exact syntax, question design, native requests, transport options, and runnable examples.

// ax-example:start
// title: Rust Jev Signature Decisions
// group: generation
// description: Converts Jev probabilities into boolean and class outputs with a provider threshold.
// provider: typesafe
// env: TYPESAFE_APIKEY
// level: beginner
// order: 35
// ax-example:end
use axllm::{ai, ax, AxResult};
use serde_json::json;
use std::env;

fn main() -> AxResult<()> {
    let mut model = ai(
        "typesafe",
        json!({"api_key":env::var("TYPESAFE_APIKEY").expect("Set TYPESAFE_APIKEY"),"trueThreshold":0.9}),
    )?;
    let decision = ax("ticket:string -> urgent:boolean(true \"Customers cannot complete a core task\", false \"Routine request\") \"Needs immediate attention?\", team:class \"support, billing, engineering\"")?.forward(&mut model,json!({"ticket":"Checkout is unavailable for all customers after the latest deployment."}))?;
    assert!(decision["urgent"].is_boolean());
    let output = decision;
    println!("{}", serde_json::to_string_pretty(&output)?);
    Ok(())
}

Use the typesafe provider profile for these signatures. Numeric outputs, including bounded numbers, are rejected: bounds do not define a rating rubric. Freeform text, optional outputs, arrays, nested objects, media, tools, and generation controls such as temperature are also unsupported. Typesafe has no token streaming API; its stream interface delivers a completed result.

Use the separate typesafe native client for rich Noul/Choice/Score questions and structured state. systemOne({ state, questions, model? }) returns native answers, probabilities, confidence, and usage. Noul remains a probability; Score remains a fractional zero-based rubric position. Custom thresholds and scale conversions are caller decisions. listModels() discovers native models without changing configured Ax model aliases. The default model is jev-latest. Choice supports up to 255 options and Score requires 2–10 descriptive levels. Their probabilities must be finite values in [0, 1], match the criteria keys, and sum to one within an inclusive 0.01 tolerance. Totals of 0.99 and 1.01 are accepted with an allowance for floating-point summation error. Ax preserves the returned probabilities without renormalizing them. The provider’s context limit covers state, instructions, and criteria; Ax does not silently truncate input or claim an exact local token count.

Usage and raw adapter answers remain available through existing usage and chat-log APIs. Routing excludes unsupported requests even with degradation enabled. Typesafe-only balancers send schemas for scalar signatures; mixed pools can select Typesafe when the actual request already has a supported schema.

TypeScript, Python, Java, C++, Go, and Rust support both interfaces. Each language has runnable examples for configurable boolean conversion, native criteria/scoring, and an explicit hybrid flow that passes decisions to a second Ax program for prose. The generated language APIs use native typed maps, records, or enums for answers; TypeScript additionally infers literal question names and Choice-label unions.

Meta Muse models

Use meta for Meta’s recommended Responses transport, meta-chat for Chat Completions, or meta-messages for Anthropic-compatible Messages. All three use MODEL_API_KEY bearer authentication and default to muse-spark-1.3. Ax maps logical highest reasoning to xhigh; unsupported reasoning and forced named-tool choices fail locally.

Muse Spark uses the normal chat API. Muse Image (muse-image-1.0) also uses chat through meta: prompts and reference images go in the existing content array, and generated images come back in results[].images with their MIME type. Preserve the result in chat memory to edit the image on a later turn. Ordinary function tools are not supported by Muse Image.

Muse Voice (muse-voice-transcribe-1.0) uses transcribe for mono WAV files and the existing streaming chat surface for realtime PCM16 at 16 or 24 kHz. Transcription supports speaker labels, turn timestamps, language bias, keywords, progress, and session IDs. It does not synthesize speech. Realtime partial captions are replacement snapshots in results[].transcript (text plus isFinal, or is_final in generated languages), keyed by result ID. Replace each partial instead of concatenating it. Normal content contains finalized turns once, in speech-start order. Audio sending and event reception run concurrently.

Contributor variants explicitly allow provider-training data use and are never defaults. Muse Glimmer runs through the existing vllm, llama-cpp, ollama, or lm-studio profiles; Ax does not download or manage weights. No standalone image-generation or file-management service methods are added.

Renewable credentials

Use the language’s credentialProvider / credential_provider callback for expiring deployment tokens. It receives the profile, operation, method, and URL for every request attempt. Returned headers override static authentication; callback failures stop before transport. Ax refreshes on retries but does not automatically replay a completed 401 or 403. Keep ADC or cloud-SDK token sources in the host application rather than Ax core.

Vertex routing and OpenAI prompt caching

Gemini and Anthropic Vertex clients accept a project, location, and optional endpoint. Ax resolves global, US/EU multi-region, and regional hosts; an explicit base URL takes precedence. Generated packages accept a renewable credential callback, leaving ADC acquisition and refresh to the host application.

The OpenAI-compatible vertex-ai profile keeps unknown models conservative. Documented Gemini MaaS IDs prefer native schema. The exact google/gemma-4-26b-a4b-it-maas rule prefers JSON-object output, excludes native schema, defaults thinking to max, writes nested enable_thinking, and extracts/replays reasoning_content.

GPT-5.6 OpenAI Chat requests can opt into stable explicit prompt-cache breakpoints. Give AxGen a stable promptCacheKey plus contextCache; those forward options reach the provider in every language. Cache reads and writes are normalized separately for usage and catalog-backed cost estimates.

GPT-6 Astra and automatic sessions (TypeScript)

Select ai({ name: 'openai', config: { model: AxAIOpenAIModel.GPT6Astra }, apiKey }). Import ai and AxAIOpenAIModel from @ax-llm/ax. Ax automatically routes Astra through Responses. Existing defaults are unchanged. Use thinkingTokenBudget: 'low' and serviceTier: 'standard'. Astra requires reasoning; minimal maps to low and none throws. Unsupported sampling and log-probability options are removed. EU residency does not support priority processing.

Keep calling forward() and streamingForward(). Declare independent tools with fn('lookup').description('...').execution('background').handler(...).build(). Ordinary tools default to blocking. JavaScript promises and MCP annotations do not opt a tool into background execution. Set asyncMode: 'off' to use the ordinary tool loop. Providers without session support retain that loop.

Use const control = runControl() and pass { control } in forward options. Call control.steer(text), control.setThinkingTokenBudget('high'), or control.abort(). control.onEvent(listener) observes queued/applied updates, run lifecycle, tool activity, and model output activity. Untargeted updates apply to the root and future descendants. { target: 'root/nodeName' } restricts an update to a flow node and its descendants. Completed nodes are not rerun. Controller-attached runs bypass result caching; provider prompt caching remains enabled.

HTTP streaming needs no WebSocket dependency. With a configured host options.webSocket, steering can apply natively during generation; otherwise it applies at the next response boundary. Observe the applied event’s timing. Reasoning updates use continuation input items, retaining the original prefix. Steering that awaits tool input is continued even when its pending notification arrives after completion. Duplicate acknowledgements do not apply an update twice.

Ax owns tool execution and result submission. Only completed calls execute; pending results are incorporated before successful final output. Streaming clients reset accumulated output when version changes; provisional answers never count as successful completion. Sessions pin the selected provider and model and do not reconnect or replay calls after a failure. Cancellation requests tool cancellation; it does not undo external work. The native Responses wire client is internal. Custom providers may implement the optional normalized openChatSession contract; existing .chat() services work. Session adapters should expose their transport abort signal so pending host work does not prevent cancellation or disconnection from ending the run. Sessions preserve provider defaults and model-alias settings; explicit request settings win.

Runnable examples: typescript/generation/astra.ts, astra-async-tools.ts, astra-steering.ts, astra-reasoning-update.ts, astra-session-lifecycle.ts, and typescript/short-agents/astra-background.ts. Python, Go, Java, C++, and Rust have provider-backed Astra generation, agent, flow, and cancellation examples in their language galleries. Their broader session parity is still being verified in the AxIR backlog. Independent flow groups now dispatch owned workers, with a traced serial fallback for unsupported custom clients or programs. Remaining agent invocation and session acceptance evidence is tracked in the backlog.

The Java, C++, and Rust WebSocket adapters track response activity as frames arrive. A completed response cannot become active again when an older buffered event is consumed. When no response is active, steering is queued for the next response; an active successor can still receive native steering. Observe lifecycle timing rather than assuming that every update applies natively. All five session adapters also reject invalid raw-schema arguments before invoking the handler, allowing the model to correct its call within the step limit. These fixes have deterministic regression coverage; they do not establish full generated-language parity.

Gemini thinking levels

Ax resolves the effective Gemini model before translating a logical thinking level. Gemini 3 requests send thinkingLevel, clamped to the levels supported by the selected model family; Gemini 2.5 and older requests send a numeric thinkingBudget. Numeric budgets fail locally for Gemini 3, and logical none always hides returned thoughts even when the model must retain its minimum thinking level.

The native google-gemini, gemini, and google_gemini deployment profile names share this behavior, including native Gemini configured for Vertex with a project and region. The separate OpenAI-compatible vertex-ai profile remains profile-owned and does not gain native Gemini request fields from its model ID.

Portable inference service tiers

Every language package accepts the shared auto, standard, flex, and priority service-tier policy. A per-call value overrides a model preset or instance default. Named provider profiles translate that policy to their wire dialect, while unsupported explicit tiers fail before transport. auto is omitted when a provider has no explicit auto value.

The applied provider value is normalized into model usage, including aliases such as default, on_demand, and performance. Tier-aware model metadata can also provide Flex or Priority token-price overrides so cost estimates follow the tier that actually served the request. Gemini supports tiers only on GenerateContent; Vertex AI and Gemini Live reject explicit tiers. Anthropic fast mode remains a separate API. The Generation catalog demonstrates the portable per-call option using Gemini Flex in every language.

let mut client = ai("google-gemini", json!({
    "api_key": api_key()?,
    "model": model,
}))?;
let out = client.chat_with_options(request, json!({"service_tier": "flex"}))?;

Adaptive balancing

AxBalancer keeps its existing ordered failover behavior by default. Set strategy.type to adaptive to rank equivalent providers per chat request using learned reliability, successful latency, a deadline, and estimated cost. Configure badOutcomeCost in the same currency or unit as the route cost estimate.

Use the native stats-store option for authoritative decision state. The built-in in-memory store can be shared by balancers in one process; multi-process applications can implement AxBalancerStatsStore with an atomic Redis or database update. The routing-event hook is best-effort telemetry, not routing state. Stable route keys are required with a shared store, and namespace plus slice keep unrelated traffic from learning from each other.

Adaptive balancing does not inspect prompt meaning or decide which model is best for a task. The application defines acceptable substitutes through shared logical aliases.

Incremental provider streaming

Generated Python, Java, Go, Rust, and C++ provider clients expose each SSE event as soon as it arrives. Their closeable streaming APIs propagate through provider routers, multi-service routers, and balancers. Retry or failover is allowed only before the first content event; after delivery begins, an upstream failure is surfaced without replaying content or switching providers. Usage and completion telemetry finalize after full consumption, while cancellation closes the HTTP response immediately.

for event in client.stream_iter(request)? {
    let event = event?;
    print!("{}", event["results"][0]["content"].as_str().unwrap_or(""));
}

Portable cancellation

Generated Python, Java, Rust, and C++ calls accept a shared, thread-safe AxCancellationToken; Go continues to use context.Context. Cancellation is one-shot and first-reason-wins, propagates through provider retries and routing, and always becomes a non-retryable AxAIServiceAbortedError. A pre-cancelled call makes no transport attempt, retry backoff wakes promptly, and an active stream stops after its current I/O boundary. Realtime WebSocket turns use their existing lifecycle and are outside this contract.

let token = AxCancellationToken::default();
token.cancel("user stopped");
let error = client.chat_with_cancellation(request, json!({}), &token).unwrap_err();
assert_eq!(error.error_type.as_deref(), Some("AxAIServiceAbortedError"));
assert!(!error.retryable);

Provider clients

Generated Package Provider Path

The Rust package exposes the AxIR-supported provider surface. Public examples use OpenAI-compatible clients, while internal fixtures cover provider normalization without credentials.

use axllm::{ax, OpenAICompatibleClient};

let mut client = OpenAICompatibleClient::new(api_key, "gpt-4.1-mini");
let mut program = ax("question:string -> answer:string")?;
let output = program.forward(&mut client, serde_json::json!({"question": "What is Ax?"}))?;

Vertex Gemini

let mut client = ai("google-gemini", json!({
    "api_key": required("GOOGLE_VERTEX_ACCESS_TOKEN")?,
    "project_id": required("GOOGLE_PROJECT_ID")?,
    "region": required("GOOGLE_REGION")?,
    "model": model,
}))?;

Use the generated package examples for exact provider API runs, prompt-cached AxGen calls, stream mapping, Responses audio mapping, and realtime event folding for this language.

Deployment profile matrix

This matrix is generated from ir/axcore/data/provider-profiles.json. Defaults are conservative; exact or pattern rules apply only inside the selected profile, and callers can supply explicit model metadata for a deployment they have verified.

Major-version migration

Profile-only branded clients were removed. Keep genuine transport clients when you need a low-level transport boundary; otherwise replace a branded constructor with the language’s named factory (NewAI("deepseek", options) in Go, ai("deepseek", ...) where that factory shape is exposed, and ai({ name: 'deepseek', ... }) in TypeScript). Model enum/catalog exports remain available.

Embeddings and audio

// Implement embedding calls through the generated AxAI client surface when present.
// Use package conformance coverage to confirm current support for this language.

// Realtime audio over WebSocket — cargo add axllm --features realtime
let client = ai("openai-responses", json!({"model": "gpt-realtime-2"}))?;
let request = json!({"model": "gpt-realtime-2", "chat_prompt": [{"role": "user", "content": "Say hello."}], "audio": {"output": {"voice": "alloy"}}});
let response = client.realtime_chat(request, None)?;  // one merged turn: transcript + base64 PCM audio
// Realtime models also route transparently through chat(); chat() accepts input_audio parts; transcribe()/speak() do batch STT/TTS.

Practical Notes

• Prefer the named deployment-profile factory over direct provider classes in new code.
• Use model catalog and provider-scoring helpers when choosing between providers.
• Use a multi-service router to dispatch caller-selected model keys; use a balancer for fallback or adaptive operational routing across equivalent services.
• Keep public provider examples separate from internal conformance fixtures.
• Trace provider requests, token usage, estimated cost, and routing decisions in production.

See ai() API.

File routing

A provider router preserves files for providers that can read them directly. File names, media types, cache metadata, and content order survive routing and later conversation turns. For providers without native file support, supply extracted text or a file-to-text callback, or choose a degradation, skip, or error policy. The router checks the selected model before preprocessing the request.