Routing

How routing works

Each request gets a difficulty score from a heuristic classifier weighing prompt structure, reasoning markers, length, and task type, and that score maps to a tier.

Tiers

What you actually save

Savings come from the queries that get downgraded. Hard queries route to the baseline model itself, so they save nothing by design: the reduction comes entirely from the tiers below it. How much depends on your provider, because the baseline differs:

Your overall reduction is therefore a function of how much of your traffic genuinely needs a frontier model. The example notebooks measure this on a mixed workload rather than asserting a number.

Auditable, not a black box

Every response carries the score, the label, the model chosen, and the counterfactual cost against your baseline model:

response = await client.messages.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarise this changelog entry."}],
)

meta = response.fluxcompute
print(meta.difficulty_score)      # 0.12
print(meta.difficulty_label)      # "easy"
print(meta.model_selected)        # the model actually dispatched to
print(meta.baseline_model)        # what savings are measured against
print(meta.cost_usd, meta.baseline_cost_usd, meta.savings_usd)

Pin a specific model any time by passing it instead of "auto". The cost accounting still runs, so you keep the comparison.

Multi-turn sessions

Sessions get tier-aware context compression. Pass a session_id and history is carried across turns and compressed when a session drops to a cheaper tier, so switching models mid-conversation doesn't re-bill the full transcript.

await client.messages.create(
    model="auto",
    messages=[{"role": "user", "content": "And the second one?"}],
    session_id="support-thread-4131",
)