Skip to main content
Use fallback models when you want the AI Gateway to try another model if the primary request fails. Fallback is useful when you want better availability across providers, or when you want to try a private model first and use a hosted model as a backup.

​
Add fallback models

Set the primary model with model. Set fallback models with models.
The gateway tries gpt-4o first. If that request fails with a rate limit, an auth error, a server error, or a timeout, it tries each entry in models in order until one succeeds. Other 4xx errors, such as an invalid request, return right away. See what triggers failover.

​
Choose the provider for each fallback

Fallback models can use model IDs or provider:model.
Use provider-qualified model names when you want to control exactly which provider each fallback uses.

​
Fall back from a private model to a hosted model

Fallback works with custom providers, too.
In this example, the gateway tries your my-ollama provider first. If that request fails, it falls back to OpenAI.

​
Choose fallback models carefully

Pick fallback models that can handle the same kind of request as the primary model. Check each model for:
  • Input modality.
  • Tool calling support.
  • Output size.
  • Context window.
  • Latency expectations.
  • Provider credentials.
Use the model catalog to compare built-in model capabilities.

​
Access restrictions still apply

Fallback models must be allowed by the access key configuration assigned to the access key. If a fallback model isn’t allowed, the gateway skips it and names it in the Ngrok-AIG-Rejected-Models response header. The request still runs with the models that are allowed. It fails only when none of the models you listed are allowed.

​
Next steps