Skip to main content
When an upstream attempt fails, the AI Gateway either moves to another candidate or stops and returns the error. This page explains which failures it retries, in what order, and what bounds the retrying.

​
What triggers failover

The gateway moves to another candidate on these failures:
  • Authentication failures: HTTP 401 or 403 because the provider rejected the key. A 401 or 403 for any other reason stops the request.
  • Rate limits: HTTP 429.
  • Server errors: HTTP 500 and above.
  • Request timeouts: HTTP 408, or the attempt exceeding your per-request timeout.
  • Network failures: connection errors, DNS failures, and TLS errors.
Any other 4xx response stops the request immediately. A malformed request or an unsupported parameter is not going to succeed on a different provider, so the gateway returns the error rather than spending your quota on it.

​
Failover order

Where the gateway goes next depends on how the attempt failed. Model candidates come from the model field first, then the models array in order:
The gateway never retries the same model and key combination, except in the one case below. A bare model ID, such as gpt-4o, also expands to every provider that serves that model. The model author’s own provider goes first, then resellers such as OpenRouter. A provider:model ID, such as openai:gpt-4o, reaches only that provider.
To fail over to a different model, list it in the models array, or set it up as a fallback model.

​
Retrying the same provider

When a provider answers 429 or 503 with a Retry-After header, the gateway waits and retries the same backend instead of moving on. It does that only when the wait is 5 seconds or less, only when the wait still fits inside your total timeout, and at most twice per request. Otherwise it treats the response as a rate limit or a server error and moves to the next candidate.

​
What bounds the retrying

Two timeouts and an attempt cap bound every request: Streams work differently after the first byte. See Streaming responses. Set the timeouts for your account in Account Settings, or through the Configuration API. When the total timeout is reached, failover stops even if candidates remain, and the gateway returns ERR_NGROK_3810.

​
Failures that never reach failover

Some requests fail before the gateway makes any upstream attempt. These return immediately, and the response carries no Ngrok-AIG-Attempts header:
  • The request body could not be parsed.
  • The model is unknown, or the model address is malformed.
  • The requested provider or model is blocked by the access key configuration’s access scope.
  • No provider key is available for the requested provider.
  • The account has no credits. This happens even if you use your own provider keys.
None of these are billed. The gateway bills an attempt only once its response reaches your client, so a request that never reached a provider costs nothing. A stream that fails partway through is billed, because its first bytes were already delivered. See Error Codes for the code each of these returns.

​
Best practices

  1. Attach several provider keys per provider so an exhausted credential fails over instead of failing the request. See Multi-key failover.
  2. List candidates from more than one provider so a provider-wide outage has somewhere to go.
  3. Set timeouts to match your latency budget. A total timeout that is too tight cuts failover short; one that is too loose makes a slow failure look like a hang.
  4. Record Ngrok-AIG-Request-Id from every response so you can look a failure up afterwards. See Debug failed requests.

​
Next steps

Error Codes

Error codes, causes, and solutions

Debug failed requests

Diagnose a failure from the response