What happens when a provider goes down

A request fails before its first token and you never hear about it. How failover works, and where it stops.

Upstream providers run out of credit, reject keys and fall over. If your app talked to one of them directly, every one of those would be your outage. Behind AuraRoute they are the router’s problem.

Failover, step by step

  1. The router sends your request to the best healthy provider for its workload.
  2. If that provider fails before sending its first token, the request moves to the next best one. It tries up to three providers.
  3. The provider that failed is marked unhealthy, so new requests stop going to it until it recovers.
  4. The answer streams back to you under our request id and model name, with no upstream details.

Where it stops

Once the first token has been streamed to you, the answer is already on its way and the request cannot move. That is why failover happens before the first token, and why the request log records every attempt.