2026-11-11 · America/Los_Angeles · 社区动态 · #7

Your LLM fallback never fires for the provider that's just slow

Most fallback code I've seen retries on the exception, which handles 429s and 500s fine because those fail before the response starts. The case it misses is a provider that accepts the request and then stalls before the first token: nothing throws, nothing falls back, and the client sits there until its own timeout fires. Covering that needs a time-to-first-byte timeout separate from the overall request timeout, and it has to fire before anything is written to the client. Once the first chunk is out, a retry is just a second answer next to a half-finished one.

热度 26.7 / 100；排名与评分保留该期记录。