The feature demoed perfectly for six weeks. Then, on a Tuesday, our model provider had a nineteen-minute latency spike, our retry logic multiplied it into a thundering herd, and the “AI-powered” search box took the entire site’s checkout flow down with it, because search and checkout shared a connection pool nobody had thought to separate. Nothing about the model was wrong that day. Everything about the engineering around it was untested.
Most writing about shipping AI features is about prompts, models, and evals — the parts that are genuinely new. Almost none of it is about the parts that are old, boring, and where we’ve actually lost the most sleep: timeouts, retries, circuit breakers, fallback paths, and the fact that a model call behaves nothing like the database queries your reliability instincts were trained on.
This is the infrastructure work it actually took to get three AI features to a point where we trust them at 3am, written down because we couldn’t find this written down anywhere when we needed it.
A model call is not a database query, and your instincts will betray you
A database query that takes 200ms is slow. A model call that takes 200ms barely exists — ours average 1.8 to 4 seconds for a real generation call, and the tail is the actual problem: our p50 is 2.1 seconds, our p99 is 14 seconds, and roughly 0.3% of calls simply never return inside any reasonable timeout at all. Every piece of infrastructure built around the assumption that “slow” means 500ms and “very slow” means two seconds will make confidently wrong decisions around an LLM call — timeouts fire too early, load balancers mark healthy instances as dead, and connection pools exhaust themselves waiting on requests that were always going to take twelve seconds.
The fix isn’t a bigger timeout. It’s treating model calls as a fundamentally different class of dependency, with their own budget, their own pool, and their own failure plan — never sharing infrastructure with anything latency-sensitive elsewhere in the system. That Tuesday outage happened specifically because it didn’t have that isolation yet.
Timeouts: pick two numbers, not one
A single timeout value is a compromise between two things that want different answers: users want a fast failure, and slow-but-successful calls want to be given a chance to finish. We now use two timeouts for every model call — a soft one and a hard one.
const SOFT_TIMEOUT_MS = 6_000; // show a fallback / loading state
const HARD_TIMEOUT_MS = 25_000; // abort the request entirely
const controller = new AbortController();
const hardTimer = setTimeout(() => controller.abort(), HARD_TIMEOUT_MS);
const softTimer = setTimeout(() => {
showInterimState(); // "still thinking" — not a failure yet
}, SOFT_TIMEOUT_MS);
try {
const result = await model.run({ signal: controller.signal, ...req });
clearTimeout(softTimer);
return result;
} finally {
clearTimeout(hardTimer);
}
The soft timeout protects the feeling of speed without sacrificing the 15–20% of calls that are legitimately just slow that day. The hard timeout protects your infrastructure from a request that’s genuinely hung. Picking one number means choosing which of those two problems you’d rather have — pick both timeouts instead.
Retries are where money and reliability actually fight each other
Naive retry logic — retry immediately, up to three times — is how a provider-side blip becomes a self-inflicted outage. Every failed request that retries instantly adds load to a system that just told you it was struggling, and if enough requests are timing out simultaneously, their simultaneous retries arrive as a synchronised spike. That’s exactly the thundering herd that took down our checkout flow.
async function callWithBackoff(fn, { maxAttempts = 3 } = {}) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return await fn();
} catch (err) {
if (!isRetryable(err) || attempt === maxAttempts) throw err;
// exponential backoff + full jitter — never retry in lockstep
const base = Math.min(1000 * 2 ** attempt, 8000);
const delay = Math.random() * base;
await sleep(delay);
}
}
}
function isRetryable(err) {
// timeouts and 5xx: retry. 4xx and content-policy errors: never —
// retrying a bad request just pays for the same failure twice.
return err.status >= 500 || err.code === 'ETIMEDOUT';
}
The jitter matters more than the backoff curve itself. Without it, every client that started failing at the same moment retries at the same moment, in the same pattern, forever synchronised. With it, the retry load smears out over time instead of arriving as a second spike on top of the first.
Circuit breakers: stop calling a provider that’s already down
Retries assume the failure is transient. Sometimes it isn’t — a provider has a real outage, and every retry during it is guaranteed to fail while still costing latency, a connection, and often money. A circuit breaker tracks the failure rate over a rolling window and, once it crosses a threshold, stops sending requests entirely for a cooldown period, failing fast to the fallback path instead.
- Closed — normal operation, requests flow through.
- Open — failure rate exceeded threshold (we use 50% over a 30-second window); requests short-circuit straight to the fallback, no call attempted.
- Half-open — after a cooldown, a small trickle of real requests is allowed through to test whether the provider has recovered, before fully reopening the circuit.
This one component is what turned “the provider is down” from a 45-minute incident into a non-event the last two times it happened. Users saw the fallback experience immediately instead of waiting through a timeout on every request while we scrambled to notice.
Every feature needs a non-AI floor, decided in advance
The question to answer before launch isn’t “what does this feature do.” It’s “what does the site do when this feature is completely unavailable for twenty minutes.” If you don’t have an answer, you don’t have a production feature — you have a demo with a single point of failure.
For our semantic search box, the floor is keyword search — worse, but functional. For the document Q&A tool, the floor is the plain document list with no generated answer. For the support-ticket classifier, the floor is the existing manual queue it was built to reduce, not replace. None of these fallbacks are exciting. All of them mean a provider outage degrades the site instead of breaking it, and deciding this in advance is the difference between an incident and a status-page update.
What we actually monitor
Four dashboards, checked or alerting automatically, for every AI feature we run:
- Latency percentiles, not averages. p50 tells you what most users feel. p99 tells you who’s about to file a complaint. We alert on p95 crossing 10 seconds, not on the average, which barely moves even during a real degradation.
- Fallback activation rate. If this is at zero, either your primary path is flawless — unlikely — or your fallback logic is broken and silently never firing. If it’s climbing, something upstream is degrading before it’s bad enough to page anyone.
- Cost per hour, not per month. A retry storm shows up as an hourly cost spike days before it shows up as a noticeable line on the monthly invoice.
- Circuit breaker state changes. Every open/close transition is logged and posted to our incident channel automatically. It’s usually the first signal we get that a provider is degrading, often minutes before their own status page updates.
The honest state of things
None of this is exotic engineering — it’s the same discipline distributed systems have always needed, applied to a dependency that’s slower, less predictable, and more expensive to retry than the ones most of us are used to budgeting for. The mistake we made in the first six weeks wasn’t choosing the wrong model or writing the wrong prompt. It was treating a model call like a normal API call and finding out, at the worst possible time, exactly how wrong that assumption was.
Key takeaways
- Model calls have a much heavier tail than typical API dependencies — our p50 was 2.1s, our p99 was 14s. Infrastructure tuned for millisecond latencies will make wrong decisions around this shape of traffic.
- Use two timeouts per call: a soft one that shows an interim state, and a hard one that actually aborts — a single timeout value forces a bad trade-off between speed and success rate.
- Retries need exponential backoff with full jitter, or simultaneous failures become a synchronised retry spike that can take down shared infrastructure.
- Circuit breakers stop calling a provider that’s already down, turning provider outages into non-events instead of 45-minute incidents.
- Every AI feature needs a decided-in-advance non-AI fallback — the real question before launch is what the site does when the feature is unavailable for twenty minutes.
- Monitor latency percentiles (not averages), fallback activation rate, hourly cost, and circuit breaker transitions — each catches a different class of degradation before users complain.
- None of this is AI-specific engineering. It’s ordinary distributed-systems discipline applied to a slower, less predictable dependency.
Frequently asked questions
What timeout values should I actually start with?
Start from your own measured p99, not a guess — set the soft timeout near your p75 and the hard timeout at roughly 1.5–2x your p99. Revisit both after two weeks of real traffic; provider latency shifts over time and a timeout tuned once tends to go stale.
Isn’t retrying a failed model call just wasting money?
Only if you retry indiscriminately. Retry timeouts and 5xx errors, since those are usually transient — never retry 4xx or content-policy errors, since those will fail identically every time and you’re paying twice for the same guaranteed failure. That distinction is most of the cost discipline right there.
Do I need a circuit breaker if I already have retries with backoff?
Yes — they solve different problems. Retries assume the failure is transient and worth another attempt. A circuit breaker recognises when failures are sustained and stops attempting entirely, saving latency and cost during an actual outage rather than retrying into a wall for the duration of the incident.
What should the fallback experience actually look like?
Functional and clearly worse, never a broken or blank state. Keyword search instead of semantic search, a plain list instead of a generated summary, a manual queue instead of automated triage. The fallback’s job is to fail gracefully in front of users, not to replicate the AI feature’s quality.
How do we decide the circuit breaker’s failure-rate threshold?
We use 50% failures over a rolling 30-second window as a starting point, then tune per feature based on how costly a false trip is versus how costly a missed one is. A feature with a cheap, graceful fallback can afford a more sensitive breaker; a feature where the fallback is expensive to trigger should trip less eagerly.
