Intelligent Retry Patterns for OpenAI-Compatible AI Gateways

Why This Topic Matters Now

The Sep 13 keyword report surfaced a small but interesting Tier 2 signal: a French-language query about an intelligent retry pattern for AI. The volume is tiny, yet the intent is useful. Teams evaluating OpenAI-compatible Chinese AI gateways often have a retry policy, but it is too broad. It retries bad requests, repeats long prompts after balance errors, or switches models without storing why. That can create reliability noise and unexpected spend at the same time.

An intelligent retry pattern is not a loop that tries harder. It is a decision table with error classes, idempotency boundaries, budget ceilings, source-dated route rows, and a final stop reason. The pattern should know when to retry the same route, when to queue, when to fall back, when to ask a human, and when to fail fast. It should also preserve enough evidence for engineering and finance to inspect the same incident later.

Source Facts Checked Today

AIWave /api/pricing checked on Sep 14, 2026 returned success=true, 64 live rows, pricing_version 5a90f2b86c08bd983a9a2e6d66c255f4eaef9c4bc934386d2b6ae84ef0ff1f1f, auto_groups=['default'], and group_ratio default=1 and vip=0.9. The static /api/v1/pricing snapshot checked the same day reported checked=2026-09-10, 64 rows, pricing_version 8c7a0c0b30661ccbc13d142cb54d1e4ae445fe774b2c6fa501080db97c7a3e56, and notes that dated base rates are adjusted by the effective account group. Selected static base rows per 1M text-token units were DeepSeek Flash at $0.70 input, $0.0233 cache-hit input, and $2.10 output; DeepSeek V4 Flash at $0.638 input, $0.0202884 cache-hit input, and $1.914 output; DeepSeek V4 Pro at $1.914 input, $0.0637362 cache-hit input, and $5.742 output; GLM-4.5 at $0.6975 input, $0.1800003375 cache-hit input, and $2.1699999225 output; Qwen3 Max at $1.5621977891181764 input and $6.248791156472706 output; Kimi K3 at $4.50 input, $0.90 cache-hit input, and $22.50 output; and Moonshot v1 128K at $1.80 input and $4.50 output. VIP-key estimates multiply the same base rows by 0.9.

AIWave status and docs checked for this run remain the public evidence path for current endpoint shape and operational context. Status evidence is useful for contract checks, but it should not be stretched into a universal uptime or success-rate claim.

QwenCloud pricing documentation checked for this run describes billing across model families, batch, context caching, thinking tokens, and tools. Retry design should therefore consider more than text tokens; repeated tool calls, repeated media calls, and long outputs can all change the cost of a retry storm.

Planning Matrix

A source-dated planning matrix keeps the page useful for engineers and procurement reviewers. It turns a search query into an auditable route decision instead of a loose model preference.

Implementation Pattern

The implementation pattern keeps credentials as placeholders, pins the AIWave base URL, records the model, and leaves room for route-specific controls. Production applications should move credentials into environment or secret storage.

import random
import time

RETRYABLE = {"rate_limit", "timeout"}
STOP = {"auth", "insufficient_balance", "bad_request", "user_cancelled"}

def next_retry(error_class: str, attempt: int):
    if error_class in STOP:
        return {"action": "stop", "reason": error_class}
    if error_class not in RETRYABLE or attempt >= 2:
        return {"action": "human_review", "reason": error_class}
    delay = min(8, 2 ** attempt) + random.random()
    return {
        "action": "retry",
        "delay_seconds": round(delay, 2),
        "api_key": "YOUR_API_KEY_HERE",
        "pricing_checked_at": "2026-09-14",
    }

time.sleep(next_retry("rate_limit", 1)["delay_seconds"])

Classify Before You Retry

The first rule is classification. A 401 authentication error does not become healthy after three attempts. A malformed request should be fixed, not retried. A balance error needs a billing action, not a fallback model. A rate limit may benefit from queueing, and an upstream timeout may permit one retry. The retry layer should read normalized error classes rather than raw strings scattered across SDKs.

Protect Idempotency

Some AI calls are safe to repeat because they only generate a response. Others trigger tool calls, write artifacts, send emails, or update user-visible state. The retry pattern should mark every operation as read-only, write-once, or side-effecting. Side-effecting operations need an idempotency key or a hard stop. Without that boundary, a retry can produce duplicate tickets, duplicate actions, or conflicting summaries.

Put a Budget Ceiling on the Loop

Retry policy is also budget policy. A long prompt retried twice with a large output cap can cost more than the original forecast. Store max attempts, max input tokens, max output tokens, allowed fallback models, and maximum applied spend per task. If the retry would exceed the ceiling, stop with a clear error. This is especially important for long-context and tool-using workloads.

Use Source-Dated Route Rows

A fallback route should not be selected by memory. It should reference a source-dated row and a route acceptance record. For example, if a primary route is Qwen3 Max and the fallback is GLM-4.5, store both base rows, the checked date, the group multiplier, and the acceptance owner. That evidence lets a post-incident review distinguish reliability decisions from hidden budget changes.

Separate Queueing From Fallback

A 429 does not always require a different model. Sometimes queueing with jitter protects the upstream and preserves answer consistency. A timeout may justify retrying the same model once before fallback. A fallback may be appropriate when the user-facing deadline matters more than model identity. The retry table should make these choices explicit by feature family and severity.

Internal Links for Reliability Buyers

This article should link reliability readers to Status, Chat Completions, Error docs, Pricing JSON, Models docs, and Trust. Those links keep reliability, cost, and operational evidence in one path.

Procurement Review

Procurement should ask whether retries are counted in forecasts, whether fallback routes have source-dated rows, and whether the system stores final stop reasons. Reliability work can create spend variance if retries are invisible. A good review asks for a sample failed request, the retry table used, the final action, and the budget ceiling that prevented unbounded attempts.

Final Checklist

A retry pattern is ready when error classes are normalized, side effects are protected, budget ceilings are enforced, fallback routes have dated price evidence, queueing and fallback are separated, and every incident stores a final stop reason. The pattern should improve reliability while making repeated spend visible rather than hiding it under a generic retry counter.