OpenAI-Compatible API Gateway: Migration and Routing Guide | AIWave

What an OpenAI-compatible gateway does

An OpenAI-compatible gateway gives an application one request shape while routing calls to several model providers. The compatibility layer usually covers authentication, chat completions, streaming, usage accounting, and a common error envelope. It does not make every provider identical: context limits, tool behavior, safety policies, and model quality still need an explicit compatibility test.

Use the comparison pages to decide which capabilities matter, then keep provider-specific differences in a routing policy. A gateway is valuable when it reduces duplicated client code and makes model or cost changes reversible, not when it hides every operational fact from the team.

Migration inventory before code changes

Start by inventorying every OpenAI call site, model name, streaming path, tool schema, timeout, and cost metric. Record whether a call expects JSON, function calls, log probabilities, or a particular refusal format. This list becomes your acceptance test and prevents a migration from succeeding only on the simplest chat endpoint.

Separate application behavior from provider configuration. Put the base URL, key, model, timeout, and retry budget in environment variables. Link each service to the chat completions documentation and keep a rollback value for the previous endpoint.

Minimal Python migration

The OpenAI SDK already supports a custom base URL, so a minimal migration often changes configuration rather than business logic. Run the sample with a test key, validate that the returned object contains the fields your code reads, and capture usage metadata for a before-and-after cost comparison.

Do not assume the model string is portable. Verify the exact identifier in the model directory, and add a mapping layer if your application exposes a logical name such as fast, balanced, or reasoning. That lets you change a route without rewriting every caller.

Routing policies that are measurable

A practical router chooses a model using task class, context length, quality requirement, and budget. For example, a short classification can use a fast model, while a complex coding change can use a reasoning model. Each rule should produce a reason code so you can later see whether a request was routed by quality, cost, or availability.

Measure cost per successful task rather than cost per request. Include retries, fallbacks, cache behavior, and failed parsing in the calculation. The live pricing page and cost comparison article provide the accounting vocabulary, but your own token logs are the source for product decisions.

Retries, fallbacks, and idempotency

Retry only transient transport and upstream failures, with exponential backoff and jitter. A fallback should be selected for the task's minimum quality requirement, not simply for being available. Keep the original error, selected fallback, and final status in your trace so a successful fallback does not erase an outage signal.

For tool calls or side effects, make the operation idempotent before adding retries. A model response may be replayed after a network timeout even if the provider completed it. Use a request identifier, a server-side deduplication record, and explicit confirmation before sending an email, charging a customer, or changing data.

Observability and security

Log metadata, not raw secrets: route, model, status, latency, token counts, and a redacted request identifier. Sample prompts only when policy allows it, and encrypt or expire any retained content. Dashboard first-token latency and total duration separately because a stream can feel fast while still taking too long to finish.

Put the gateway behind your own authorization layer when end users call it. Enforce quotas, tenant isolation, model allow-lists, and maximum token budgets before the request reaches a provider. Rotate keys and use separate credentials for development, staging, and production.

Testing a multi-model gateway

Use contract tests for authentication, chat, streaming, error mapping, usage, and tool arguments. Then run a workload set that includes short answers, long context, code, Chinese text, JSON extraction, and blocked content. Compare schema validity and task success as well as latency and price.

Run shadow traffic or a canary before full cutover. Keep a fixed prompt set and record the date, provider, model, and gateway version. When a route changes, you can attribute a quality or cost shift to the actual change rather than to a noisy production week.

Operations checklist

Document the model map, provider status page, retry limits, fallback order, spend budget, and rollback command. Review it after a provider model update and after a pricing change. Link operators to the migration guide and the fallback patterns article.

A gateway should make change safer, not make responsibility invisible. Keep a human owner for routing policy, alert on spend and error trends, and schedule a monthly check of model availability, pricing, and compatibility. This turns a one-time migration into a maintainable platform capability.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AIWAVE_API_KEY"],
    base_url=os.environ.get("AIWAVE_BASE_URL", "https://aiwave.live/v1"),
    timeout=30.0,
)
response = client.chat.completions.create(
    model=os.environ.get("AIWAVE_MODEL", "deepseek-v4-flash"),
    messages=[{"role": "user", "content": "Return a one-line health check."}],
    max_tokens=40,
)
print(response.choices[0].message.content)

import os, time
from openai import OpenAI

client = OpenAI(api_key=os.environ["AIWAVE_API_KEY"], base_url="https://aiwave.live/v1")
for attempt in range(3):
    try:
        out = client.chat.completions.create(
            model=os.environ.get("AIWAVE_MODEL", "deepseek-v4-flash"),
            messages=[{"role": "user", "content": "Classify this request as billing or technical."}],
            max_tokens=20,
        )
        print(out.choices[0].message.content)
        break
    except Exception:
        if attempt == 2:
            raise
        time.sleep(2 ** attempt)

Chat completions docs · AIWave pricing · Model directory · OpenAI migration guide · Fallback patterns · Compare models

Gateway policy before implementation

An OpenAI-compatible gateway standardizes a request shape, not model quality or safety behavior. Define task classes, context needs, output formats, and budgets before choosing routes. Keep the policy in version control and link operators to the model directory.

Inventory every OpenAI call site, streaming path, tool schema, timeout, and cost metric. This list becomes a contract-test plan and gives you a rollback target before the first canary.

Prompt and token budgeting

Token cost starts with the prompt that the application sends, not with the model name alone. Put stable instructions in a reusable system message, remove duplicated context, and retrieve only the passages required for the current task. Measure input tokens for English, Chinese, and code separately because character counts are not a reliable cost estimate.

Output limits are equally important. Ask for the smallest useful answer, validate structured output, and set a maximum completion budget. A response that explains every intermediate decision may cost more than a response that returns the required fields. Compare quality after applying the same budget instead of comparing unbounded answers.

Evaluation fixtures that resemble production

Create a versioned fixture set from real task categories: chat, extraction, code, long documents, multilingual prompts, tool arguments, and refusal boundaries. Score correctness, schema validity, and useful completion separately. A high average can hide a failure in a critical workflow, so publish a per-category threshold.

Run the fixtures against the current route and a fallback with the same prompt version. Record model, date, input tokens, output tokens, latency, parse failures, and cost. Keep the raw measurements so a later provider change can be separated from a prompt change. The model comparison explains why a single benchmark number is not enough.

Streaming and partial responses

Streaming improves perceived responsiveness, but a first token does not mean the answer completed. Track time to first token, total duration, bytes delivered, and whether a finish event was observed. The client should display an incomplete state and allow a bounded retry when a connection closes early.

For background work, non-streaming requests can be simpler to retry and audit. Choose the path per product surface and measure both with the same fixtures. Store the model identifier and gateway request ID so a provider delay can be separated from a slow renderer.

Retries, fallbacks, and idempotency

Retry only transient timeouts, connection resets, and selected 5xx responses. Use exponential backoff with jitter and a small attempt budget. Treat authentication, malformed JSON, unknown model, and quota errors as configuration or policy failures. A circuit breaker prevents an incident from multiplying traffic.

Choose a fallback by the task's minimum quality and context requirement. Record the original error, fallback model, and final status. For tool calls and other side effects, store an idempotency key before execution so a replayed response cannot create a duplicate charge or record. See the fallback patterns guide.

Cache, batching, and queue controls

Cache discounts depend on a repeated prefix that the provider recognizes. Keep stable instructions at the beginning, put request-specific material later, and record usage fields instead of assuming a cache hit. Run cold and warm evaluations separately and budget from the observed mix.

Move summaries, classification, and enrichment into a queue with a deadline, retry limit, and dead-letter record. Apply a semaphore per route and tenant, expose queue age, and pause low-priority work when a provider is degraded. Batching changes the user experience and must be measured with the same chat-completions contract.

Security and data handling

Keep keys in a secret manager, separate environment credentials, and rotate them on a schedule. Put authorization, tenant quotas, model allow-lists, and maximum token budgets before the model call. Never put a server key in browser JavaScript or a public repository.

Prompts, documents, and tool results may be sensitive. Redact by default, encrypt retained traces, expire them on a defined schedule, and expose an opaque request ID for support. Log route, model, status, tokens, and latency without making raw content visible to every operator.

Pricing verification and cost per task

Use the live pricing page and the official provider documentation on the deployment date. Keep input, output, cached input, retries, and fallbacks as separate fields. Divide spend by successful task rather than request count; a low request price is not useful if parsing failures cause repeated calls.

Publish the pricing date and model identifiers in the runbook. Re-check them before a launch and alert when token spend grows faster than request volume. The cost comparison gives the accounting vocabulary, while your own token logs are the source for product decisions.

Canary, rollback, and monthly review

Use contract tests first, then shadow traffic, then a canary for a low-risk route. Monitor status classes, schema failures, first-token latency, total duration, tokens, cost, and fallback rate. Keep the last known-good model map and base URL so rollback is a configuration change.

Once a month, review provider status, model identifiers, limits, safety behavior, prices, and documentation links. Retire routes that no longer meet the minimum threshold. A dated checklist and a small regression suite make model changes manageable instead of surprising.