LiteLLM vs Portkey: routing, budgets, and deployment costs
Compare LiteLLM Proxy and Portkey for model routing, retries, streaming failover, team budgets, and self-hosted versus managed deployment.
A model gateway earns its place when a second provider arrives. The first application can call its provider's SDK directly. Once several teams share models and keys, somebody has to decide what happens when a provider returns 429, who can use the backup model, and where the cost of the extra attempts appears.
Start with LiteLLM Proxy if your team can run the gateway and already has a way to manage access and budgets. If the work you need to avoid is building a shared model catalog, workspace provisioning, and administrative workflows, look at the managed Portkey platform. The open-source Portkey Gateway is a separate candidate when the job is routing and you want to run it yourself.
Guance Editorial Team · Updated November 10, 2026. This comparison is based on the vendors' documentation. Portkey's commercial product now carries the Prisma AIRS AI Gateway name. The open-source project is still Portkey Gateway. Portkey product scope
What you run and what you buy
| Requirement | LiteLLM | Portkey |
|---|---|---|
| Run your own gateway | The open-source Proxy exposes a gateway API and uses Router for deployment selection. | The open-source Portkey Gateway can run independently. |
| Handle an upstream failure | Retry within a model group or fall back to another group according to policy. | Set retry conditions and a fallback strategy with its own target order and status-code conditions. |
| Give teams separate access and limits | With PostgreSQL connected, configure model access, budgets, and rate limits through Virtual Keys and the relevant account scope. | In the commercial platform, use an Integration and Workspace Provisioning to grant provider access and configure limits. |
| Add enterprise controls | The enterprise license adds controls such as model-specific budgets per Virtual Key. | Compare the managed service with enterprise hybrid, where the data plane runs in your environment and Portkey manages the control plane. |
A self-hosted gateway still needs upgrades, a state store where applicable, and someone to respond when it fails. A hybrid deployment shifts part of that work while leaving a data plane to run. Portkey’s current documentation says it no longer provisions new fully air-gapped deployments. Check where configuration, logs, and request data go before treating any two deployment options as equivalent. LiteLLM Enterprise · LiteLLM Virtual Keys · Portkey Gateway · Portkey deployment options
Count the attempts before tuning the router
Say the application retries twice and the gateway also retries twice. If all failures qualify, each outer attempt starts a fresh gateway cycle, and each inner attempt makes one upstream call, that is 3 × 3 = up to 9 primary-path calls. A deadline or cooldown can cut it short. A backup target or another SDK retry layer can raise the total. The arithmetic describes this example, not a product default.
LiteLLM separates retries inside a model group from fallback to another group and considers the failure type. Portkey can restrict the two independently with retry.on_status_codes and strategy.on_status_codes. With a fallback strategy configured and no status override, Portkey documents non-2xx responses as triggers. Automatic retries have their own default set of codes. A 429 can justify backing off or switching deployments, but sending the same malformed request to another equivalent deployment usually repeats the error. LiteLLM reliability · Portkey retries · Portkey fallbacks
Work backward from the application's deadline. Connect time, time to first token, and the total response window are separate measurements. If a user has ten seconds to wait, three targets cannot each have a full ten-second budget.
A successful fallback can still fail the task
A second deployment of the same model mainly changes capacity, credentials, or location. A different model can also change context limits, tool schemas, or structured output. An order lookup response with a missing field can return HTTP 200 and still be useless to the application. Check the exact model, endpoint, and tool contract before routing production traffic to it.
The boundary moves once content has reached the client. LiteLLM documents an opt-in continuation for eligible plain-text Chat Completions streams on a fallback deployment that supports assistant prefill. It excludes streams that have already emitted tool calls or constrained output such as JSON mode. Portkey's status-code fallback configuration describes switching targets. Verify how the client handles a broken stream before assuming it can resume one. For a write action, assemble and validate the complete tool arguments, then pass an idempotency identifier that the executor actually honors. LiteLLM streaming and fallback · Portkey fallback conditions
Set budgets where you can reconcile them
Requests per minute bound call rate. Tokens per minute bound usage rate. A spend budget depends on the prices and usage the gateway records. LiteLLM's spend budgets and Virtual Keys require a database. Portkey documents cost- or token-based budgets and request- or token-based rate limits when provisioning an Integration to a workspace. Check the scope and reset behavior before several teams share a provider credential. LiteLLM budgets · Portkey Integrations
For a cost comparison, put billable provider attempts, cached-token pricing where applicable, gateway fees, databases or caches, and operational work on the same sheet. A failed attempt can still incur a provider charge. Requests already in flight and differences between the gateway's price map and the provider invoice can leave a gap even when a budget is configured.
A gateway can report a completed model call while the user is still waiting for a tool. With supported Agent or LLM instrumentation reporting the model, token usage, and trace context, Guance can show the model span alongside the tool call and instrumented downstream service. Use those spans to investigate where the time went. Routing and budget enforcement remain gateway decisions. Guance Agent Monitoring
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
