LiteLLM Proxy Provider¶

LiteLLM is best known as a Python library, but its proxy is a different thing: a self-hosted AI gateway that fronts 100+ upstream providers behind one OpenAI-compatible endpoint, adding virtual keys, per-key budgets, rate limits, and central routing. Enterprises deploy it so applications never hold vendor keys — they hold a gateway key with a budget.

RoomKit talks to that gateway with LiteLLMAIProvider, a thin subclass of OpenAIAIProvider: message building, tool calling, streaming, and thinking-trace surfacing all come for free. Only the reasoning parameters and model discovery differ.

The proxy, not the SDK

pip install roomkit[litellm] installs the openai SDK, not the litellm package. RoomKit is already a provider abstraction with native, full-fidelity providers; running LiteLLM's in-process abstraction underneath it would add a second normalisation layer and a heavy dependency for nothing. The gateway keeps all of LiteLLM's value (keys, budgets, routing) on the server where it belongs.

Install¶

pip install roomkit[litellm]

Quick start¶

from roomkit import AIChannel, RoomKit
from roomkit.providers.litellm import LiteLLMAIProvider, LiteLLMConfig

provider = LiteLLMAIProvider(
    LiteLLMConfig(
        api_key="sk-...",                  # virtual key or master key
        base_url="http://localhost:4000",  # your proxy; /v1 suffix also works
        model="claude-sonnet",             # public model name from the proxy's config.yaml
    )
)

kit = RoomKit()
ai = AIChannel("ai-assistant", provider=provider, system_prompt="You are helpful.")
kit.register_channel(ai)

The model field is the public alias the proxy's operator configured (model_name in the proxy's config.yaml) — which alias routes to which upstream is the deployment's decision, not roomkit's. Both model and api_key are required: a gateway's whole point is per-key auth and budgets (for a dev proxy running without authentication, pass any placeholder key).

Listing models¶

A gateway serves whatever its operator configured, so available_models() is empty by design and context_window is None for any alias — the honest answer for a name roomkit cannot know offline. list_models() asks the proxy, which does know:

models = await provider.list_models()
for m in models:
    print(m.id, m.context_window, m.supports_vision, m.pricing)
await provider.close()

It reads LiteLLM's /model/info endpoint rather than the barer /v1/models: alongside each public name it reports the context window, vision support, and per-token costs from the proxy's own cost map, mapped into ModelInfo (context_window, supports_vision, pricing with per-million rates including cache reads/writes). History trimming and budget dashboards work against the deployment's real numbers. Load-balanced model groups (several deployments under one public name) collapse to a single model.

Two honesty rules, verified against a live proxy:

An operator-defined alias absent from LiteLLM's cost map reports nothing (None window, None vision, no pricing) rather than gaining invented metadata.
LiteLLM defaults unknown costs to 0 rather than null, so a 0/0 rate pair maps to pricing=None — a $0 price would tell a budget dashboard the route is free while the gateway may well be billing it.

Thinking / reasoning¶

LiteLLM normalises reasoning across every upstream it fronts: OpenAI's reasoning_effort is translated to Anthropic thinking budgets, Gemini thinking, DeepSeek, and the rest, and the trace comes back in reasoning_content — exactly the field RoomKit's inherited streaming reader surfaces as StreamThinkingDelta (rendered 💭 by CLIChannel(show_thinking=True)).

ai = AIChannel(
    "assistant",
    provider=LiteLLMAIProvider(LiteLLMConfig(
        api_key="sk-...",
        model="claude-sonnet",
        reasoning_effort="high",   # used when thinking_budget is None
    )),
    thinking_budget=4096,          # >0 → explicit token budget; 0 → off
)

`thinking_budget`	Request sent to the proxy
`None`	`reasoning_effort: <configured or per-turn effort>` — omitted entirely if unset (model decides)
`0`	nothing — no reasoning parameters at all, the routed model's own default applies
`> 0`	`thinking: {type: "enabled", budget_tokens: <budget>}` — explicit budget

Reasoning is omitted on tool-call turns, with one warning per provider when a setting is left out: the model behind an alias is not known, and some upstreams the gateway fronts reject it alongside function tools. Where every upstream of the deployment takes it, set supports_reasoning_effort_with_tools=True on the config and tool-call turns carry the effort or the budget as any other turn does; False omits it without the warning. Run a two-round tool loop on each route before turning it on (examples/openai_compatible_tool_effort.py runs one against any OpenAI-compatible endpoint): a route whose model needs its reasoning passed back in a form the OpenAI-shaped request does not carry fails on the second round, not the first.

Why 0 sends nothing

LiteLLM has no disable token that survives every upstream translator — verified live against 1.79.0: its Gemini mapper rejects reasoning_effort="none" with a 500, and its Anthropic mapper rejects both "none" and "disable". Any explicit spelling would break on some routes while working on others, so omission is the one portable behaviour. To force thinking off for an alias, state it where the upstream is known: per-model in the proxy's config.yaml, or via extra_body with the token that route's translator accepts (e.g. extra_body={"reasoning_effort": "disable"} for a Gemini route).

Trying it without an upstream key¶

The proxy can mock responses, so the whole path is testable with no vendor account:

# litellm-config.yaml
general_settings:
  master_key: sk-litellm-test
model_list:
  - model_name: mock-model
    litellm_params:
      model: openai/mock
      api_key: fake-upstream-key
      mock_response: "Hello from the gateway!"

uvx --from 'litellm[proxy]' litellm --config litellm-config.yaml --port 4000
LITELLM_API_KEY=sk-litellm-test python examples/litellm_ai.py --model mock-model

See examples/litellm_ai.py for an interactive chat and a --list-models mode.

How it works¶

Aspect	Behaviour
Transport	OpenAI Chat Completions API at your proxy's URL (with or without `/v1`)
Config	`LiteLLMConfig` subclasses `OpenAIConfig`, inheriting every request field (`reasoning_effort`, `include_stream_usage`, `extra_body`, …) so the two never drift
Streaming & tools	Inherited unchanged from `OpenAIAIProvider`, including `reasoning_content` thinking handling
Vision	`supports_vision` is `True`: whether a routed model reads images is the gateway's call, so images pass through and a text-only route answers with an error
Telemetry	Errors and TTFB metrics are tagged `provider="litellm"`