Skip to content

fix: scope router metadata to the request that produced it - #2582

Open
revanth-045 wants to merge 1 commit into
huggingface:mainfrom
revanth-045:fix/per-request-router-metadata
Open

revanth-045 wants to merge 1 commit into
huggingface:mainfrom
revanth-045:fix/per-request-router-metadata

Conversation

@revanth-045

Copy link
Copy Markdown
Contributor

Problem

endpointOai creates one OpenAI client per model and stores the router metadata read from response headers (X-Router-Route, X-Router-Model, x-inference-provider) in a single closure variable. Every request through that model shares it, and each request reads it back when its own stream ends (openAIChatToTextGenerationStream / openAIChatToTextGenerationSingle). Two consequences:

  • Overlapping requests: a generation that finishes later reports the provider/model from whichever response arrived most recently — possibly another user's — so the provider badge on the message is wrong.
  • Stale metadata: a response with no router headers replays the previous response's metadata, because the variable is never reset.

Fix

Build the SDK client per request, so the headers captured by the custom fetch are scoped to the request that produced them. Constructing the client does no I/O and nothing is pooled on it, so there is no cost to creating it per call. The three call sites (completions, streaming and non-streaming chat_completions) now take openai / getRouterMetadata from the per-request client.

Tests

New endpointOai.spec.ts with two cases that drive the real SDK against a stubbed fetch:

  • two interleaved requests whose headers arrive in one order and whose bodies finish in the other — each must report its own provider;
  • a header-less response following one with headers must report no provider.

Both fail on main (expected 'beta' to be 'alpha', expected 'alpha' to be undefined) and pass with this change.

npm test (1418 tests), npm run check and npm run lint are clean.

🤖 Generated with Claude Code

`endpointOai` built one OpenAI client per model and captured the router
headers (`X-Router-Route`, `X-Router-Model`, `x-inference-provider`) into
a single closure variable shared by every request on that model. Each
request read it back at the end of its own stream, so:

- with overlapping requests, a generation that finished later reported
  whichever provider/model the most recent response — anyone's — carried;
- a response with no router headers replayed the previous response's
  metadata, since the variable was never reset.

Create the client per request instead, so the captured headers belong to
that request alone. Constructing the SDK client does no I/O and nothing is
pooled on it. Regression tests cover both cases.

Co-Authored-By: Claude Opus 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant