fix: scope router metadata to the request that produced it - #2582
Open
revanth-045 wants to merge 1 commit into
Open
revanth-045 wants to merge 1 commit into
revanth-045 wants to merge 1 commit into
Conversation
`endpointOai` built one OpenAI client per model and captured the router headers (`X-Router-Route`, `X-Router-Model`, `x-inference-provider`) into a single closure variable shared by every request on that model. Each request read it back at the end of its own stream, so: - with overlapping requests, a generation that finished later reported whichever provider/model the most recent response — anyone's — carried; - a response with no router headers replayed the previous response's metadata, since the variable was never reset. Create the client per request instead, so the captured headers belong to that request alone. Constructing the SDK client does no I/O and nothing is pooled on it. Regression tests cover both cases. Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
endpointOaicreates oneOpenAIclient per model and stores the router metadata read from response headers (X-Router-Route,X-Router-Model,x-inference-provider) in a single closure variable. Every request through that model shares it, and each request reads it back when its own stream ends (openAIChatToTextGenerationStream/openAIChatToTextGenerationSingle). Two consequences:Fix
Build the SDK client per request, so the headers captured by the custom
fetchare scoped to the request that produced them. Constructing the client does no I/O and nothing is pooled on it, so there is no cost to creating it per call. The three call sites (completions, streaming and non-streamingchat_completions) now takeopenai/getRouterMetadatafrom the per-request client.Tests
New
endpointOai.spec.tswith two cases that drive the real SDK against a stubbedfetch:Both fail on
main(expected 'beta' to be 'alpha',expected 'alpha' to be undefined) and pass with this change.npm test(1418 tests),npm run checkandnpm run lintare clean.🤖 Generated with Claude Code