Track cached prompt tokens through to Langfuse

OpenAI's chat completions response includes prompt_tokens_details.
cached_tokens when prompt caching kicks in - previously dropped
entirely. Now extracted in LlmClient and forwarded to Langfuse as the
flat cache_read_input_tokens usage key.

Used the flat key rather than OpenAI's nested prompt_tokens_details
shape: the installed @langfuse/tracing SDK's usageDetails type is
Record<string, number> | OpenAiUsage (a 3-field camelCase shape, no
_details nesting) even though the wire schema in @langfuse/core
supports nesting - a gap in the SDK's public types, not something
worth fighting with a cast.

Verified: full suite (82/82) passing, tsc/eslint clean.
This commit is contained in:
2026-08-12 23:15:42 +02:00
parent 6f0300f038
commit df4ca0bf76
5 changed files with 65 additions and 2 deletions

View File

@@ -32,6 +32,28 @@ describe('LlmClient', () => {
});
});
it('extracts cachedTokens from prompt_tokens_details when present', async () => {
createMock.mockResolvedValue({
choices: [{ message: { content: 'hello' } }],
usage: {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 120,
prompt_tokens_details: { cached_tokens: 80 },
},
});
const client = new LlmClient({ baseUrl: 'http://localhost:11434/v1' });
const result = await client.chat([{ role: 'user', content: 'hi' }], { model: 'gpt-4o-mini' });
expect(result.usage).toEqual({
promptTokens: 100,
completionTokens: 20,
totalTokens: 120,
cachedTokens: 80,
});
});
it('throws when the response has no choices', async () => {
createMock.mockResolvedValue({ choices: [] });