Skip to content

Extend OpenAITelemetryPlugin: Add tracking & pricing for cached tokens #1582

@waldekmastykarz

Description

@waldekmastykarz

Feature Request

Extend the OpenAITelemetryPlugin to support token tracking and pricing information for cached tokens.


Background
Currently, the OpenAITelemetryPlugin tracks and calculates token usage and associated pricing. However, many OpenAI APIs (especially when caching is present) price cached tokens differently than non-cached tokens.

Requested Enhancement

  • Update the plugin to distinguish between regular and cached tokens.
  • Track cached tokens separately and calculate/report cost accurately based on their distinct rates.
  • Factor in OpenAI pricing models that treat cached tokens differently, and expose the split and totals in pricing information/output.
  • Update relevant reporting and metrics methods to show cached token counts and costs distinctly.

Impacted Code References:

Benefits

  • Improved reporting and transparency for users with cache-aware pricing.
  • More accurate cost/control for workloads with cache/usage mix.
  • Better insights in telemetry and exported reports for complex scenarios.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions