Report real Grok token usage and list-price cost from CLI logs - #3135
Report real Grok token usage and list-price cost from CLI logs#3135olddonkey wants to merge 19 commits into
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09cf7edb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
clawsweeper
Bot
added
merge-risk: 🚨 compatibility
|
Codex review: needs maintainer review before merge. Reviewed August 25, 2026, 6:40 PM ET / 22:40 UTC. ClawSweeper reviewWhat this changesThis PR reads bounded completed-turn Grok CLI logs for usage, estimates xAI list-price cost, and publishes the freshest local result to menu and Usage & Spend views. Merge readinessThe patch is coherent and evidence-backed, but defaulting Grok to a visible non-billed list-price amount changes established display semantics and needs owner approval before merge. Priority: P2 Review scores
Verification
Live VerificationCommand: Result: FAIL (failed) — execution before step 1 Assertions:
How this fits togetherCodexBar gathers remote provider data and local CLI logs, then projects usage snapshots to its menu and Usage & Spend dashboard. This changes Grok’s local-log input, pricing calculation, and snapshot selection. flowchart LR
A[Grok CLI session logs] --> B[Bounded local scanner]
C[xAI price catalog] --> D[List-price calculation]
B --> D
D --> E[Usage snapshot publication]
F[Remote billing probe] --> E
E --> G[Menu and Usage and Spend views]
Decision needed
Why: This PR changes the user-visible meaning of Grok cost from unavailable to a non-billed USD estimate, which cannot be resolved by implementation correctness alone. Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Merge the bounded completed-turn implementation only after the owner explicitly accepts disclosed public xAI list-price estimates as the default Grok cost presentation. Do we have a high-confidence way to reproduce the issue? Not applicable: this is a provider-reporting enhancement rather than a single current-main failure report; the PR supplies exact-head runtime evidence for its new behavior. Is this the best way to solve the issue? Unclear: the implementation and disclosure are well-tested, but an owner must decide whether this estimate belongs in the default product surface. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 41d904fd6d88. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (23 earlier review cycles; latest 8 shown)
|
08360b5 to
e3cd3b9
Compare
olddonkey
changed the title
clawsweeper
Bot
added
merge-risk: 🚨 auth-provider
|
Both automated findings are addressed, plus the review's other checklist items. The inline comments were left against P1 — Preserve the Grok fallback on repeated probe failuresFixed in if provider == .grok {
if self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil {
Task { @MainActor [weak self] in
await self?.scanAndPublishGrokLocalTokenSnapshot(...)
}
}
} else if Self.tokenCostRequiresProviderSnapshot(provider) {
self.clearTokenSnapshot(for: provider)
}Regression coverage is in P2 — Refresh pricing before scanning Grok sessionsCorrect, and thank you — this was a genuine gap and not one the local tests would have surfaced. Fixed in Note the inline comment still points at Coverage: Real-session evidence
The same corpus on Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs; the two agree to the cent. Merge risk / branch stateRebased onto current One thing deliberately left undone: no |
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
Both new findings addressed at P1 — Do not map every xAI log record to the Grok subscriptionAgreed, and taken as specified rather than argued down. Routing on the prefix alone is right for the case that motivated this — traffic authenticated with the user's Grok account, which is what makes it consume SuperGrok quota — but it silently folds an API-key user's pay-as-you-go xAI spend into the subscription row. CodexBar already models the developer platform as its own The usage log carries no per-record credential evidence; I checked every field emitted for So attribution now requires positive OAuth evidence:
Coverage:
P2 — Republish the Grok snapshot after a missing catalog refreshesI looked at this closely and am deliberately not adding a republish path. Reasoning, so you can overrule it if you disagree: The refresh is fire-and-forget, so the scan that requests it returns whatever the cache currently holds — that part is accurate. But the parse cache stores parsed turns, not prices, so aggregation and pricing re-run on every The alternative — plumbing a completion signal back across the actor boundary into the Happy to build it if you would rather have it. EvidenceThe attribution itself only becomes visible in the app: The two halves stay distinguishable in the UI, which makes the attribution legible rather than something you have to take on trust: the CLI goes through the responses API so its SKU is
Independently recomputing the same corpus agrees to the cent on both halves: 54,121,501 tokens / $50.52 for the CLI logs, and $166.25 across 1,520 OpenCodex The negative direction — API-key traffic staying off the Grok row — is covered by tests rather than a screenshot, since demonstrating it live would mean rewriting the machine's OpenCodex config. StateFull suite on 
|
clawsweeper
Bot
added
proof: 📸 screenshot
olddonkey
changed the title
|
Addressed both current findings in
Validation on the exact pushed head:
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
merge-risk: 🚨 availability
clawsweeper
Bot
added
rating: 🦪 silver shellfish
|
Addressed the two latest findings on exact head
Exact-head validation:
The PR body now contains the exact-head deterministic dashboard proof and still distinguishes the earlier native-session proof from the final head. No credentialed live-account probe was run for this repair. Exact-head CI is running. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
rating: 🦐 gold shrimp
|
Final exact-head status for
No further code comments are outstanding. Remaining external gates are the requested exact-head native-session proof, owner approval of the default non-billed list-price estimate semantics, and maintainer re-review of the older |
|
Added final-head native-session evidence for exact SHA Redacted real-corpus transcript from the opt-in production-path proof: All 3 tests passed. This route reads the installed native Grok session corpus and local models.dev catalog through the shipped bounded scanner/snapshot/spend path; it does not inject temporary session fixtures. The same suite verifies the populated-surface disclosure I also packaged the exact tree with Evidence boundary: packaged The remaining owner decision about default estimate semantics is unchanged. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
the
proof: sufficient
…on logs two ways and expensive in a third. Wrong tokens: the scanner summed `contextTokensUsed` from `signals.json`, which is the session's ENDING context-window occupancy, not what it consumed. On a real machine that reported 653K where actual consumption was 48.0M. Read the sibling `updates.jsonl` instead, where every `turn_completed` event carries the turn's real usage, and bucket by the per-line timestamp so a session crossing local midnight lands in both days. No cost: `toCostUsageTokenSnapshot` hardcoded nil dollars, and nothing could have priced a Grok model anyway because `codexModelsDevProviderIDs` had no `xai`. Add it, and resolve `grok-<version>-build` onto its base catalog model — the `-build` suffix is an artifact of the responses-API surface, not a separate SKU. `grok-build-0.1` is a real model and is never rewritten. Cost is the public xAI card via models.dev, provenance `.listPriceEstimate`, so Grok stays comparable with Claude and Codex. grok's own `costUsdTicks` is deliberately not used for display. A turn's `usage` is the aggregate of `modelCalls` API calls, so tiering on the turn total would push nearly every multi-call turn into the >=200k bracket. Price on the per-call average instead, in closed form over the two synthetic call groups. This under-tiers slightly when context grows within a turn (measured ~4% below the vendor's own accounting on a 27-turn sample, against ~+10% for aggregate tiering); the trade is documented at the call site and pinned by a test. Main-actor cost: the scan ran synchronously inside `@MainActor UsageStore` on every menu-card build, refresh and dashboard load. It now reads the projection the async probe already produced, and the remaining fallback scans on a detached task with one scan in flight at a time. The probe projects the maximum window and consumers narrow it, so `costUsageHistoryDays` and the dashboard's 365-day request are both honoured. Hardening: `modelCalls` comes from a file, so it is validated before it can size any work; parsing is cached per (path, size, mtime) with entries evicted when a file is no longer visited; the cache lock is not held across file reads. Note for upgraders: adding `xai` to `codexModelsDevProviderIDs` changes the Codex pricing-cache key, so the first launch after this re-prices existing Codex history once. Same one-time cost as when kimi and deepseek were added.
OpenCodex sends inference straight to api.x.ai using the Grok account's OAuth credentials, so it burns the same SuperGrok subscription the Grok provider reports on. It only spawns the `grok` binary to refresh tokens, so those requests never reach ~/.grok/sessions and the local session scanner cannot see them — 1,435 requests on one real machine that CodexBar attributed to nothing. Route the `xai` provider prefix to the Grok subscription, the same way `openai` already routes to Codex. Like that mapping, this routes on the prefix and does not distinguish OAuth from API-key traffic. The `-build` suffix seen in the data is a responses-API protocol artifact, not a separate billing pool, so traffic is not split by it. Routing alone would have produced tokens with no dollars. The aggregator priced the bare `entry.model`, and a name without a route prefix is resolved against the `openai` provider — which is why `gpt-5.6-sol` prices today and `grok-4.6` resolved to `openai/grok-4.6` and missed. Qualify an unprefixed model with its provider before pricing. Codex rows are unaffected (the qualified name resolves to the same target), and providers outside the supported set keep returning nil.
Grok resolved list prices straight out of the cached models.dev catalog, but nothing in its path ever fetched that catalog. The only fetch trigger is CostUsageFetcher.refreshPricingIfAllowed, which is gated to Codex and Claude — and Grok never reaches it at all, because its snapshot comes from the provider probe rather than the shared token-cost pipeline. On a machine where Codex or Claude is also enabled the cache is already there, so this is invisible. Enable only Grok and the file never appears: every price lookup returns nil and the Cost row shows tokens with no money, permanently. Request ModelsDevPricingPipeline.refreshIfNeeded from the Grok scan paths. It is safe to call repeatedly — it returns immediately unless the cache is stale and serialises through its own coordinator — and it is detached rather than awaited, matching how the Codex and Claude paths already treat it: pricing availability must never delay or fail a local scan, and the next refresh fills in the value. `summarize` stays synchronous and side-effect free; the refresh lives in a wrapper so the parse-cache behaviour and existing tests are untouched. Reported as P2 by the automated review on the pull request.
… proof The opt-in live proof scanned real sessions but printed tokens only, which cannot evidence the half of this change that is about money. It now also reports today's and the window's list-price cost, the provenance, the window actually used, and how many days carried a price versus tokens — so an all-unpriced result is visible in the output instead of reading as zero. Still skipped unless CODEXBAR_LIVE_GROK_CATALOG_PROOF=1.
The regression guard drove a single failing refresh after a local publication existed. The defect it covers is specifically about the *second* failure: the first one publishes through the fallback scan, and only the next one arrives with a publication already in place — which is what used to hit the generic clear branch. Drive the failure twice and assert the row and the scan count both hold.
Routing every OpenCodex `xai` record to the Grok subscription is right for the case that motivated it — traffic authenticated with the user's Grok account, which is what makes it burn the SuperGrok quota. It is wrong for anyone using an xAI API key: their pay-as-you-go developer-platform spend gets folded into the subscription row, silently inflating it. CodexBar models that platform as its own xAI provider precisely to keep the two apart. The usage log carries no per-record credential evidence, so the decision has to come from the OpenCodex provider config, which records `authMode` per provider. Read it, and attribute to Grok only when that mode is OAuth; anything else is token-only spend that belongs to no tracked subscription. Fail closed: a missing or malformed config, no `xai` entry, or an absent `authMode` all count as no OAuth evidence and keep the records off the Grok row. The dispatcher stays a pure function — the set of OAuth-backed provider ids is threaded in from the caller rather than read at the routing site — and the gate applies only to `xai`, leaving the other routes exactly as they were. Also records why the Grok pricing refresh stays fire-and-forget: the parse cache holds parsed turns rather than prices, so the next scan reprices against the refreshed catalog, and plumbing completion back to republish was judged disproportionate to a delay Codex and Claude already share. Raised as P1 by the automated review; the owner chose verifiable attribution over prefix-only routing.
|
Rebased PR #3135 onto current upstream
Exact-head validation:
The PR body now carries the rebased exact-head evidence. ClawSweeper is already reviewing |
2 participants
Problem
#3085 enabled Grok token cost, but the local session projection was not measuring actual consumption:
signals.jsonexposes ending context-window occupancy, not per-turn token usage. On a real corpus it reported 653K tokens where the completed turns contained 54.1M.nil, and the models.dev resolver did not include the xAI catalog.updates.jsonlfiles could not safely remain on@MainActor.What this changes
Read the completed-turn usage that the Grok CLI actually records
GrokLocalSessionScannerreadsturn_completedevents from each session'supdates.jsonl, matches bothsession/updateand_x.ai/session/update, and buckets each line by its own timestamp. It preserves raw aggregate token totals and uses the recordedmodelCallscount only to approximate per-call tiered list pricing in O(1).The parser now streams through the shared chunked JSONL reader instead of loading whole files. Production bounds are explicit: a 64 MiB tail per file, 1 MiB per record, 20,000 retained turns per file, and global scan budgets of 256 recent sessions, 256 MiB, and 100,000 turns. The process-wide LRU cache retains at most 64 files or 50,000 turns. Cancellation is checked between chunks, I/O/cancelled results are not cached, and any truncation marks history incomplete rather than presenting a partial total as complete.
Price the Grok models and preserve provenance
The models.dev xAI catalog is now eligible for cost pricing. Responses-API names such as
grok-4.6-buildresolve to the basegrok-4.6SKU after exact lookup, while real independent names such asgrok-build-0.1remain untouched. Cost is published as.listPriceEstimate; Grok's internalcostUsdTicksis not presented as billed spend.On a stale catalog, the scan prices immediately and refreshes in the background. On a fresh install with no catalog artifact, the first scan now awaits the initial best-effort refresh attempt before creating the snapshot, so a successful first refresh is visible in the first publication. A refresh failure still degrades to token-only data rather than failing the local scan.
Keep scanning and publication off the main actor
The provider projection consumes the async probe's snapshot. Remaining fallback paths scan on a detached utility task with a single scan in flight. A maximum-window snapshot is narrowed by each consumer through
CostUsageTokenSnapshot.narrowed(toHistoryDays:calendar:), including the spend dashboard's 365-day request.Keep OpenCodex xAI history out of the Grok subscription row
OpenCodex
usage.jsonlrecords provider/model usage but does not retain the credential mode used for each request. Reading today'sconfig.jsoncannot distinguish older API-key traffic from older OAuth traffic after a configuration switch.For that reason, xAI entries now remain
.tokenOnlyand are not merged into the Grok subscription row until the producer records request-time credential provenance. The current-config reader and its routing parameter were removed, with dispatcher/fan-out regressions and updated documentation. Other existing OpenCodex subscription routes are unchanged.Review findings addressed
P1 — preserve record-time xAI credential attribution: resolved by pausing OpenCodex xAI-to-Grok attribution when record-time evidence is unavailable. Historical traffic can no longer move between provider rows when the current auth config changes.
P2 — publish pricing after the first catalog refresh: resolved by awaiting the initial refresh only when no cached catalog exists, with a regression proving that a successful refresh prices the first returned summary. Stale catalogs still use the non-blocking refresh path.
P2 — bound growing session logs: resolved at
bdfc10e0dwith chunked tail reads, per-file and global byte/turn/session budgets, bounded LRU retention, cancellation checks, and incomplete-history propagation. Regressions cover tail truncation, global scan budgets, and cache eviction limits.P3 — correct the documented session source: the earlier
signals.json/30-day description now documentsupdates.jsonl, the requested window up to 365 days, list-price provenance, and every production bound.signals.jsonis metadata-only and also byte-limited.The earlier repeated-probe failure was also fixed: an existing Grok local publication survives consecutive remote refresh failures instead of falling through to the generic clear path.
P1 — restore the published fallback for Grok menu consumers: resolved on exact head
2f80ae834with a live-consumer-only Grok projection fallback. Menu cards and cost history now consume the current-config local publication when the remote snapshot is absent, while override cards retain the original isolated projection and cannot inherit provider-level data. Regressions cover both the visible fallback and override isolation.P1 — rescan after an empty Grok publication: resolved on exact head
c85067dda. A current-config publication with no snapshot is now treated as no usable fallback, so a later remote failure rescans newly written local turns. Once a nonempty snapshot is published, subsequent failures reuse it without redundant scans. The empty-to-nonempty transition is covered by the scanner regression.P1 — keep bare OpenCodex xAI history token-only: standalone aggregation no longer assigns list-price dollars when request-time credential provenance is unavailable.
P1 — label populated Grok cost surfaces: menu details, charts, dashboard rows, and spend views identify the amount as a public xAI list-price estimate rather than a bill.
P1 — keep failed-refresh local totals advancing: every failed Grok remote billing refresh schedules the bounded local scan; the existing single-flight task coalesces concurrent scans.
P1 — isolate xAI pricing invalidation: xAI now has a separate pricing fingerprint, so xAI-only catalog changes do not invalidate Codex caches.
P2 — bound discovery I/O: Grok session discovery stops at 4,096 tree entries and marks history incomplete when that bound is reached.
P2 — refresh after preservable network failures: resolved on exact head
c4c4000e0. A timeout can retain the last remote provider snapshot while still rescanning local sessions; live Grok consumers compare timestamps and select the newer current-config local publication. Empty local scans leave the retained remote projection in place, and account/context override cards remain isolated from provider-level live data.P1 — keep Usage & Spend on the freshest Grok data: resolved on exact head
fcdffd107. Dashboard capture and retained-publication paths now use the same timestamp selector as live menu consumers; a newer 100-token local publication wins over a preserved 77-token remote snapshot.P3 — leave release notes to the release flow: the normal-PR
CHANGELOG.mdentry was removed; the release-note context remains in this PR body.Latest-main rebase integration: rebased onto main
41d904fd6, retained main’s off-main Grok scanning path, migrated its projection regression fromsignals.jsonmetadata to completed-turn JSONL, and adopted the previous main parser hash without rebuilding compatible Codex rows.Real-session evidence
The opt-in proof was rerun on exact head
fcdffd1077147dc538ddd1412f37fac7336acacfagainst the native Grok CLI session corpus through the shipped bounded scanner, snapshot projection, models.dev xAI pricing, and spend-catalog path:All 3 tests in the suite passed. This opt-in route reads the real local
~/.grok/sessions/**/updates.jsonlcorpus and the local models.dev catalog rather than temporary session fixtures; its transcript prints aggregate fields only. The current corpus has no completed-turn tokens today and one priced day in the last 30 days, so these values naturally change as local logs age. The populated-surface assertion in the same suite also verifies the visiblePublic xAI list-price estimate · not a bill.disclosure.The same exact tree was then packaged with
./Scripts/package_app.sh. The production build, widget packaging, code-sign validation, helper-resource smoke, symlinked-helper smoke, and app-launch smoke passed. The packaged helper reportsCodexBar 0.55.0,config validatereturnsConfig: OK, and the running process is/Users/olddonkey/Documents/CodexBar-pr3135/CodexBar.app/Contents/MacOS/CodexBar.Evidence boundary: the public packaged
usageJSON intentionally does not encodeUsageSnapshot.costUsage, and a forced--provider grok --source clicall in this environment returned a provider error. I am therefore not presenting that command as spend proof. The package result establishes exact-tree production build/sign/launch health; the redacted transcript above establishes final-head native-session token/pricing behavior. No credential values, account identity, session contents, or interactive Keychain/browser-cookie reads were emitted.Exact-head deterministic behavior proof
Exact head
fcdffd1077147dc538ddd1412f37fac7336acacfdrives the production bounded JSONL scanner, publication path, failure preservation, timestamp selection, menu consumers, and spend-dashboard capture through temporary on-disk Grok session files:The regression first writes a completed 77-token turn and installs its projection as the retained remote snapshot. It then appends a second completed 23-token turn, injects
URLError.timedOut, verifies the remote snapshot is preserved at 77, verifies the local publication advances to 100, and verifies the live consumer selects the newer 100-token snapshot. A second failed refresh proves that local logs are rescanned again. The status-menu regression separately proves that newer local tokens beat stale remote tokens in both the visible card and cost-history submenu, while the override-card isolation regression remains green. The dashboard regression independently installs a one-minute-older 77-token remote snapshot and a current 100-token local publication, then verifies capture-only Usage & Spend selects the local 100-token, $1 list-price snapshot and its newer timestamp.This is exact-head deterministic production-path evidence using real JSONL files in a temporary Grok home. The separate native-session aggregate above was rerun on the same exact final head against the installed Grok CLI corpus; the two proofs are intentionally distinguished.
Testing
fcdffd1077147dc538ddd1412f37fac7336acacf:make checkpassed — parser hash678fd59821eccb04, provider/package/documentation gates, SwiftFormat (0/2002), and SwiftLint (0 violations in 2001 files).fcdffd1077147dc538ddd1412f37fac7336acacf:make testpassed — 933 selections, 78/78 groups successful on the first pass, 0 failed groups, 0 retries, and 0 timeouts (648.5 seconds).SpendDashboardGrokFreshnessTests(1/1),SpendDashboardControllerTests(25/25), andProviderArchitectureGatekeeperTests(38/38); the preservable-timeout and stale-remote menu regressions also remain in the fully green suite.main41d904fd6d884b7df73ac180a913ccbf04a9d7d6, 0 behind / 19 ahead, andgit diff --checkis clean. GitHub reportsMERGEABLE; all exact-head CI checks passed.3b642530a: focused OpenCodex/Grok suites (35 tests), production bundle/signing/launch smoke, packagedCodexBarCLI config validate, packaged Grok usage throughgrok-cli-proxy, and the authorized two-test live scanner/pricing/catalog proof passed.Normal automated tests use temporary homes, injected catalogs/transports, and no live credentials or interactive Keychain reads. The earlier explicitly authorized live checks used the configured Grok provider and local session logs; no credential values were printed.
Maintainer decision
The owner-requested code findings are addressed on exact head
fcdffd107, including preserved-timeout freshness in both menu and Usage & Spend consumers and removal of the release-owned changelog entry. The branch is rebased and locally green. All exact-head CI checks passed; ClawSweeper re-review is in progress, and the final-head native-session proof is recorded above. Owner approval of the default estimate semantics and maintainer re-review remain the external gates. The xAI amount remains clearly labeled as a non-billed public list-price estimate, distinct from SuperGrok subscription credits.Release-note context is retained in this PR body; the normal-PR changelog entry was removed per repository release ownership.