Skip to content

test: fix stale/flaky account-identity key assertion#162

Open
vigneshsubbiah16 wants to merge 1 commit into
stagingfrom
fix/account-identity-test-keys
Open

test: fix stale/flaky account-identity key assertion#162
vigneshsubbiah16 wants to merge 1 commit into
stagingfrom
fix/account-identity-test-keys

Conversation

@vigneshsubbiah16

@vigneshsubbiah16 vigneshsubbiah16 commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

build_account_identity now intentionally returns user_email (and env-dependent device_serial), so test_keys_limited_to_identity_fields failed its exact-set assertion on any machine with a device serial — staging's hook suite was red. Switches to a subset assertion: no key outside the known identity fields may leak, without coupling to the optional fields. 25/25 identity tests pass.

Greptile Summary

This PR fixes a flaky test in test_identity.py where test_keys_limited_to_identity_fields was asserting an exact key set that excluded user_email and device_serial, both of which build_account_identity legitimately returns. The fix replaces the exact-equality assertion with a subset check against the full superset of known identity fields.

  • Root cause addressed correctly: read_account_identity() always includes user_email in its output dict, and build_account_identity conditionally appends device_serial \u2014 the old assertEqual was simply wrong about what the function returned.
  • Scope is narrow and safe: only the assertion strategy changes; no production code is touched, and the remaining 24 tests are unaffected.

Confidence Score: 5/5

Safe to merge — only a test assertion strategy changes, no production code is touched.

The change is a single-line test fix replacing an exact-set equality with a subset check. The production function build_account_identity is unchanged, the new assertion correctly reflects the function's actual output contract, and all 25 identity tests pass.

No files require special attention.

Important Files Changed

Filename Overview
claude-code/hooks/test_identity.py Switches test_keys_limited_to_identity_fields from exact-set equality to subset assertion, adding user_email and device_serial to the known-fields superset to match what build_account_identity actually returns.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[test_keys_limited_to_identity_fields] --> B[_write_config empty dict]
    B --> C[build_account_identity]
    C --> D[read_account_identity]
    D --> E["Returns: org_id, plan, auth_mode, user_email, email_domain"]
    C --> F{device_serial available?}
    F -->|Yes| G["Adds: device_serial"]
    F -->|No| H["Omitted"]
    E --> I[result dict]
    G --> I
    H --> I
    I --> J{Old test: assertEqual set == exact 4 keys}
    I --> K{New test: assertLessEqual set ⊆ 6-key superset}
    J -->|FAIL if user_email or device_serial present| L[❌ Flaky on staging]
    K -->|Passes regardless of optional fields| M[✅ Stable]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[test_keys_limited_to_identity_fields] --> B[_write_config empty dict]
    B --> C[build_account_identity]
    C --> D[read_account_identity]
    D --> E["Returns: org_id, plan, auth_mode, user_email, email_domain"]
    C --> F{device_serial available?}
    F -->|Yes| G["Adds: device_serial"]
    F -->|No| H["Omitted"]
    E --> I[result dict]
    G --> I
    H --> I
    I --> J{Old test: assertEqual set == exact 4 keys}
    I --> K{New test: assertLessEqual set ⊆ 6-key superset}
    J -->|FAIL if user_email or device_serial present| L[❌ Flaky on staging]
    K -->|Passes regardless of optional fields| M[✅ Stable]
Loading

Reviews (1): Last reviewed commit: "test: fix stale account-identity key ass..." | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

build_account_identity now intentionally carries user_email (read_account_identity
pulls it) and, env-dependent, device_serial. test_keys_limited_to_identity_fields
asserted an exact 4-key set, so it failed on any machine with a device serial and
was inherently flaky. Assert the keys are a SUBSET of the known identity fields
instead — preserves the "limited to identity fields" intent (no leak of a non-
identity key) without coupling to optional fields.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vigneshsubbiah16 vigneshsubbiah16 requested a review from a team June 17, 2026 21:45
Comment on lines +239 to 243
self.assertLessEqual(
set(result.keys()),
{"org_id", "plan", "auth_mode", "email_domain",
"user_email", "device_serial"},
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Subset assertion is correct but has no lower-bound check

assertLessEqual only verifies that no unexpected key leaks out; it doesn't assert that the core required fields (org_id, plan, auth_mode, email_domain) are always present in the result. If build_account_identity were changed to drop one of those keys silently, this test would still pass. The parallel test test_returns_full_identity does check specific keys, but it relies on a non-empty oauth config — this empty-config variant exercises the fallback path with all-None values and would benefit from also asserting the required keys are at least present as keys (even if their values are None).

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant