test: fix stale/flaky account-identity key assertion#162
test: fix stale/flaky account-identity key assertion#162vigneshsubbiah16 wants to merge 1 commit into
Conversation
build_account_identity now intentionally carries user_email (read_account_identity pulls it) and, env-dependent, device_serial. test_keys_limited_to_identity_fields asserted an exact 4-key set, so it failed on any machine with a device serial and was inherently flaky. Assert the keys are a SUBSET of the known identity fields instead — preserves the "limited to identity fields" intent (no leak of a non- identity key) without coupling to optional fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| self.assertLessEqual( | ||
| set(result.keys()), | ||
| {"org_id", "plan", "auth_mode", "email_domain", | ||
| "user_email", "device_serial"}, | ||
| ) |
There was a problem hiding this comment.
Subset assertion is correct but has no lower-bound check
assertLessEqual only verifies that no unexpected key leaks out; it doesn't assert that the core required fields (org_id, plan, auth_mode, email_domain) are always present in the result. If build_account_identity were changed to drop one of those keys silently, this test would still pass. The parallel test test_returns_full_identity does check specific keys, but it relies on a non-empty oauth config — this empty-config variant exercises the fallback path with all-None values and would benefit from also asserting the required keys are at least present as keys (even if their values are None).
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
build_account_identity now intentionally returns user_email (and env-dependent device_serial), so test_keys_limited_to_identity_fields failed its exact-set assertion on any machine with a device serial — staging's hook suite was red. Switches to a subset assertion: no key outside the known identity fields may leak, without coupling to the optional fields. 25/25 identity tests pass.
Greptile Summary
This PR fixes a flaky test in
test_identity.pywheretest_keys_limited_to_identity_fieldswas asserting an exact key set that excludeduser_emailanddevice_serial, both of whichbuild_account_identitylegitimately returns. The fix replaces the exact-equality assertion with a subset check against the full superset of known identity fields.read_account_identity()always includesuser_emailin its output dict, andbuild_account_identityconditionally appendsdevice_serial\u2014 the oldassertEqualwas simply wrong about what the function returned.Confidence Score: 5/5
Safe to merge — only a test assertion strategy changes, no production code is touched.
The change is a single-line test fix replacing an exact-set equality with a subset check. The production function build_account_identity is unchanged, the new assertion correctly reflects the function's actual output contract, and all 25 identity tests pass.
No files require special attention.
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[test_keys_limited_to_identity_fields] --> B[_write_config empty dict] B --> C[build_account_identity] C --> D[read_account_identity] D --> E["Returns: org_id, plan, auth_mode, user_email, email_domain"] C --> F{device_serial available?} F -->|Yes| G["Adds: device_serial"] F -->|No| H["Omitted"] E --> I[result dict] G --> I H --> I I --> J{Old test: assertEqual set == exact 4 keys} I --> K{New test: assertLessEqual set ⊆ 6-key superset} J -->|FAIL if user_email or device_serial present| L[❌ Flaky on staging] K -->|Passes regardless of optional fields| M[✅ Stable]%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% flowchart TD A[test_keys_limited_to_identity_fields] --> B[_write_config empty dict] B --> C[build_account_identity] C --> D[read_account_identity] D --> E["Returns: org_id, plan, auth_mode, user_email, email_domain"] C --> F{device_serial available?} F -->|Yes| G["Adds: device_serial"] F -->|No| H["Omitted"] E --> I[result dict] G --> I H --> I I --> J{Old test: assertEqual set == exact 4 keys} I --> K{New test: assertLessEqual set ⊆ 6-key superset} J -->|FAIL if user_email or device_serial present| L[❌ Flaky on staging] K -->|Passes regardless of optional fields| M[✅ Stable]Reviews (1): Last reviewed commit: "test: fix stale account-identity key ass..." | Re-trigger Greptile