Stop mix mob.deploy reporting success after shipping nothing (MOB-150) - #62
Merged
Conversation
Four ways it could, all observed rather than theorised. The one that started this: `mix mob.deploy --android --native` with no `sdk.dir` printed a skip warning, built nothing, and exited 0 — because `ok_count == length(results)` is `0 == 0` for a run that produced no results. The interesting half is telling a skip that was REQUESTED from one that was INCIDENTAL. A skip means "this device is not a target for this app", which is ordinary for a phone that happens to be attached and a failure when the run asked for that platform. `requested_platforms/1` reads the raw flags rather than the resolved list, since resolution collapses "no flag" into every platform and would make every default run fatal. An adversarial review caught that the first version of this rule was much coarser than its own description. It filtered SKIPPED devices by requested platform — but `deploy_all/1` only enumerates devices for platforms in the resolved list, and that list is a subset of the requested one, so the filter was always true. The shipped rule would have been "any skip at all is fatal once you name a platform", which fails ordinary setups: two booted simulators with the app on only the one you are working on, or a spare phone plugged in. Since `--ios`/`--android` is the only way to scope a run on macOS, that is the common invocation. It is now per platform — fatal only when nothing of that platform landed — and the partial-success case is device-verified. The same review found `--json` was unusable for its only purpose. Every progress line in this task, the deployer and the native build is a plain `IO.puts/1` to `:stdio`, so the document arrived buried in ANSI prose. The run now repoints its group leader at stderr, which moves all of it — subprocess output included — and writes the document to the real stdout captured beforehand. That last part matters: `:standard_io` also resolves through the group leader, so the obvious fix sent the JSON to stderr as well and the flag emitted nothing a pipe could read. Caught by testing it rather than by reading it. The document is also emitted on the native-build failure path, which produced none. Three further fixes from the review: an auto-detected iPhone narrows Android out of the build, so `--android --ios --native` with a phone attached counted Android as unserved and failed a run that did what was asked; a named `--device` that was found and deployed nothing still exited 0; and two build skips printed no reason while the failure told the user to read one. Test gaps it found, both since closed and mutation-checked: nothing in the suite called `build_all/1`, so deleting either half of the `requested:` wiring fully restored the bug with everything green — the same shape as the `Mix.raise` gap found last time, which is why those assertions now live in one file named for what they do. And the "only the requested platform's skips count" test asserted a state `deploy_all/1` cannot produce, so it was green and proved nothing. One review finding was wrong and is recorded rather than acted on: `&opts[&1]` was called a truthiness bug for `--no-android`, but `false` is already falsy, so the negated flag was excluded correctly. The explicit `== true` is kept for clarity, and the test says so. Device-verified on a physical iPhone SE and an iOS simulator: no flags with a skip present exits 0; `--ios` with one simulator deployed and the phone skipped exits 0; `--ios --device <phone>` where the phone was skipped exits 1; `--device NOPE` exits 1; `--android --native` with `sdk.dir` removed exits 1; and `--json` parses as a single document from stdout with the prose on stderr. Refs MOB-150 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four ways
mix mob.deploycould exit 0 having shipped nothing, all observed rather than theorised. The one that started this:mix mob.deploy --android --nativewith nosdk.dirprinted a skip warning, built nothing, and exited 0 — becauseok_count == length(results)is0 == 0for a run that produced no results.The interesting half is telling a requested skip from an incidental one. A skip means "this device is not a target for this app": ordinary for a phone that happens to be attached, a failure when the run asked for that platform.
requested_platforms/1reads the raw flags rather than the resolved list, since resolution collapses "no flag" into every platform and would make every default run fatal.What the review caught
The first version of the rule was much coarser than its own description. It filtered skipped devices by requested platform — but
deploy_all/1only enumerates devices for platforms in the resolved list, and that list is a subset of the requested one, so the filter was always true. The shipped rule would have been "any skip at all is fatal once you name a platform", which fails ordinary setups: two booted simulators with the app on only the one you're working on, or a spare phone plugged in. Since--ios/--androidis the only way to scope a run on macOS, that's the common invocation. Now per platform — fatal only when nothing of that platform landed.--jsonwas unusable for its only purpose. Every progress line in this task, the deployer and the native build is a plainIO.puts/1to:stdio, so the document arrived buried in ANSI prose. The run now repoints its group leader at stderr — which moves all of it, subprocess output included — and writes the document to the real stdout captured beforehand. That last part matters::standard_ioalso resolves through the group leader, so the obvious fix sent the JSON to stderr too and the flag emitted nothing a pipe could read. Caught by testing it, not by reading it.Three more: an auto-detected iPhone narrows Android out of the build, so
--android --ios --nativewith a phone attached counted Android as unserved and failed a run that did what was asked; a named--devicethat was found and got nothing still exited 0; two build skips printed no reason while the failure told the user to read one.Test gaps, both closed and mutation-checked. Nothing in the suite called
build_all/1, so deleting either half of therequested:wiring fully restored the bug with everything green — same shape as theMix.raisegap found last time, which is why those assertions now live in one file named for what they do. And the "only the requested platform's skips count" test asserted a statedeploy_all/1cannot produce: green, proving nothing.One review finding was wrong and is recorded rather than acted on.
&opts[&1]was flagged as a truthiness bug for--no-android, butfalseis already falsy so the negated flag was excluded correctly. The explicit== trueis kept for clarity and the test says so.Device verification
Physical iPhone SE plus an iOS simulator:
--ios, one sim deployed + phone skipped--ios --device <phone>, phone skipped--device NOPE--android --native,sdk.dirremoved--json --ios2252 tests, credo and format clean. Ten mutations checked against an exact captured baseline; nine caught, the tenth being the no-op described above.
Also corrected on the ticket
Two claims I'd written into MOB-149/150 from the agent feedback and propagated without checking:
--jsonis not already on eight tasks (onlyaudit_otpandsecurity_scandeclare the switch), and output is not silent until exit when piped — measured, it streams progressively from the first sample, and 21 of 77System.cmdcalls already useinto: IO.stream(). That second item is dropped rather than implemented.