Add MAUI iOS Inner Loop measurements for CI#5187
Draft
davidnguyen-tech wants to merge 114 commits into
Draft
Conversation
- Create iOSInnerLoopParser.cs: binlog parser for iOS inner loop build timings, extracting iOS-specific tasks (AOTCompile, Codesign, MTouch, etc.) and targets (_AOTCompile, _CodesignAppBundle, _CreateAppBundle, etc.) plus shared tasks (Csc, XamlC, LinkAssembliesNoShrink) - Wire into Startup.cs: add iOSInnerLoop to MetricType enum and map it to iOSInnerLoopParser in the parser switch expression - Fix Reporter.cs: guard against null/empty PERFLAB_BUILDTIMESTAMP to prevent ArgumentNullException on DateTime.Parse(null) when the env var is unset (falls back to DateTime.Now) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- const.py: Add IOSINNERLOOP constant and SCENARIO_NAMES mapping - ioshelper.py: New module with iOSHelper class for simulator and physical device management (boot, install, launch, terminate, uninstall, find bundle) - runner.py: Add iosinnerloop subparser, attribute assignment, and full execution branch (first build+deploy+launch, incremental loop with source toggling, binlog parsing, report aggregation, and Helix upload) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pre.py: Install maui-ios workload, create MAUI template (no-restore for Helix), strip non-iOS TFMs with flexible regex, inject MSBuild properties (AllowMissingPrunePackageData, UseSharedCompilation), copy merged NuGet.config for Helix-side restore, create modified source files for incremental edit loop, check Xcode compatibility. test.py: Thin entrypoint that builds TestTraits and invokes Runner. post.py: Uninstall app from simulator, shut down dotnet build server, clean directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create the Helix machine setup script for MAUI iOS inner loop measurements. This script runs on the macOS Helix machine before test.py and handles: 1. DOTNET_ROOT/PATH configuration from the correlation payload SDK 2. Xcode selection — auto-detects highest versioned Xcode_*.app, matching the pattern used by maui_scenarios_ios.proj PreparePayloadWorkItem 3. iOS simulator runtime validation via xcrun simctl 4. Simulator device boot with graceful already-booted handling 5. maui-ios workload install using rollback file from pre.py, with --ignore-failed-sources for dead NuGet feeds 6. NuGet package restore with --ignore-failed-sources /p:NuGetAudit=false 7. Spotlight indexing disabled via mdutil to prevent file-lock errors Follows the same structure as the Android inner loop setup_helix.py: context dict pattern, step-by-step functions, structured logging to HELIX_WORKITEM_UPLOAD_ROOT for post-mortem debugging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define the Helix .proj file for iOS inner loop measurements, modeled after the Android inner loop .proj and existing maui_scenarios_ios.proj patterns. Key design decisions: - Build on Helix machine (not build agent) because deploy requires a connected device/simulator. PreparePayloadWorkItem only creates the template and modified source files via pre.py. - Workload packs stripped from correlation payload (RemoveDotnetFromCorrelation Staging) and reinstalled on Helix machine by setup_helix.py. - Environment variables set via shell 'export' in PreCommands (not in Python) because os.environ changes don't persist across process boundaries. - No XHarness — iOS inner loop uses xcrun simctl directly. - Simulator-only for now; physical device support (ios-arm64, code signing) is structured as a future TODO pending runner.py device support. - 01:30 timeout to accommodate iOS build + workload install + NuGet restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- sdk-perf-jobs.yml: Add Mono Debug job entry for maui_scenarios_ios_innerloop on osx-x64-ios-arm64 (Mac.iPhone.17.Perf queue) - run-performance-job.yml: Add maui_scenarios_ios_innerloop to the in() check so --runtime-flavor is forwarded to run_performance_job.py - run_performance_job.py: Add maui_scenarios_ios_innerloop to get_run_configurations() (CodegenType, RuntimeType, BuildConfig) and to the binlog copy block for PreparePayloadWorkItems artifacts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ioshelper.py: Add detect_connected_device() with auto-detection via xcrun devicectl (JSON + fallback text parsing), uninstall_app_physical, terminate_app_physical, close_physical_device, and cleanup() dispatch - runner.py: Add --device-type arg (simulator/device) to iosinnerloop subparser, auto-infer from RuntimeIdentifier, auto-detect device UDID, branch setup/install/startup/cleanup for physical vs simulator - setup_helix.py: Detect device type from IOS_RID env var, skip simulator boot for physical device, add detect_physical_device() for Helix - post.py: Handle physical device uninstall via devicectl with UDID auto-detection fallback - maui_scenarios_ios_innerloop.proj: Add physical device HelixWorkItem (conditioned on iOSRid=ios-arm64), pass IOS_RID to Pre/PostCommands, add --device-type arg to both simulator and device workitems Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e sort Fix 1 (Major): Replace non-existent 'devicectl terminate --bundle-id' with '--terminate-existing' flag on launch command. Make terminate_app_physical() a no-op with documentation explaining why. Fix 2 (Medium): Write devicectl JSON output to temp file instead of /dev/stdout, which mixes human-readable table and JSON. Applied in both ioshelper.py and setup_helix.py with proper temp file cleanup. Fix 3 (Medium): Add standard UUID pattern (8-4-4-4-12) to UDID regex in _detect_device_fallback() for CoreDevice UUID format compatibility. Fix 4 (Medium): Normalize MAUI template to always use Pages/ subdirectory in pre.py. If template puts MainPage files at root, move them to Pages/. Add explanatory comment in .proj documenting the coupling. Fix 5 (Minor): Use tuple-of-ints version sort for Xcode selection instead of string comparison (fixes 16.10 < 16.2 ordering bug). Fix 6 (Minor): Make simulator boot failure fatal with sys.exit(1). Add dynamic fallback to latest available iPhone simulator before failing. Fix 7 (Nit): Add missing trailing newline to runner.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace deprecated tempfile.mktemp() with tempfile.mkstemp() in both ioshelper.py and setup_helix.py to avoid TOCTOU race condition. - Fix unreachable fallback in detect_connected_device(): when devicectl exits non-zero (e.g., older Xcode without --json-output), call _detect_device_fallback() instead of returning None immediately. - Guard against missing JSON report in runner.py IOSINNERLOOP branch: Startup.cs only writes reports when PERFLAB_INLAB=1, so local runs would crash with FileNotFoundError. Now degrades gracefully with empty counters and a warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Temporarily disable all other scenario jobs to speed up CI iteration while validating the new MAUI iOS Inner Loop scenario. This change should be reverted before merging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Capture dotnet build output instead of crashing on CalledProcessError - Create traces/ directory before first build - Fix setup_helix.py to write output.log (matches .proj expectation) - Improve error handling for build failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet build stdout/stderr wasn't appearing in Helix console logs, making it impossible to diagnose build failures. Explicitly capture and print build output through Python logging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 2943141 hit the 90-minute timeout. iOS first build with AOT compilation can take 30+ minutes, plus 3 incremental iterations. Increasing to 2.5 hours to allow full completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Helix machines have Xcode 26.2 but the iOS SDK requires 26.3. The minor version difference shouldn't affect build correctness, so bypass the check with ValidateXcodeVersion=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mac.iPhone.17.Perf queue uses Intel x64 machines which need iossimulator-x64, not iossimulator-arm64. Add architecture detection in setup_helix.py and update default RID in .proj. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The traces upload directory already exists from the first build, causing copytree() to fail on subsequent iterations. Clear it before each parsetraces() call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The -v:n flag was added to debug build errors but produces excessive file copy logs. Default verbosity shows errors/warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add /p:MtouchLink=None to disable managed linker for Debug inner loop builds, avoiding MT0180 errors on machines without Xcode 26.3 - Add minimum Xcode version check in setup_helix.py for fast failure with clear diagnostics when machine has Xcode < 26.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove temporary ${{ if false }}: wrappers that disabled all jobs
except iOS inner loop during iterative CI debugging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…loop - Separate install from simulator/device setup in ioshelper.py - Capture install time for first and incremental deploys in runner.py - Add "Install Time" counter to both perf reports - Add CoreCLR Debug job entry in pipeline YAML - Add device (ios-arm64) job entries for both Mono and CoreCLR - Wire iOSRid env var through to MSBuild for device builds - [TEMP] Disable non-iOS-inner-loop jobs for CI validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the dynamically-resolved manifest references SDK packs not yet propagated to NuGet feeds, fall back to installing without the rollback file. This avoids CI being blocked by transient feed propagation delays. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the rollback file references SDK packs not yet propagated to NuGet feeds, retry without the rollback file. Matches the fallback pattern already added to pre.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The simulator HelixWorkItem was unconditionally included, even when iOSRid=ios-arm64. This caused the simulator to receive device RID in _MSBuildArgs, producing ARM64 binaries that can't install on a simulator. Add Condition to exclude it from device jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t.py Consolidate 12 duplicated simulator/device methods in ioshelper.py into a unified API (setup_device, install_app, measure_cold_startup, cleanup) that dispatches internally based on is_physical_device. Removes all if-is_physical dispatch branches from runner.py. Extract merge_build_deploy_and_startup and _make_counter to module-level helpers. Inline the incremental iteration loop (was a nested function with 10 parameters). Simplify post.py to reuse ioshelper instead of duplicating device detection. Extract inject_csproj_properties in pre.py. Net reduction: -232 lines across 4 files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ling Re-apply run_env_vars (including iOSRid) right before perf_send_to_helix() so the MSBuild .proj ItemGroup conditions can correctly exclude the simulator work item from device jobs. Add '|| exit $?' to setup_helix.py PreCommands so that when setup_helix exits non-zero (e.g., Xcode too old), the Helix shell stops instead of continuing to run test.py which would fail with a less clear error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Env var inheritance through msbuild.sh/tools.sh is unreliable for iOSRid. Add ios_rid field to PerfSendToHelixArgs and pass it as /p:iOSRid=<value> on the MSBuild command line so it reaches .proj evaluation deterministically. Also set it via set_environment_variables as belt-and-suspenders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mono_InnerLoop → Mono_InnerLoop_Simulator CoreCLR_InnerLoop → CoreCLR_InnerLoop_Simulator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace simctl (simulator) and devicectl (device) install/launch commands with mlaunch to match the real Visual Studio F5 developer experience: - Simulator: --launchsim combines install + launch (install_app returns 0) - Device: --installdev for install, --launchdev for launch - Device cleanup: --uninstalldevbundleid replaces devicectl uninstall - Simulator cleanup: unchanged (simctl terminate + uninstall) - Added _resolve_mlaunch() to find mlaunch from iOS SDK packs Device detection (devicectl) and simulator management (simctl boot/ terminate/uninstall) remain unchanged. The install_app/measure_cold_startup API is preserved so runner.py requires no changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of making install_app() a no-op for simulator, use mlaunch --installsim to get a separate install measurement. measure_cold_startup() still uses --launchsim for launch timing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stall After installing the maui-ios workload, read _RecommendedXcodeVersion from the SDK's Versions.props and switch to the matching Xcode_*.app if the currently active Xcode doesn't match. This handles the case where Helix agents have a newer Xcode than the SDK requires. Falls back gracefully to the already-selected Xcode if no matching installation is found. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| if self.framework: | ||
| base_cmd.extend(['-f', self.framework]) | ||
| if self.msbuildargs: | ||
| base_cmd.extend([arg for arg in self.msbuildargs.split(';') if arg]) |
Comment on lines
+97
to
+100
| m = re.search(r'Microsoft\.iOS\.Sdk\.([^/\\]+)', p) | ||
| if not m: | ||
| return () | ||
| parts = re.split(r'[.\-+]', m.group(1)) |
…n space and semicolon - _resolve_mlaunch now sorts on the pack <version> directory (parent of tools/) rather than the pack-name suffix, so 26.10 ranks above 26.2. - The inner-loop proj builds _MSBuildArgs with a mix of ';' and space separators; splitting --msbuild-args on ';' alone left embedded spaces inside arguments and broke 'dotnet build'. Split on both. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… rm hang
The previous code shared a single `ioshelper_startup.logarchive` path across
all iterations and bracketed the `sudo log collect` call with `sudo rm -rf`
calls to purge any stale archive (since `log collect` refuses to overwrite
an existing --output path with exit 74). That works when sudo credentials
are cached, but hangs indefinitely in non-TTY background contexts (e.g.,
campaigns started via detached caffeinate, nohup, or any agent runner)
because the sudo password prompt has no terminal to read from.
NOPASSWD is only practical to scope to `/usr/bin/log collect` (per the
existing /etc/sudoers.d/log-collect convention); scoping it to /bin/rm is
broader than most users want.
Switch to a per-call unique logarchive path
`ioshelper_startup_{pid}_{ms}.logarchive`. macOS's tmp reaper cleans /tmp
on its own schedule, so per-iteration archives don't accumulate harmfully.
`log collect` succeeds on the first try every time since the path never
pre-exists. No sudo rm needed.
Validated by a 4-side M365 campaign overnight: 11 cold-startup events per
side across all 4 configs, zero hangs, with the orchestrator running fully
detached from any user TTY.
| ios_rid_env = os.environ.get('IOS_RID', '') | ||
| if ios_rid_env and 'RuntimeIdentifier=' in self.msbuildargs: | ||
| self.msbuildargs = re.sub( | ||
| r'RuntimeIdentifier=\S+', |
| import os | ||
| import glob | ||
| import re | ||
| import shlex |
|
|
||
| import os | ||
| import subprocess | ||
| import sys |
Comment on lines
3
to
7
|
|
||
| <PropertyGroup> | ||
| <IncludeXHarnessCli>true</IncludeXHarnessCli> | ||
| <_IsInnerLoop Condition="'$(RunKind)' == 'maui_scenarios_ios_innerloop'">true</_IsInnerLoop> | ||
| <IncludeXHarnessCli Condition="'$(_IsInnerLoop)' != 'true'">true</IncludeXHarnessCli> | ||
| </PropertyGroup> |
- Pulls macos-26 fix for Xcode 26.3 vs 26.5 mismatch - Pulls MAUI Android Inner Loop scaffolding (workload_name signature, shared mauisharedpython.install_latest_maui, AndroidInnerLoop parser enum, runner.py top-level ANDROIDINNERLOOP elif branch) - Adds osx-x64-ios-arm64-innerloop build machine (iPhone 13 + macos-26) for iOS Inner Loop scenarios; regular MAUI iOS jobs stay on osx-x64-ios-arm64 (iPhone 17) - Splices iOS Inner Loop pieces back into shared runner.py / Startup.cs alongside Android Inner Loop equivalents - Re-applies non-iOS job disables for WIP iteration speed (top-level runPublicJobs / runScheduledPrivateJobs + per-block scenario / NativeAOT / Android Inner Loop wrappers) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
davidnguyen-tech
force-pushed
the
nguyendav/maui-ios-inner-loop
branch
from
0935de6 to
3b66e55
Compare
… iPhone13 The shared maui_scenarios_ios.proj uses $(RunKind) conditions to switch between regular and inner-loop ItemGroups, but RunKind was never set as an env var or pipeline variable on the build agent. As a result, the inner-loop jobs in build 2986285 ran the regular work items (Build Time, SOD, Device Startup) instead of the inner-loop ones. Fix: add run_kind to PerfSendToHelixArgs and emit it via ##vso[task.setvariable]/MSBuild property, symmetric with the iOSRid plumbing added in 5c092e4. Also revert the regular iOS jobs (osx-x64-ios-arm64) from Mac.iPhone.17.Perf back to Mac.iPhone.13.Perf. The iPhone17 queue has unhealthy macOS hosts as of 2026-05 and was producing Device Startup work item failures on Helix. Note: both iPhone13 and iPhone17 queues have estimated removal date 2026-06-01 and will need follow-up coordination with dnceng. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| ios_rid_env = os.environ.get('IOS_RID', '') | ||
| if ios_rid_env and 'RuntimeIdentifier=' in self.msbuildargs: | ||
| self.msbuildargs = re.sub( | ||
| r'RuntimeIdentifier=\S+', |
Comment on lines
+1172
to
+1205
| ios_rid = os.environ.get("IOS_RID", "iossimulator-x64") | ||
| is_physical_device = (ios_rid == "ios-arm64") | ||
|
|
||
| # Detect host architecture to select the correct simulator RID. | ||
| # Mac.iPhone.17.Perf queue uses Intel x64 machines which need | ||
| # iossimulator-x64, not iossimulator-arm64. Apple Silicon needs | ||
| # iossimulator-arm64. Physical device builds (ios-arm64) target the | ||
| # iPhone hardware, not the Mac, so skip architecture override. | ||
| if not is_physical_device: | ||
| host_arch = platform.machine() | ||
| if host_arch == "x86_64": | ||
| ios_rid = "iossimulator-x64" | ||
| elif host_arch == "arm64": | ||
| ios_rid = "iossimulator-arm64" | ||
| else: | ||
| log(f"WARNING: Unknown architecture '{host_arch}', " | ||
| f"keeping IOS_RID={ios_rid}", tee=True) | ||
| os.environ["IOS_RID"] = ios_rid | ||
| log(f"Host architecture: {host_arch}, using IOS_RID={ios_rid}", tee=True) | ||
|
|
||
| # The simulator device name env var is no longer used: we always create | ||
| # our own fresh simulator under this user's CoreSimulator namespace via | ||
| # create_and_boot_simulator(). Pre-existing devices on Helix machines | ||
| # may belong to other users and refuse to boot with permission errors. | ||
|
|
||
| # Framework and MSBuild args are passed as command-line arguments when | ||
| # available (from the .proj PreCommands), or fall back to env vars. | ||
| framework = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("PERFLAB_Framework", "") | ||
| msbuild_args = sys.argv[2] if len(sys.argv) > 2 else "" | ||
|
|
||
| ctx = { | ||
| "framework": framework, | ||
| "msbuild_args": msbuild_args, | ||
| "workitem_root": workitem_root, |
| return (the same as what mlaunch internally reports), then | ||
| verify the process is still alive after a short stabilization | ||
| window so we don't report success for a crashed launch. | ||
| - Device: mlaunch --launchdev (returns immediately with PID) |
| ###################################################### | ||
|
|
||
| - ${{ if parameters.runPublicJobs }}: | ||
| - ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP |
run_performance_job.py runs 'dotnet msbuild /t:PreparePayloadWorkItems'
on the build agent (line 1202) to bundle scenario payloads before
SendToHelix. The proj's _IsInnerLoop condition depends on RunKind, but
RunKind was never exported to os.environ for this MSBuild invocation —
only the subsequent SendToHelix step received it (via the
##vso[task.setvariable] emitted by set_environment_variables).
Without RunKind, inner-loop runs silently selected the regular
PreparePayloadWorkItem items ('pre.py publish' for mauiios/netios/etc.)
instead of the inner-loop items ('pre.py default' in the
mauiiosinnerloop payload dir). As a result, rollback_maui.json was
never created in the inner-loop payload dir, so the Helix work item's
setup_helix.py fell back to installing the latest maui-ios workload
manifest — which currently references missing Android cross-targeting
packs (microsoft.android.sdk.darwin 36.1.69) and fails.
Mirrors the iOSRid propagation pattern at lines 1178-1180.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Helix Mac perf queues (Mac.iPhone.13/17.Perf) are EOL 2026-06-01 and need an Xcode 26.5 infra update that isn't available. The hosted macos-26 image already ships Xcode + iOS simulators and is auto-updated, so for the simulator inner-loop scenario run setup_helix.py -> test.py -> post.py directly on the agent and emulate the HELIX_* env those scripts expect, bypassing Helix entirely. PerfLab upload still works via AzureCLI@2 with the '.NET Performance' service connection (DefaultAzureCredential/UAMI). - sdk-perf-jobs.yml: runOnAgent=true on the two simulator jobs; disable the two physical-device jobs (depend on the EOL Helix Mac queues). - run-performance-job.yml: new runOnAgent param; skip send-to-helix and add an AzureCLI@2 step that runs the on-agent driver when set. - run-ios-innerloop-on-agent.sh: the on-agent driver (mirrors the simulator HelixWorkItem in maui_scenarios_ios.proj). Upload gated to internal/non-PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
machine-setup.sh references $HELIX_CORRELATION_PAYLOAD (DOTNET_ROOT/PATH), so under 'set -u' sourcing it before exporting HELIX_CORRELATION_PAYLOAD failed with 'HELIX_CORRELATION_PAYLOAD: unbound variable'. Move the HELIX_* exports above the source and relax nounset around it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The on-agent path bypasses Helix, which installs the scenario's Python dependencies (azure SDK, cryptography, etc.) via helix_pre_commands. runner.py imports upload.py unconditionally, which imports azure.storage.blob / azure.identity, so without these packages test.py crashes with 'ModuleNotFoundError: No module named azure'. Install requirements.txt on the agent before setup_helix.py/test.py, using --break-system-packages --user to match the existing osx pip handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The on-agent simulator run uploads results from the hosted macOS agent, where the only available identity is the '.NET Performance' service connection that AzureCLI@2 logs in. upload.py's existing auth chain is Helix-specific: it wraps DefaultAzureCredential in a ClientAssertionCredential federated exchange against api://AzureADTokenExchange (the service-connection SP is forbidden from that: AADSTS7002341) and then falls back to a Helix-only cert path. Add an opt-in, fail-closed branch to upload.py's get_credential() that uses AzureCliCredential directly for a Storage data-plane token, gated by PERFLAB_UPLOAD_USE_AZURE_CLI so Helix behavior is unchanged. The on-agent driver sets that env var when uploading. The service connection has write access to pvscmdupload, the same account/identity used by register-build-job/upload-build-artifacts-job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AzureCliCredential defaults to a 10s subprocess timeout. On the loaded hosted agent, cold-starting 'az account get-access-token' for the storage resource exceeds 10s and fails with CredentialUnavailableError/TimeoutExpired. Allow more time so the service-connection token can be acquired at upload time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that the Helix Mac queue (Mac.iPhone.13.Perf) is updated to Xcode 26.5, run the iOS inner-loop jobs on Helix instead of the AzDO agent: - Simulator jobs: drop runOnAgent: true so they dispatch to the Helix Mac queue again. This avoids the AzDO on-agent PerfLab upload blocker (the service-connection SP lacks a Storage Queue Data role on pvscmdupload, so the resultsqueue enqueue 403s); on Helix the cert-based upload path works. - Device jobs: re-enable (remove the if-false wrapper). They run on the same queue with iOSRid=ios-arm64. - Regular maui_scenarios_ios jobs (SOD/build-time/startup): disable during the iOS inner-loop WIP for faster iteration; re-enable before merge. The on-agent path (run-ios-innerloop-on-agent.sh, the runOnAgent wiring in run-performance-job.yml, and upload.py's AzureCliCredential branch) is kept dormant as a fallback until the Helix path is confirmed working. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 2992869 showed Mac.iPhone.13.Perf is a stopgap queue missing the infra
these jobs need:
- Simulator scenario succeeded but PerfLab upload failed: the cert
/Users/helix-runner/certs/LabCert1.pfx is not provisioned there.
- Device jobs failed fast in signing-artifact discovery: embedded.mobileprovision
and the 'sign' tool are absent ('DEVICE INFRA UNAVAILABLE').
ioshelper.py documents that the device code-signing infra lives on
Mac.iPhone.17.Perf, and main's regular iOS perf jobs run/upload from .17 (so the
upload cert is there too). .13 was only a temporary home while .17 had unhealthy
hosts in 2026-05. Point the inner-loop queue back to Mac.iPhone.17.Perf so both
the simulator upload and the device signing path have what they need.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…code 26.5) Build 2992885 on Mac.iPhone.17.Perf aborted immediately for both sim and device: 'No Xcode matching 26.5 found. Available: [Xcode_26.3.app]'. The Xcode 26.5 update landed on .13, not .17. So neither queue is complete for the net11.0_26.5 scenario: - Mac.iPhone.13.Perf: Xcode 26.5 yes; upload cert + device signing NO. - Mac.iPhone.17.Perf: upload cert + device signing yes; Xcode 26.3 (too old). Keep the queue on .13 since it's the only one that actually runs the scenario; provisioning the upload cert + device-signing infra onto .13 (or installing Xcode 26.5 on .17) is an infra task tracked separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
NOTE: The scenario basically works, there are just some infra issues - MI & Xcode. Postponing this PR until we agree on a yes/no. |
Member
Author
|
Update - We've come to conclusion that we would like to move this PR forward. |
The .NET 11 SDK blocks UseMonoRuntime for mobile TFMs (NETSDK1242). Pass /p:_DisableCheckForUnsupportedMonoMobileRuntime=true (the SDK's supported opt-out, dotnet/sdk#54713) alongside UseMonoRuntime=true for the Mono inner-loop variant so restore + build succeed on net11.0-ios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| ###################################################### | ||
|
|
||
| - ${{ if parameters.runPublicJobs }}: | ||
| - ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP |
| displayName: Run tooling tests | ||
|
|
||
| - ${{ if parameters.runPublicJobs }}: | ||
| - ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP |
|
|
||
| # Scheduled runs will run all of the jobs on the PerfTigers, as well as the Arm64 job | ||
| - ${{ if parameters.runScheduledPrivateJobs }}: | ||
| - ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP |
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
| - ${{ if false }}: # disabled: iOS-only WIP test run |
Comment on lines
+1519
to
+1523
| self.traits.add_traits(overwrite=True, apptorun="app", startupmetric=const.IOSINNERLOOP, | ||
| tracename=f'{binlog_prefix}first-build-and-deploy.binlog', | ||
| scenarioname=scenarioprefix + " - First Build and Deploy", | ||
| upload_to_perflab_container=False) | ||
| startup.parsetraces(self.traits) |
Comment on lines
+1594
to
+1601
| # Clear stale traces upload dir so copytree in parsetraces doesn't collide | ||
| helix_upload_dir = helixuploaddir() | ||
| if helix_upload_dir is not None: | ||
| traces_upload = os.path.join(helix_upload_dir, 'traces') | ||
| if os.path.exists(traces_upload): | ||
| rmtree(traces_upload) | ||
|
|
||
| startup.parsetraces(self.traits) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds MAUI iOS Inner Loop performance measurements to CI, supporting both iOS simulators and physical devices.
What's included:
Measurements:
Targets:
Based on: