diff --git a/.github/scripts/CLASSIFICATION.md b/.github/scripts/CLASSIFICATION.md new file mode 100644 index 000000000..033de70f5 --- /dev/null +++ b/.github/scripts/CLASSIFICATION.md @@ -0,0 +1,91 @@ +# Sample validator classification contract + +Authoritative `failure`-vs-`error` contract for `.github/scripts/validate-sample.sh`. +The public-first validation pipeline — and specifically the quarantine loop (P4.4) — +**keys on this contract**. Read this before changing the classifier or anything that +consumes its verdict. + +## Exit contract (three-way, load-bearing) + +| Exit | Verdict | Meaning | Downstream action | +|------|---------|---------|-------------------| +| `0` | `pass` | The sample loads at Build Readiness Level 3 (builds/compiles; `sample.yaml` commands, if any, all exit 0). | none | +| `1` | `fail` | **The SAMPLE is broken**, or a runtime failure we cannot positively attribute to our infra. | P4.4 may count a strike / quarantine (advisory-first). | +| `2` | `error` | **OUR INFRA is sick** — a precondition error, or a positively-identified transport/registry failure. | Page us. **P4.4 must NEVER count or quarantine on `error`.** | + +The split is a safety interlock. A real breakage mislabeled `error` hides broken code forever +(there is no counter behind `error`). A transient infra blip mislabeled `failure` risks +quarantining a healthy sample. Both directions are dangerous; keep the split honest. + +## What is `error` (exit 2) + +**Precondition errors (validator can't even run the check):** +- bad / missing CLI args; unknown or missing `--language`; missing or nonexistent `--sample-dir` +- `sample.yaml` unreadable or malformed (`yq` read fails) +- a required toolchain binary missing from `PATH` (`require_tool`) +- `yq` unavailable when a `sample.yaml` must be read +- Python venv create/activate failure + +**Runtime transport/registry failures (added in P1.3):** +- a `pip install` or `npm install` that failed with **positive transport evidence**: + DNS failure, connection refused/reset, connect/read timeout, or a registry `5xx`. +- Detection lives in `dep_infra_signature()` — a **narrow allow-list of tool-specific + transport strings** (e.g. `ECONNREFUSED`, `ENOTFOUND`, `Temporary failure in name + resolution`, `Max retries exceeded`, `503 Server Error`). It is the authoritative signature + list; this doc deliberately does not duplicate it. + +## What is `failure` (exit 1) + +- any `sample.yaml` build/validate/test command exits non-zero +- a compile/build step fails: `dotnet build`, `mvn compile`, `gradle build`, `go build`, + `npm run build`, `node --check`, `py_compile` +- a dependency install (`pip install` / `npm install`) fails **without** transport evidence — + i.e. a genuine resolution error: package does not exist, no matching version, version + conflict (`No matching distribution found`, npm `E404` / `ETARGET` / `ERESOLVE`). + +## Ambiguity-bias rule + +When a dependency-install failure shows **no** positive transport evidence, it is classified +`failure`, **never** `pass`. When we genuinely cannot tell an infra blip from a sample break, +we bias to **`failure`**. + +Rationale — *recourse asymmetry*, not "strikes are cheap": +- A false `error` has **no counter**: it silently hides real breakage forever. +- A false `failure` is recoverable: P4.4 runs advisory-first, quarantine is strike-based, and a + human reviews before a sample moves. +- We keep the transport allow-list **narrow** so false `failure`s stay rare in the first place — + we do not lean on the strike mechanism to absorb sloppiness (infra outages are correlated and + can survive multiple runs). + +Fail-loud (ADO 5247751 lesson): an indeterminate probe must resolve to a defined verdict. It +never fails **open** to `pass`. + +## Honest v1 limits + +- Only the two **dedicated** dependency-install steps — `pip install` and `npm install` — are + inspected for transport evidence. The **merged resolve+compile tools** (`dotnet build`, + `mvn compile`, `gradle build`, `go build`, `npm run build`) interleave dependency download + with compilation and user scripts; grepping their combined output would manufacture false + `error`s, so in v1 they classify as `failure`. A network blip that strikes *during* one of + those merged steps can therefore be misread as a sample `failure`. This is a known limit, not + a bug — extending it needs tool-specific restore phases and is deferred. +- `dep_infra_signature()` is a **narrow heuristic**, not a robust classifier. It matches known + transport strings and **will need maintenance** as runner `pip` / `npm` versions drift their + error wording. Unknown output stays `failure`. +- **Blocked-registry / proxy `403`.** The transport allow-list catches connection-level failures + (DNS, refused/reset, timeout) and registry `5xx`, but **not** an HTTP `403 Forbidden` or a + blocked-page response from a policy proxy. This matters where a registry is deliberately fenced + off — e.g. Microsoft Security is blocking direct access to public npm registries on + Microsoft-managed devices, routing installs through a CFS-protected feed; a package still inside + the release-hold window can come back `403`/blocked. Such a block is arguably *infra*, but in v1 + it classifies as `failure` (bias-to-`failure`, no false `error`). This is moot for the deliverable + itself — the pipeline runs on GitHub-hosted runners, not managed devices — but bites local runs on + managed devices. Promoting proxy-`403`/blocked-registry to `error` is deferred follow-up. + +## Proof + +The contract is exercised end-to-end on a real GitHub Actions runner by +`.github/scripts/test/run-tests.sh` (invoked from `.github/workflows/validate-sample-selftest.yml`): +a forced-unreachable registry (`127.0.0.1:1` blackhole) for `pip install` / `npm install` +classifies as `error` (exit 2), while an unaccompanied resolution failure classifies as +`failure` (exit 1). diff --git a/.github/scripts/WRAPPER-MAPPING.md b/.github/scripts/WRAPPER-MAPPING.md new file mode 100644 index 000000000..e56dc4fe4 --- /dev/null +++ b/.github/scripts/WRAPPER-MAPPING.md @@ -0,0 +1,114 @@ +# ADO → GitHub Actions wrapper mapping (P1.4 retirement audit) + +**Task:** ADO 5449688 — "P1.4 · Swap ADO task wrappers for GH actions" +(), parent Feature 5443705. + +**Purpose.** This is the audit-of-record proving that the public-first validation path in this repo +carries **no Azure-Pipelines task wrappers or expressions** — every ADO construct from the private +port source has a first-class GitHub-Actions equivalent here, or was intentionally dropped with a +recorded reason. It also documents which constructs are deliberately **deferred to Phase 2** and +therefore intentionally absent from the Phase-1 foundation. + +## Port source (read-only ground truth) + +Private ADO pipeline: `microsoft-foundry/foundry-samples-pr` → `.azure-pipelines/validation.yml` @ `main` +(). +Its stages are **named, not numbered** — referred to by name below. (The file's own L199 comment +mislabels the `Validate` stage as "Stage 2"; that comment is ignored.) + +| ADO stage (name) | Role | Ported to (Phase) | +| --- | --- | --- | +| `CheckMailmap` | commit-author mailmap gate | **DROPPED** (frozen plan) | +| `DetectChanges` | which samples changed | **P1.2** — `.github/scripts/detect-changed-samples.sh` | +| `Validate` | per-language L3 load validation | **P1.1** — `.github/scripts/validate-sample.sh` | +| `Report` | status posting / PR comment / manifest | **DROPPED**; returns as P2.3/P3 via commit statuses | + +## GitHub-Actions targets on this branch (`brandom-msft-fluffy-lamp`, stacked on P1.3 `01cdde17`) + +- `.github/workflows/validate-sample-selftest.yml` — real-runner harness for the `Validate` port. +- `.github/workflows/detect-changed-samples-selftest.yml` — real-runner harness for the `DetectChanges` port, + including the `needs.*.outputs` cross-job plumbing proof. +- `.github/scripts/validate-sample.sh` — the once-only validator that collapses the five duplicated ADO + per-language jobs into one function set. +- `.github/scripts/detect-changed-samples.sh` — the in-job change detector. +- `.github/scripts/test/**` — hermetic unit proofs driven by the self-tests. + +--- + +## Acceptance-criteria verdicts + +| AC | Requirement | Verdict | Evidence | +| --- | --- | --- | --- | +| **AC1** | No ADO-task-specific wrappers remain in the validation path | ✅ PASS | grep of the entire `.github` tree for `@N` task refs, `task:`, `$[ ]`, `stageDependencies`, `dependsOn`, `pool:`/`vmImage:`, `condition: eq(...)`, `##vso[...]`, `$(Build.*)`/`$(System.*)`/`$(Agent.*)` → **zero matches** | +| **AC2** | Language setup via GH actions | ✅ PASS | all 5 languages pinned via first-class setup actions in `validate-sample-selftest.yml` (table below) | +| **AC3** | Cross-job data flows via `needs.*.outputs` | ✅ PASS | `detect-changed-samples-selftest.yml` proves `detect`→`consume` and `detect-docs-only`→`consume-docs-only` via `needs.detect.outputs.{has_changes,count,samples}`; no `stageDependencies`/`$[coalesce(...)]` remains | + +--- + +## AC2 — language-setup wrapper mapping + +Each ADO `Use*/*Tool@N` setup task → the first-class GitHub setup action, with the version **pinned** +(not incidental to the runner image). Line numbers are `validate-sample-selftest.yml`. + +| Language | ADO wrapper (`validation.yml`) | GH-native equivalent (this branch) | Pinned version | +| --- | --- | --- | --- | +| .NET | `UseDotNet@2` (L220) | `actions/setup-dotnet@v4` (L36–39) | `8.0.x` | +| Python | `UsePythonVersion@0` (L330) | `actions/setup-python@v5` (L41–44) | `3.12` | +| Node/TS | `NodeTool@0` (L465) | `actions/setup-node@v4` (L46–49) | `20` | +| Java | `JavaToolInstaller@0` (L604) | `actions/setup-java@v4` (L51–55) | `temurin` `17` | +| Go | `GoTool@0` (L729) | `actions/setup-go@v5` (L57–60) | `1.22` | + +> **`GoTool@0` "for yq installation" (L226/L336/L471/L611).** In the ADO `Validate` stage the non-Go +> lanes each pulled Go *only* to `go install` yq. That incidental dependency is dropped: yq is installed +> directly from the mikefarah release binary in one `run:` step (`validate-sample-selftest.yml` L62–67), +> so no lane needs Go unless it is actually validating Go samples. + +--- + +## AC1 / AC3 — full construct mapping (wrapper → GH equivalent OR intentionally dropped/deferred) + +| ADO construct (`validation.yml` line) | Kind | Disposition | GH equivalent / reason (file · symbol) | +| --- | --- | --- | --- | +| `pool: { vmImage: ubuntu-latest }` (L44–45) | agent pool | **Ported** | `runs-on: ubuntu-latest` — `validate-sample-selftest.yml` L32, `detect-changed-samples-selftest.yml` L33 | +| `checkout: self` (L62, L97, L215, …) | checkout | **Ported** | `actions/checkout@v4` — self-tests L34 / L35 (`fetch-depth: 0` for the detector's history walk) | +| `UseDotNet@2` (L220) | task | **Ported** | `actions/setup-dotnet@v4` — self-test L36 | +| `UsePythonVersion@0` (L330) | task | **Ported** | `actions/setup-python@v5` — self-test L41 | +| `NodeTool@0` (L465) | task | **Ported** | `actions/setup-node@v4` — self-test L46 | +| `JavaToolInstaller@0` (L604) | task | **Ported** | `actions/setup-java@v4` — self-test L51 | +| `GoTool@0` (L729, and L226/L336/L471/L611 "for yq") | task | **Ported** / incidental use dropped | `actions/setup-go@v5` — self-test L57; yq via `run:` (L62–67) | +| `Install yq` inline `Bash@3` (L234/L343/L478/…) | task | **Ported** | `run:` step downloading the mikefarah release binary — self-test L62–67 | +| `Bash@3` "Validate samples" ×5 (L313/L448/L587/L712/L829) | task | **Ported + collapsed** | one script `validate-sample.sh` (invoked `run: bash …/run-tests.sh`, self-test L78–79). The five byte-identical per-language ADO job bodies become one function set: `run_sample_yaml` + `default_validate_{csharp,python,typescript,java,go}` (each cites its ADO source line in-comment) | +| `Bash@3` "Detect changed samples" (L165) | task | **Ported** | `detect-changed-samples.sh` (driven by `run:` in the detect self-test) | +| `condition: eq(variables.hasCSharp,'true')` ×5 (L213/L323/L458/L597/L722) | runtime expr | **Ported → data-driven** | per-sample fan-out from `needs.detect.outputs.samples`; language derived from the sample path (`samples//…`), not a stage-level boolean. Phase-1 self-test proves the `needs.*.outputs` mechanism (`consume` job, L73–97) | +| `$[coalesce(stageDependencies.DetectChanges.…outputs[…])]` → `hasX` (L204–208) | cross-stage expr | **Ported → `needs.*.outputs`** | `needs.detect.outputs.{has_changes,count,samples}` — detect self-test L44–47 (job outputs) + L79–92 (downstream read). No `stageDependencies`/`coalesce` remains | +| `condition: and(eq(parameters.validateAll,false), ne(Build.Reason,'Schedule'))` (L167) / `or(…,'Schedule')` (L193) | runtime expr | **Deferred (P2)** | validate-all / schedule mode is a Phase-2 trigger concern; not part of the Phase-1 foundation. No ADO expression carried over | +| `publish: … artifact: ChangedSamples` (L195–196) | task | **Ported → job outputs** | detected set flows as `needs.detect.outputs.samples` (self-test), not a pipeline artifact | +| `publish: … artifact: Results{CSharp,…}` (L315/L450/L589/L714/L831) | task | **Deferred (P2)** | per-lane result artifacts (`actions/upload-artifact`) belong to the P2 fan-out; Phase-1 validator writes `passed/failed/errored.txt` via `--results-dir` (`validate-sample.sh` `emit_verdict`) | +| download `ChangedSamples` / `Results*` (L857–877) | task | **Deferred (P2)** | `actions/download-artifact` in the P2 aggregation lane — not Phase 1 | +| `$[stageDependencies.Validate.Validate*.result]` (L845–849) | cross-stage expr | **Deferred (P2)** | future aggregation reads `needs..result`/job outputs; **no aggregator/gate-shim job in Phase 1** (frozen) | +| `$(Build.Reason)` / `$(Build.SourcesDirectory)` / `$(Build.ArtifactStagingDirectory)` (L71/L922/L195/…) | predefined var | **Ported where present / else deferred** | GH contexts: `$GITHUB_EVENT_NAME`, `$GITHUB_WORKSPACE` (used in the detect self-test L54), runner temp. No ADO `$( )` macro remains on this branch | +| `##vso[task.logissue type=error]` (L83–84/L900/L911/L938/L962/L985) | logging cmd | **Dropped with stages** | belonged to `CheckMailmap` + `Report`; both dropped. Phase-1 scripts fail loud via non-zero exit + plain stderr (`validate-sample.sh` `error()`, `detect-changed-samples.sh` fail-loud invariant) | + +--- + +## Intentionally dropped (frozen plan) — NOT a gap + +| ADO stage / step (line) | Why dropped | +| --- | --- | +| `CheckMailmap` stage (L56–87) | mailmap author-gating is not part of the public PR validation path (frozen). | +| `Report` stage — `Bash@3` mint GitHub-App token (L881) + publish statuses (L915) | status-posting is removed from the Phase-1 foundation; it returns as **P2.3/P3** via GitHub **commit statuses**, not an ADO task. Reintroducing token minting here would pull credentials into a credential-free harness (self-tests are `permissions: contents: read`). | +| `Report` — publish `ValidationSummary` / PR-comment (L1169–1175) | PR-comment surface is a Phase-2/3 reporting concern. | +| `Report` — publish `ValidationManifest` (L1343–1344, non-PR L1391) | manifest/roll-up is a Phase-2 aggregation concern. | + +## Frozen constraints honored + +No `pull_request_target`. No trust-scoring, aggregator/gate-shim jobs, or org-membership API calls. +The (future) trusted lane is **not** converted into a per-language matrix. These are P2 concerns and stay +outside the Phase-1 foundation. + +## Residual gap + +**None.** The literal ADO→GH swap was completed natively during P1.1/P1.2 authoring. This audit found no +Azure-Pipelines wrapper or expression remaining in the validation path; the only ADO constructs not +represented here are the intentional drops and Phase-2 deferrals recorded above. Proven additionally by a +green real-runner self-test run (recorded in the P1.4 close-out packet, not embedded here). diff --git a/.github/scripts/detect-changed-samples.README.md b/.github/scripts/detect-changed-samples.README.md new file mode 100644 index 000000000..4157c77bb --- /dev/null +++ b/.github/scripts/detect-changed-samples.README.md @@ -0,0 +1,81 @@ +# How the changed-sample detector works + +Companion doc for [`detect-changed-samples.sh`](./detect-changed-samples.sh) — the in-job +step (P1.2 of the public-first validation pipeline) that answers one question for the +validation gate: **"which samples did this PR/push actually change?"** so we validate only +those, not every sample in the tree. + +## The flow + +```mermaid +flowchart TD + A["git diff:
which FILES changed under samples/?"] --> B{diff succeeded?} + B -- "no (error)" --> X["exit non-zero
FAIL LOUD — never guess"] + B -- "yes" --> C["For each changed file,
walk UP to nearest sample.yaml dir"] + C --> D["Dedupe + sort → unique sample set"] + D --> E["Emit 3 outputs:
has_changes · count · samples (JSON)"] + E --> F["Downstream job reads
needs.detect.outputs.*"] + F --> G{"has_changes?"} + G -- "false (docs-only)" --> H["short-circuit — validate nothing"] + G -- "true" --> I["validate each sample in the set"] +``` + +## The five moves + +1. **Diff** — `git diff HEAD -- samples/`. Base is: an explicit `--base-ref`, else + `origin/` for PRs, else `HEAD~1` for push. +2. **Walk up** — a file matters only if it lives under a folder containing a `sample.yaml`. + That marker file *defines* a sample; the loop climbs to the nearest one. Files with no + `sample.yaml` above them (docs, shared infra) contribute nothing. +3. **Dedupe** — many changed files in one sample collapse to one entry; output is + sorted and deterministic. +4. **Emit** — three values to three places: the run log (humans), `$GITHUB_OUTPUT` (the + next job), and optionally disk via `--output-dir` (tests). +5. **Hand off** — a downstream job reads `needs..outputs.*` and either + short-circuits (empty) or validates the set. + +## The one rule that matters most + +**"I couldn't look" must never look like "nothing changed."** If the diff errors, the +script exits non-zero and stops — it refuses to emit an empty set. An empty set is only ever +produced by a diff that *succeeded* and genuinely found no sample touched. (This is the scar +from ADO 5247751 — a silent empty diff once let an unvalidated change through the gate.) + +## The contract + +| | | +|---|---| +| **Input** | samples root (`--samples-root`, default `samples`), a base ref (explicit `--base-ref` or derived from the GitHub event) | +| **Outputs** | `has_changes` (true/false) · `count` (int) · `samples` (JSON array of dirs) | +| **Exit code** | `0` = detection ran (set may be empty) · non-zero = couldn't run (never "nothing changed") | +| **Requires** | every sample directory has a `sample.yaml` — that's a hard contract, not a convention | + +To be consumable as `needs..outputs.*`, the emitting step needs an `id`, **and** the job +must re-export the step outputs under its own `outputs:` block. Writing `$GITHUB_OUTPUT` alone +is necessary but not sufficient. See `detect-changed-samples-selftest.yml` for the wiring. + +## Known sharp edges (tracked, not blockers) + +These are accepted characteristics of a faithful port of the frozen ADO `DetectChanges` +design — documented in-code in the "Known limitations" block above `resolve_base_ref`. + +- **Base-ref selection** — two-dot PR diff (can over-validate when the target advances), + `HEAD~1` on a multi-commit push (misses earlier commits), best-effort fetch (can pass on a + stale ref). Tracked: ADO 5473511 — + https://msdata.visualstudio.com/Vienna/_workitems/edit/5473511 +- **Post-diff fail-loud coverage** — only the `git diff` stage is provably fail-loud today; + the `sort`/`mktemp`/emit pipeline and exotic (control-char) filenames are the gap. Tracked: + ADO 5473651 — https://msdata.visualstudio.com/Vienna/_workitems/edit/5473651 + +## Where it's proven + +Inert self-test workflow ([`../workflows/detect-changed-samples-selftest.yml`](../workflows/detect-changed-samples-selftest.yml), +`contents: read`, no secrets/OIDC): a hermetic check harness +([`test/run-detect-tests.sh`](./test/run-detect-tests.sh)) plus real-runner jobs proving both +a changed-sample payload and a docs-only empty set flow through `needs.*.outputs` +uncorrupted. + +## Related + +- Port source: the private ADO `validation.yml` **`DetectChanges`** stage. +- Parent work item: ADO 5449686 — https://msdata.visualstudio.com/Vienna/_workitems/edit/5449686 diff --git a/.github/scripts/detect-changed-samples.sh b/.github/scripts/detect-changed-samples.sh new file mode 100644 index 000000000..ef860c82e --- /dev/null +++ b/.github/scripts/detect-changed-samples.sh @@ -0,0 +1,221 @@ +#!/usr/bin/env bash +# detect-changed-samples.sh — reusable in-job change detector for the public-first +# validation pipeline. Ports the private ADO `DetectChanges` stage (job AnalyzeChanges, +# step detectChanges) from .azure-pipelines/validation.yml into a GitHub-Actions-native, +# dependency-light script (git + coreutils only). +# +# Single responsibility: answer "which sample directories were affected by this push/PR?" +# by mapping every changed file under / to its NEAREST ANCESTOR sample.yaml +# directory, then deduping. The caller (P2.1/P2.2 validate lanes) consumes the result as +# GitHub Actions job outputs via needs..outputs.* and fans out — deriving each +# sample's --language from its path (samples//...); this script deliberately does +# NOT pre-group by language (see "Design" below). +# +# Usage: +# detect-changed-samples.sh [--base-ref ] [--event ] +# [--target-branch ] [--samples-root samples] +# [--output-dir ] [--print-base-ref] +# +# Base-ref resolution (faithful to ADO DetectChanges): +# --base-ref wins outright (used by tests + callers that already know the base). +# event pull_request git fetch origin (best-effort) then BASE_REF=origin/, +# where is --target-branch or $GITHUB_BASE_REF. +# otherwise (push/etc) BASE_REF=HEAD~1. +# +# Outputs (the contract P2 consumes): +# has_changes "true" | "false" — false == docs-only / no sample touched (short-circuit flag) +# count integer — number of affected sample dirs +# samples JSON array — deduped, sorted affected sample dirs (repo-relative) +# Written to $GITHUB_OUTPUT when set; always printed to stdout. When --output-dir is given, +# also writes changed_files.txt + unique_changed_samples.txt (mirrors ADO's artifacts). +# +# Exit codes (2-way — intentionally simpler than validate-sample.sh's 0/1/2 classifier; +# detection has no "sample broken" tier): +# 0 OK detection completed — INCLUDING an empty (docs-only) result. +# 1 ERROR fail loud: bad args, not a git repo, or the git diff itself errored. +# +# FAIL-LOUD invariant (ADO 5247751 lesson): a git diff *error* must NEVER be swallowed into +# an empty "nothing changed" result — that fail-opens the sync gate. A diff that SUCCEEDS +# with zero lines is legitimate docs-only (has_changes=false, exit 0). The two are never +# conflated: only `git diff` returning non-zero triggers the exit-1 refusal below. +# +# NOTE: `set -e` is intentionally NOT used; error paths are routed through error() so no +# external command can abort the script with an unclassified status. +set -uo pipefail + +BASE_REF="" +EVENT="" +TARGET_BRANCH="" +SAMPLES_ROOT="samples" +OUTPUT_DIR="" +PRINT_BASE_REF=false + +usage() { + cat <<'EOF' +Usage: detect-changed-samples.sh [options] + + --base-ref Explicit base ref to diff HEAD against (overrides resolution). + --event Event kind: pull_request | push | dispatch (default: $GITHUB_EVENT_NAME or push). + --target-branch
PR target branch (default: $GITHUB_BASE_REF). Used only for pull_request. + --samples-root Root under which samples live (default: samples). + --output-dir If set, write changed_files.txt + unique_changed_samples.txt here. + --print-base-ref Resolve and print the base ref, then exit 0 (no fetch, no diff). + +Exit codes: 0 = detection ok (incl. empty/docs-only), 1 = error (fail loud). +EOF +} + +# error : fail loud — print to stderr and exit 1. Never emit a fake empty result. +error() { + echo "ERROR: ${1:-detection failure}" >&2 + exit 1 +} + +# --- Argument parsing -------------------------------------------------------- +while [ $# -gt 0 ]; do + case "$1" in + --base-ref) BASE_REF="${2:-}"; shift $(( $# > 1 ? 2 : 1 )) ;; + --event) EVENT="${2:-}"; shift $(( $# > 1 ? 2 : 1 )) ;; + --target-branch) TARGET_BRANCH="${2:-}"; shift $(( $# > 1 ? 2 : 1 )) ;; + --samples-root) SAMPLES_ROOT="${2:-}"; shift $(( $# > 1 ? 2 : 1 )) ;; + --output-dir) OUTPUT_DIR="${2:-}"; shift $(( $# > 1 ? 2 : 1 )) ;; + --print-base-ref) PRINT_BASE_REF=true; shift ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; error "unknown argument: $1" ;; + esac +done + +# --- Guards ------------------------------------------------------------------ +git rev-parse --git-dir >/dev/null 2>&1 || error "not a git repository (cwd=$(pwd))" +[ -n "$SAMPLES_ROOT" ] || error "missing/empty --samples-root" +SAMPLES_ROOT="${SAMPLES_ROOT%/}" + +# --- Base-ref resolution ----------------------------------------------------- +# Known limitations (faithful port of the frozen ADO DetectChanges design; see +# the P1.2 close-out for the risk analysis). These are documented, accepted +# characteristics — not bugs — but the next engineer should know the sharp edges: +# +# 1. PR diff is TWO-DOT (`diff origin/ HEAD`), not merge-base +# (three-dot). If advances after the PR diverged, the diff can +# surface files the PR never touched -> extra samples validated. This is a +# FALSE POSITIVE (wasteful, never fail-open) and is usually neutralized by +# GitHub's synthetic PR merge-commit checkout (refs/pull/N/merge). +# +# 2. Non-PR (push/dispatch) base is `HEAD~1` -> only the FINAL commit +# transition. A direct multi-commit push can miss samples changed in +# earlier commits (the event's `before` SHA is ignored). Merge/squash +# commits are fine (HEAD~1 first-parent captures the whole delta). This is +# the one direction that could FALSE-NEGATIVE on the push path. +# +# 3. The fetch below is BEST-EFFORT (`|| true`). If a fresh fetch fails but a +# stale `origin/` already exists, the diff runs against the stale +# ref -> exit 0 with a wrong answer. Only fails loud when NO usable ref +# remains. Low-probability on a fresh runner (checkout populates the ref +# seconds earlier) but in tension with the fail-loud principle (ADO 5247751). +# +# Tracked refinement (base-ref robustness): use github.event.before on push and +# stop swallowing fetch failure. Do NOT change the frozen logic here ad hoc. +resolve_base_ref() { + if [ -n "$BASE_REF" ]; then + echo "$BASE_REF" + return 0 + fi + local ev="$EVENT" + [ -n "$ev" ] || ev="${GITHUB_EVENT_NAME:-push}" + if [ "$ev" = "pull_request" ]; then + local target="$TARGET_BRANCH" + [ -n "$target" ] || target="${GITHUB_BASE_REF:-}" + [ -n "$target" ] || error "pull_request event but no target branch (--target-branch / \$GITHUB_BASE_REF)" + # Best-effort fetch (ADO parity). If offline/absent, the diff below fails loud. + if [ "$PRINT_BASE_REF" != true ]; then + git fetch origin "$target" --quiet 2>/dev/null || true + fi + echo "origin/$target" + else + echo "HEAD~1" + fi +} + +BASE_REF="$(resolve_base_ref)" || exit 1 + +if [ "$PRINT_BASE_REF" = true ]; then + echo "$BASE_REF" + exit 0 +fi + +# --- Diff (FAIL LOUD on error; empty-but-successful is legit docs-only) ------- +CHANGED_FILES="$(mktemp)" +trap 'rm -f "$CHANGED_FILES"' EXIT +# core.quotepath=false keeps non-ASCII paths literal (no octal escaping) so dirname works. +if ! git -c core.quotepath=false diff --name-only "$BASE_REF" HEAD -- "$SAMPLES_ROOT/" > "$CHANGED_FILES"; then + error "git diff '$BASE_REF'..HEAD failed; refusing to continue with an empty change set (would fail-open the gate)" +fi + +# --- Walk each changed file UP to its nearest ancestor sample.yaml dir -------- +UNIQUE="$(mktemp)" +trap 'rm -f "$CHANGED_FILES" "$UNIQUE"' EXIT +{ + while IFS= read -r f; do + [ -z "$f" ] && continue + dir="$(dirname "$f")" + while [ "$dir" != "$SAMPLES_ROOT" ] && [ "$dir" != "." ] && [ "$dir" != "/" ]; do + if [ -f "$dir/sample.yaml" ]; then + echo "$dir" + break + fi + dir="$(dirname "$dir")" + done + done < "$CHANGED_FILES" +} | sort -u > "$UNIQUE" + +# --- Compute outputs --------------------------------------------------------- +# `grep -c . file` prints 0 AND exits 1 on an empty file; a `|| echo 0` fallback +# then double-prints, making COUNT="0\n0". That later leaks a bare "0" line into +# $GITHUB_OUTPUT, which GitHub Actions rejects ("Invalid format '0'") — crashing +# the detect step on any docs-only PR. Guard on file-non-empty so COUNT is always +# a single clean integer. +if [ -s "$UNIQUE" ]; then + COUNT="$(grep -c . "$UNIQUE")" +else + COUNT=0 +fi +if [ "$COUNT" -gt 0 ]; then HAS_CHANGES="true"; else HAS_CHANGES="false"; fi + +# Build a JSON array (dependency-light; escape backslash and double-quote). +json_escape() { printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g'; } +SAMPLES_JSON="[" +first=true +while IFS= read -r line; do + [ -z "$line" ] && continue + if [ "$first" = true ]; then first=false; else SAMPLES_JSON="$SAMPLES_JSON,"; fi + SAMPLES_JSON="$SAMPLES_JSON\"$(json_escape "$line")\"" +done < "$UNIQUE" +SAMPLES_JSON="$SAMPLES_JSON]" + +# --- Emit -------------------------------------------------------------------- +echo "=== detect-changed-samples: base=$BASE_REF root=$SAMPLES_ROOT ===" +echo "has_changes=$HAS_CHANGES" +echo "count=$COUNT" +echo "samples=$SAMPLES_JSON" +if [ "$COUNT" -gt 0 ]; then + echo "affected sample dirs:" + sed 's/^/ - /' "$UNIQUE" +else + echo "no affected samples (docs-only / nothing under a sample.yaml) — short-circuit" +fi + +if [ -n "${GITHUB_OUTPUT:-}" ]; then + { + echo "has_changes=$HAS_CHANGES" + echo "count=$COUNT" + echo "samples=$SAMPLES_JSON" + } >> "$GITHUB_OUTPUT" +fi + +if [ -n "$OUTPUT_DIR" ]; then + mkdir -p "$OUTPUT_DIR" || error "could not create --output-dir: $OUTPUT_DIR" + cp "$CHANGED_FILES" "$OUTPUT_DIR/changed_files.txt" + cp "$UNIQUE" "$OUTPUT_DIR/unique_changed_samples.txt" +fi + +exit 0 diff --git a/.github/scripts/test/fixtures/python-deps/requirements.txt b/.github/scripts/test/fixtures/python-deps/requirements.txt new file mode 100644 index 000000000..205fa6dd2 --- /dev/null +++ b/.github/scripts/test/fixtures/python-deps/requirements.txt @@ -0,0 +1,6 @@ +# P1.3 fixture (ADO 5449687): a REAL, normally-resolvable package so that a forced-unreachable +# index makes pip emit transport evidence AND its misleading "No matching distribution" +# conclusion — proving transport evidence wins (ERROR). Reused with PIP_NO_INDEX + an empty +# --find-links to prove an unaccompanied resolution failure stays FAIL. Never actually +# installed in either case. +requests diff --git a/.github/scripts/test/fixtures/typescript-deps/package.json b/.github/scripts/test/fixtures/typescript-deps/package.json new file mode 100644 index 000000000..6c400a13c --- /dev/null +++ b/.github/scripts/test/fixtures/typescript-deps/package.json @@ -0,0 +1,12 @@ +{ + "name": "typescript-deps", + "version": "1.0.0", + "private": true, + "description": "P1.3 fixture (ADO 5449687): declares a dependency so `npm install` must contact the registry. Pointed at a forced-unreachable registry to prove a transport failure classifies as ERROR (exit 2), never FAIL.", + "dependencies": { + "is-odd": "3.0.1" + }, + "scripts": { + "build": "node -e \"process.exit(0)\"" + } +} diff --git a/.github/scripts/test/run-detect-tests.sh b/.github/scripts/test/run-detect-tests.sh new file mode 100644 index 000000000..288341321 --- /dev/null +++ b/.github/scripts/test/run-detect-tests.sh @@ -0,0 +1,216 @@ +#!/usr/bin/env bash +# run-detect-tests.sh — hermetic exit gate for detect-changed-samples.sh. +# +# Sibling to run-tests.sh (the P1.1 validator harness). Where that harness needs real +# toolchains, THIS one needs only git + coreutils, so it runs fully GREEN both locally +# and on a runner — no BLOCKED-on-env cases. It builds a throwaway git repo with a +# crafted samples/ tree, makes controlled commits, and asserts the detector's outputs +# over specific base..HEAD ranges. +# +# Proves the three acceptance criteria of ADO 5449686 plus the fail-loud invariant: +# (a) a changed file inside a sample resolves to that sample's nearest sample.yaml dir +# (b) a docs-only change yields has_changes=false / samples=[] / exit 0 (short-circuit) +# (c) multiple changed files in one sample dir dedupe to a single entry +# (d) a deeply-nested changed file resolves to its nearest ancestor sample.yaml (walk-up) +# (e) two different samples changed -> both present, sorted +# (f) FAIL LOUD: a nonexistent base ref makes git diff error -> exit 1 (ADO 5247751) +# (g) base-ref RESOLUTION: pull_request -> origin/, push -> HEAD~1 (--print-base-ref) +set -uo pipefail + +# The unit harness asserts on the detector's STDOUT only. It must NOT leak into the +# runner's real $GITHUB_OUTPUT — otherwise the ~10 detector invocations below append +# has_changes/count/samples lines to the step's output file and GitHub rejects it +# ("Unable to process file command 'output'"). The GITHUB_OUTPUT emission is proven +# separately by the detect->consume plumbing job in the selftest workflow. +unset GITHUB_OUTPUT GITHUB_ENV GITHUB_STATE 2>/dev/null || true + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$HERE/../detect-changed-samples.sh" + +PASS_N=0 +FAIL_N=0 + +pass() { echo " PASS $1"; PASS_N=$((PASS_N + 1)); } +fail() { echo " FAIL $1"; FAIL_N=$((FAIL_N + 1)); } + +# run_detect [extra args...] -> populates OUT (stdout+stderr) and RC. +run_detect() { + local base="$1"; shift + OUT="$(bash "$SCRIPT" --base-ref "$base" "$@" 2>&1)" + RC=$? +} + +# field — extract the last `key=value` line the script printed to stdout. +field() { printf '%s\n' "$OUT" | sed -n "s/^$1=\(.*\)$/\1/p" | tail -1; } + +# expect_out — assert the `samples=` JSON equals expected. +expect_samples() { + local desc="$1" want="$2" got + got="$(field samples)" + if [ "$got" = "$want" ]; then pass "$desc (samples=$got)"; else fail "$desc expected samples=$want got=${got:-}"; fi +} +expect_field() { + local desc="$1" key="$2" want="$3" got + got="$(field "$key")" + if [ "$got" = "$want" ]; then pass "$desc ($key=$got)"; else fail "$desc expected $key=$want got=${got:-}"; fi +} +expect_rc() { + local desc="$1" want="$2" + if [ "$RC" = "$want" ]; then pass "$desc (exit=$RC)"; else fail "$desc expected exit=$want got=$RC"; fi +} + +# --- Build a throwaway git repo with a crafted samples/ tree ----------------- +REPO="$(mktemp -d)" +trap 'rm -rf "$REPO"' EXIT +cd "$REPO" || { echo "cannot cd to temp repo"; exit 1; } + +git init -q +git config user.email t@t.test +git config user.name test +git config commit.gpgsign false + +mkdir -p samples/python/quickstart/foo +mkdir -p samples/csharp/quickstart/bar +mkdir -p samples/python/deep/a/b/c/src +mkdir -p samples/docs # docs area under samples/, NO sample.yaml +printf 'name: foo\n' > samples/python/quickstart/foo/sample.yaml +printf 'print("v1")\n' > samples/python/quickstart/foo/main.py +printf 'name: bar\n' > samples/csharp/quickstart/bar/sample.yaml +printf 'x\n' > samples/csharp/quickstart/bar/Program.cs +printf 'y\n' > samples/csharp/quickstart/bar/extra.cs +printf 'name: deep\n' > samples/python/deep/a/b/c/sample.yaml +printf 'deep\n' > samples/python/deep/a/b/c/src/x.py +printf '# top\n' > samples/docs/README.md +printf '# repo\n' > README.md # root doc, entirely outside samples/ +git add -A >/dev/null +git commit -qm "base" +BASE="$(git rev-parse HEAD)" + +echo "==============================================================" +echo " detect-changed-samples: acceptance cases (git + coreutils only)" +echo "==============================================================" + +# (a) change a file inside foo -> resolves to foo +printf 'print("v2")\n' > samples/python/quickstart/foo/main.py +git commit -qam "touch foo/main.py" +run_detect "$BASE" +expect_rc "(a) changed sample -> ok" 0 +expect_field "(a) has_changes true" has_changes true +expect_field "(a) count 1" count 1 +expect_samples "(a) resolves to foo" '["samples/python/quickstart/foo"]' +BASE="$(git rev-parse HEAD)" + +# (b) docs-only: change a file under samples/ with no sample.yaml ancestor + a root doc +printf '# top v2\n' > samples/docs/README.md +printf '# repo v2\n' > README.md +git commit -qam "docs only" +run_detect "$BASE" +expect_rc "(b) docs-only -> ok" 0 +expect_field "(b) has_changes false" has_changes false +expect_field "(b) count 0" count 0 +expect_samples "(b) empty set" '[]' +BASE="$(git rev-parse HEAD)" + +# (c) multiple files in ONE sample dir -> dedupe to a single entry +printf 'x2\n' > samples/csharp/quickstart/bar/Program.cs +printf 'y2\n' > samples/csharp/quickstart/bar/extra.cs +git commit -qam "two files in bar" +run_detect "$BASE" +expect_rc "(c) multi-file one dir -> ok" 0 +expect_field "(c) count 1 (deduped)" count 1 +expect_samples "(c) single bar entry" '["samples/csharp/quickstart/bar"]' +BASE="$(git rev-parse HEAD)" + +# (d) deeply-nested changed file -> nearest ancestor sample.yaml (walk-up) +printf 'deep v2\n' > samples/python/deep/a/b/c/src/x.py +git commit -qam "nested deep change" +run_detect "$BASE" +expect_rc "(d) nested -> ok" 0 +expect_samples "(d) walks up to c" '["samples/python/deep/a/b/c"]' +BASE="$(git rev-parse HEAD)" + +# (e) two different samples changed -> both present, sorted (csharp < python) +printf 'print("v3")\n' > samples/python/quickstart/foo/main.py +printf 'x3\n' > samples/csharp/quickstart/bar/Program.cs +git commit -qam "two samples" +run_detect "$BASE" +expect_rc "(e) two samples -> ok" 0 +expect_field "(e) count 2" count 2 +expect_samples "(e) both, sorted" '["samples/csharp/quickstart/bar","samples/python/quickstart/foo"]' +BASE="$(git rev-parse HEAD)" + +echo "" +echo "==============================================================" +echo " FAIL-LOUD invariant (ADO 5247751)" +echo "==============================================================" +# (f) nonexistent base ref -> git diff errors -> exit 1, and NOT a fake empty set +OUT="$(bash "$SCRIPT" --base-ref does-not-exist-ref 2>&1)"; RC=$? +expect_rc "(f) bad base ref -> fail loud" 1 +if printf '%s\n' "$OUT" | grep -q "refusing to continue"; then + pass "(f) prints refusal (no fake empty set)" +else + fail "(f) missing refusal message" +fi + +echo "" +echo "==============================================================" +echo " Base-ref RESOLUTION (--print-base-ref, no network)" +echo "==============================================================" +# (g) pull_request event -> origin/; push -> HEAD~1 +G="$(bash "$SCRIPT" --event pull_request --target-branch main --print-base-ref 2>&1)"; GRC=$? +if [ "$GRC" = 0 ] && [ "$G" = "origin/main" ]; then pass "(g) pull_request -> origin/main"; else fail "(g) pull_request expected origin/main got '$G' (rc=$GRC)"; fi +G="$(bash "$SCRIPT" --event push --print-base-ref 2>&1)"; GRC=$? +if [ "$GRC" = 0 ] && [ "$G" = "HEAD~1" ]; then pass "(g) push -> HEAD~1"; else fail "(g) push expected HEAD~1 got '$G' (rc=$GRC)"; fi +# env fallback: GITHUB_BASE_REF supplies the target when --target-branch is absent +G="$(GITHUB_EVENT_NAME=pull_request GITHUB_BASE_REF=release bash "$SCRIPT" --print-base-ref 2>&1)"; GRC=$? +if [ "$GRC" = 0 ] && [ "$G" = "origin/release" ]; then pass "(g) env pull_request -> origin/release"; else fail "(g) env pull_request expected origin/release got '$G' (rc=$GRC)"; fi + +echo "" +echo "==============================================================" +echo " --output-dir artifacts" +echo "==============================================================" +# The last real diff (e) had 2 samples; re-run it with --output-dir and assert files. +ART="$(mktemp -d)" +bash "$SCRIPT" --base-ref "$(git rev-parse HEAD~1)" --output-dir "$ART" >/dev/null 2>&1 +if [ -f "$ART/unique_changed_samples.txt" ] && [ -f "$ART/changed_files.txt" ]; then + if grep -q "samples/python/quickstart/foo" "$ART/unique_changed_samples.txt"; then + pass "--output-dir wrote unique_changed_samples.txt" + else + fail "--output-dir unique_changed_samples.txt missing expected entry" + fi +else + fail "--output-dir did not write expected artifact files" +fi +rm -rf "$ART" + +echo "" +echo "==============================================================" +echo " GITHUB_OUTPUT hygiene on EMPTY result (regression: bare '0')" +echo "==============================================================" +# An empty (docs-only) result must write ONLY clean key=value lines to the real +# $GITHUB_OUTPUT. Regression guard for the grep -c double-'0' bug that emitted a +# bare "0" line and made GitHub reject the step ("Invalid format '0'"). +# Diffing HEAD against itself is a guaranteed empty-but-successful diff. +GO="$(mktemp)" +GITHUB_OUTPUT="$GO" bash "$SCRIPT" --base-ref HEAD >/dev/null 2>&1 +if grep -qxE '[0-9]+' "$GO"; then + fail "(h) empty result leaked a bare numeric line into GITHUB_OUTPUT: $(tr '\n' '|' < "$GO")" +elif [ "$(grep -c '^count=0$' "$GO")" = 1 ] \ + && grep -q '^has_changes=false$' "$GO" \ + && grep -q '^samples=\[\]$' "$GO"; then + pass "(h) empty result writes clean count=0 / has_changes=false / samples=[]" +else + fail "(h) empty-result GITHUB_OUTPUT malformed: $(tr '\n' '|' < "$GO")" +fi +rm -f "$GO" + +echo "" +echo "===================================================" +echo " checks passed: $PASS_N failed: $FAIL_N" +echo "===================================================" + +if [ "$FAIL_N" -ne 0 ]; then + echo "detect exit gate: RED — a check failed." + exit 1 +fi +echo "detect exit gate: GREEN — all detection acceptance cases + fail-loud proved." diff --git a/.github/scripts/test/run-tests.sh b/.github/scripts/test/run-tests.sh index bef0b8a3f..4531c6a30 100644 --- a/.github/scripts/test/run-tests.sh +++ b/.github/scripts/test/run-tests.sh @@ -12,6 +12,10 @@ # - guardrails: unknown language, missing --sample-dir, missing --language => error # - --results-dir plumbing lands each sample path in passed/failed/errored.txt # +# Plus P1.3 (ADO 5449687) dependency-install transport classification: +# - pip/npm install against a forced-unreachable registry (127.0.0.1:1) => error (exit 2) +# - an unaccompanied resolution failure (package not found, no transport) => fail (exit 1) +# # Designed to run on a GitHub Actions ubuntu-latest runner AFTER setup-dotnet / # setup-python / setup-node / setup-java / setup-go have put the toolchains on PATH # (that is the "callable from GH Actions on a real runner" proof). It also runs @@ -167,6 +171,45 @@ if [ "$YQ_OK" = true ]; then assert_file_has "$RESULTS/failed.txt" "$FIX/csharp-yaml-broken" fi +echo "" +echo "==============================================================" +echo " Dependency-install transport classification (P1.3 / ADO 5449687)" +echo "==============================================================" +# Proves the failure-vs-error split on dependency RESOLUTION — the P1.3 gap: +# - a forced-unreachable registry (localhost blackhole) => ERROR (exit 2), NEVER fail. +# - an unaccompanied resolution failure (package not found) => FAIL (exit 1). +# Both are hermetic: 127.0.0.1:1 has nothing listening (=> connection refused), and the FAIL +# case uses PIP_NO_INDEX + an empty --find-links so pip resolves nothing WITHOUT any network. +EMPTY_LINKS="$(mktemp -d)" + +# pip: blackhole index => transport evidence => ERROR. requirements.txt names a REAL package, +# so pip ALSO prints its misleading "No matching distribution" conclusion after exhausting +# retries — this case proves transport evidence WINS over that generic line (the precedence fix). +# The FAIL case (no index at all) prints the SAME "No matching distribution" WITHOUT transport +# evidence => must stay FAIL. Same fixture, opposite verdicts: the whole point of P1.3. +if toolchain_ready python; then + check "pip install: registry unreachable -> error" 2 error -- \ + env "PIP_INDEX_URL=http://127.0.0.1:1" "PIP_RETRIES=1" "PIP_DEFAULT_TIMEOUT=5" \ + bash "$SCRIPT" --language python --sample-dir "$FIX/python-deps" --results-dir "$RESULTS" + check "pip install: unresolvable package (no transport) -> fail" 1 fail -- \ + env "PIP_NO_INDEX=1" "PIP_FIND_LINKS=$EMPTY_LINKS" \ + bash "$SCRIPT" --language python --sample-dir "$FIX/python-deps" --results-dir "$RESULTS" +else + echo " SKIP pip transport cases — python toolchain absent (BLOCKED-on-env; the runner has it)" + SKIP_N=$((SKIP_N + 2)) +fi + +# npm: blackhole registry => connect ECONNREFUSED => ERROR (exit 2). +if toolchain_ready typescript; then + check "npm install: registry unreachable -> error" 2 error -- \ + env "NPM_CONFIG_REGISTRY=http://127.0.0.1:1" "NPM_CONFIG_FETCH_RETRIES=0" \ + bash "$SCRIPT" --language typescript --sample-dir "$FIX/typescript-deps" --results-dir "$RESULTS" +else + echo " SKIP npm transport case — node/npm toolchain absent (BLOCKED-on-env; the runner has it)" + SKIP_N=$((SKIP_N + 1)) +fi +rm -rf "$EMPTY_LINKS" + echo "" echo "===================================================" echo " checks passed: $PASS_N failed: $FAIL_N skipped: $SKIP_N" diff --git a/.github/scripts/validate-sample.sh b/.github/scripts/validate-sample.sh index 973410df2..71577b242 100644 --- a/.github/scripts/validate-sample.sh +++ b/.github/scripts/validate-sample.sh @@ -25,13 +25,21 @@ # (a sample.yaml command exits non-zero; or the language default's # compile/build/py_compile exits non-zero) # 2 ERROR OUR INFRA is sick — page us, do not quarantine -# (bad/missing args, unknown language, missing sample dir, unreadable or -# malformed sample.yaml, a required toolchain binary missing from PATH, -# or yq unavailable when a sample.yaml must be read) +# PRECONDITION errors (bad/missing args, unknown language, missing sample dir, +# unreadable or malformed sample.yaml, a required toolchain binary missing from +# PATH, yq unavailable when a sample.yaml must be read), OR a RUNTIME transport +# failure: a `pip install` / `npm install` that failed with positive +# transport/registry evidence (DNS failure, connection refused/reset, connect/read +# timeout, or a registry 5xx). See dep_infra_signature() + CLASSIFICATION.md. # -# Gray area (honest v1): dependency-resolution failures (pip install / npm install) -# stay FAIL — usually the sample's declared deps are wrong — but are log-tagged -# `[classify:gray]` so a later advisory pass can audit registry/network mis-classifications. +# failure-vs-error on dependency install (honest v1): only the two DEDICATED install steps +# (`pip install`, `npm install`) are inspected. A positive transport signature => ERROR (this +# wins even when pip later prints a generic "No matching distribution" line, which it emits +# AFTER exhausting retries against an unreachable index). An UNACCOMPANIED resolution error +# (bad/missing package, version conflict) => FAIL. Anything unmatched => FAIL (bias +# ambiguous->fail, never fail-open to pass). The merged resolve+compile tools (dotnet build, +# mvn compile, gradle build, go build, npm run build) are NOT inspected in v1 — their output +# mixes restore+compile+user-script, so they stay FAIL. Documented limit in CLASSIFICATION.md. # # Outputs: # - prints `verdict=pass|fail|error` on stdout (also appended to $GITHUB_OUTPUT if set) @@ -111,6 +119,49 @@ ensure_yq() { command -v yq >/dev/null 2>&1 || error "yq not available on PATH (required to read sample.yaml)" } +# dep_infra_signature : return 0 IFF the captured dependency-install log shows +# HIGH-CONFIDENCE transport/registry evidence (DNS / connection refused-reset / connect-read +# timeout / registry 5xx). This is a deliberately NARROW heuristic, not a robust classifier: +# - it matches only tool-specific transport strings, never generic words like "timeout"; +# - a positive match wins even when pip later prints a generic "No matching distribution" +# line (pip emits that AFTER exhausting retries against an unreachable index); +# - it is scoped to the direct `pip install` / `npm install` steps only; +# - anything unmatched stays FAIL (bias ambiguous->fail, never fail-open to pass); +# - EXPECT maintenance as runner pip/npm versions drift their error strings. +# The full captured log is printed by the caller before classifying, so diagnosis is preserved. +dep_infra_signature() { + local log="${1:-}" + [ -n "$log" ] && [ -f "$log" ] || return 1 + grep -Eiq \ + -e 'temporary failure in name resolution' \ + -e 'could not resolve host' \ + -e 'name or service not known' \ + -e 'getaddrinfo' \ + -e 'eai_again' \ + -e 'enotfound' \ + -e 'econnrefused' \ + -e 'connection refused' \ + -e 'econnreset' \ + -e 'connection reset' \ + -e 'etimedout' \ + -e 'connection timed out' \ + -e 'read timed out' \ + -e 'readtimeouterror' \ + -e 'connecttimeouterror' \ + -e 'network timeout' \ + -e 'newconnectionerror' \ + -e 'max retries exceeded' \ + -e 'failed to establish a new connection' \ + -e 'proxyerror' \ + -e 'network is unreachable' \ + -e '50[0234] server error' \ + -e 'service unavailable' \ + -e 'bad gateway' \ + -e 'gateway time-?out' \ + -e 'internal server error' \ + "$log" +} + # --- Part 1: the once-only shared sample.yaml runner ------------------------- # Byte-identical across all five ADO jobs today. Reads build -> validate -> test # and runs them in order; the first non-zero command is a sample FAIL. @@ -177,10 +228,18 @@ default_validate_csharp() { default_validate_python() { if [ -f "$SAMPLE_DIR/requirements.txt" ]; then echo "Installing requirements..." - if ! pip install -r "$SAMPLE_DIR/requirements.txt" -q; then - echo "[classify:gray] dependency-resolution failure (pip install) — kept FAIL per v1 gray-area rule" + local piplog; piplog="$(mktemp)" + if ! pip install -r "$SAMPLE_DIR/requirements.txt" -q >"$piplog" 2>&1; then + cat "$piplog" + if dep_infra_signature "$piplog"; then + rm -f "$piplog" + error "pip install: registry/network unreachable (transport evidence in log)" + fi + rm -f "$piplog" + echo "[classify:fail] pip install dependency-resolution failure (no transport evidence)" fail "pip install failed" fi + rm -f "$piplog" fi local pyfile for pyfile in "$SAMPLE_DIR"/*.py; do @@ -198,10 +257,21 @@ default_validate_typescript() { if [ -f "$SAMPLE_DIR/package.json" ]; then require_tool npm echo "Installing dependencies..." - if ! ( cd "$SAMPLE_DIR" && npm install --silent ); then - echo "[classify:gray] dependency-resolution failure (npm install) — kept FAIL per v1 gray-area rule" + local npmlog; npmlog="$(mktemp)" + # NOT --silent: that suppresses error output too, which would blind + # dep_infra_signature to transport errors. --loglevel=error keeps success + # quiet while still emitting the network/registry failure text we classify on. + if ! ( cd "$SAMPLE_DIR" && npm install --no-audit --no-fund --loglevel=error ) >"$npmlog" 2>&1; then + cat "$npmlog" + if dep_infra_signature "$npmlog"; then + rm -f "$npmlog" + error "npm install: registry/network unreachable (transport evidence in log)" + fi + rm -f "$npmlog" + echo "[classify:fail] npm install dependency-resolution failure (no transport evidence)" fail "npm install failed" fi + rm -f "$npmlog" if ( cd "$SAMPLE_DIR" && npm run build --if-present ); then pass else diff --git a/.github/workflows/detect-changed-samples-selftest.yml b/.github/workflows/detect-changed-samples-selftest.yml new file mode 100644 index 000000000..04b3afb31 --- /dev/null +++ b/.github/workflows/detect-changed-samples-selftest.yml @@ -0,0 +1,146 @@ +name: detect-changed-samples self-test + +# INERT DEV HARNESS — proves .github/scripts/detect-changed-samples.sh (the P1.2 in-job +# path detector) is callable from a real GitHub Actions runner, that its acceptance cases +# hold, and that its results flow through GH Actions job outputs into a downstream job via +# needs..outputs.* (the exact plumbing P2.1/P2.2 validate lanes will consume). +# +# Scoped so it NEVER fires on real public-repo traffic: +# - workflow_dispatch: inert stub for the FINAL intended manual trigger. (Dispatch only +# works once the workflow lives on the default branch, which we are NOT touching yet.) +# - pull_request scoped to base == the integration branch below. Real public PRs target +# `main`, so they can never match — only a throwaway internal PR whose BASE is the +# integration branch runs this. +# +# Deliberately NO `on: paths:` filter (frozen constraint): "what changed" is decided IN-JOB +# by the detector script, never by a trigger-path shim. Adding paths here would reintroduce +# the rejected required-check brick + gate-shim. +# +# Credential-free by construction: permissions is contents:read only. No secrets, no OIDC, +# no privileged runners. Do NOT add broad push/schedule triggers here — activation on main +# is a separate, final, reviewed change. +on: + workflow_dispatch: + pull_request: + branches: [ brandom-msft-brandom-public-validation-pipeline ] + +permissions: + contents: read + +jobs: + # --- Unit proof: the detector's acceptance cases + fail-loud, hermetic (git only) ------- + detect-selftest: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Detect exit gate — resolve/dedupe/walk-up/docs-only/fail-loud + run: bash .github/scripts/test/run-detect-tests.sh + + # --- Plumbing proof: detect -> job outputs -> downstream job reads needs.*.outputs ------ + detect: + runs-on: ubuntu-latest + outputs: + has_changes: ${{ steps.detect.outputs.has_changes }} + count: ${{ steps.detect.outputs.count }} + samples: ${{ steps.detect.outputs.samples }} + steps: + - uses: actions/checkout@v4 + - name: Build a deterministic temp scenario and run the detector + id: detect + run: | + set -euo pipefail + SCRIPT="$GITHUB_WORKSPACE/.github/scripts/detect-changed-samples.sh" + REPO="$(mktemp -d)" + cd "$REPO" + git init -q + git config user.email t@t.test + git config user.name test + mkdir -p samples/python/quickstart/foo samples/csharp/quickstart/bar + printf 'name: foo\n' > samples/python/quickstart/foo/sample.yaml + printf 'print("v1")\n' > samples/python/quickstart/foo/main.py + printf 'name: bar\n' > samples/csharp/quickstart/bar/sample.yaml + printf 'x\n' > samples/csharp/quickstart/bar/Program.cs + git add -A >/dev/null; git commit -qm base + BASE="$(git rev-parse HEAD)" + printf 'print("v2")\n' > samples/python/quickstart/foo/main.py + printf 'x2\n' > samples/csharp/quickstart/bar/Program.cs + git commit -qam change + # Runs with the REAL $GITHUB_OUTPUT -> emits has_changes/count/samples as STEP outputs. + bash "$SCRIPT" --base-ref "$BASE" + + consume: + needs: detect + runs-on: ubuntu-latest + steps: + - name: Read needs.detect.outputs.* and assert + env: + HAS_CHANGES: ${{ needs.detect.outputs.has_changes }} + COUNT: ${{ needs.detect.outputs.count }} + SAMPLES: ${{ needs.detect.outputs.samples }} + run: | + set -euo pipefail + echo "consumed via needs.detect.outputs.*:" + echo " has_changes=$HAS_CHANGES" + echo " count=$COUNT" + echo " samples=$SAMPLES" + fail=0 + [ "$HAS_CHANGES" = "true" ] || { echo "FAIL: has_changes expected true"; fail=1; } + [ "$COUNT" = "2" ] || { echo "FAIL: count expected 2"; fail=1; } + expected='["samples/csharp/quickstart/bar","samples/python/quickstart/foo"]' + [ "$SAMPLES" = "$expected" ] || { echo "FAIL: samples expected $expected"; fail=1; } + if [ "$fail" -ne 0 ]; then + echo "needs.*.outputs plumbing proof: RED" + exit 1 + fi + echo "needs.*.outputs plumbing proof: GREEN — downstream job consumed the detected set." + + # --- Docs-only proof: an EMPTY result must flow through job outputs WITHOUT --- + # --- corrupting the real $GITHUB_OUTPUT (regression guard for the bare-'0' bug, + # --- and the end-to-end proof of acceptance criterion (b): docs-only short-circuit). + detect-docs-only: + runs-on: ubuntu-latest + outputs: + has_changes: ${{ steps.detect.outputs.has_changes }} + count: ${{ steps.detect.outputs.count }} + samples: ${{ steps.detect.outputs.samples }} + steps: + - uses: actions/checkout@v4 + - name: Build a docs-only temp scenario and run the detector + id: detect + run: | + set -euo pipefail + SCRIPT="$GITHUB_WORKSPACE/.github/scripts/detect-changed-samples.sh" + REPO="$(mktemp -d)" + cd "$REPO" + git init -q + git config user.email t@t.test + git config user.name test + mkdir -p samples/docs + printf '# top\n' > samples/docs/README.md # under samples/, NO sample.yaml + git add -A >/dev/null; git commit -qm base + BASE="$(git rev-parse HEAD)" + printf '# top v2\n' > samples/docs/README.md + git commit -qam "docs only" + # Real $GITHUB_OUTPUT: an empty result must NOT corrupt it (was: bare '0'). + bash "$SCRIPT" --base-ref "$BASE" + + consume-docs-only: + needs: detect-docs-only + runs-on: ubuntu-latest + steps: + - name: Assert the docs-only short-circuit contract via needs.*.outputs + env: + HAS_CHANGES: ${{ needs.detect-docs-only.outputs.has_changes }} + COUNT: ${{ needs.detect-docs-only.outputs.count }} + SAMPLES: ${{ needs.detect-docs-only.outputs.samples }} + run: | + set -euo pipefail + echo "docs-only outputs: has_changes=$HAS_CHANGES count=$COUNT samples=$SAMPLES" + fail=0 + [ "$HAS_CHANGES" = "false" ] || { echo "FAIL: has_changes expected false"; fail=1; } + [ "$COUNT" = "0" ] || { echo "FAIL: count expected 0"; fail=1; } + [ "$SAMPLES" = "[]" ] || { echo "FAIL: samples expected []"; fail=1; } + if [ "$fail" -ne 0 ]; then echo "docs-only short-circuit proof: RED"; exit 1; fi + echo "docs-only short-circuit proof: GREEN — empty set flows through needs.*.outputs uncorrupted." diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml new file mode 100644 index 000000000..eab369f7b --- /dev/null +++ b/.github/workflows/validate.yml @@ -0,0 +1,137 @@ +name: validate + +# The REAL PR-triggered L3 "floor" lane (P2.1, ADO 5449695) — distinct from the inert +# *-selftest.yml dev harnesses. It validates every changed sample to Build Readiness +# Level 3 (Load) using the shared scripts ported in Phase 1. +# +# FORK-SAFE BY CONSTRUCTION (this is the headline property): +# - on: pull_request (NEVER pull_request_target) — so PRs from forks run this with a +# read-only GITHUB_TOKEN and NO access to secrets, per GitHub's documented behavior. +# - permissions: contents: read only. No `environment:`. No secret references anywhere. +# - The job checks out and EXECUTES untrusted fork code (dotnet build, npm install, +# pip install, mvn compile, ...). That is intentional and safe *because* the runner is +# barren: nothing here can exfiltrate a credential, because there is no credential. +# +# Frozen constraints honored: no pull_request_target, no trust-scoring, no org-membership +# API calls, no aggregator/gate-shim job, no `on: paths:` trigger filter. "What changed" is +# decided IN-JOB by detect-changed-samples.sh and fanned out via needs.*.outputs. The +# trusted L4 lane (secrets, fork-gated) is a SEPARATE workflow (P2.2) — not built here. + +on: + pull_request: + # No `branches:` filter (runs for every PR, incl. those targeting main) and no `paths:` + # filter (path detection is data-driven in-job, not a trigger shim). No draft guard — + # the floor lane runs on all PRs including drafts; draft-skip is a P2.2 concern. + +permissions: + contents: read + +jobs: + # --- Which samples did this PR touch? (in-job, data-driven) ------------------- + detect: + runs-on: ubuntu-latest + outputs: + has_changes: ${{ steps.detect.outputs.has_changes }} + count: ${{ steps.detect.outputs.count }} + samples: ${{ steps.detect.outputs.samples }} + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # detector diffs origin/..HEAD; needs full history + - name: Detect changed samples + id: detect + # Reads $GITHUB_EVENT_NAME (pull_request) + $GITHUB_BASE_REF (target branch), diffs + # scoped to samples/, and emits has_changes/count/samples to $GITHUB_OUTPUT. Fails + # loud (exit 1) if the git diff itself errors — never a fake empty set. + run: bash .github/scripts/detect-changed-samples.sh + + # --- Validate each changed sample to L3, one runner per sample --------------- + validate: + needs: detect + if: needs.detect.outputs.has_changes == 'true' # docs-only PRs skip this job entirely + strategy: + fail-fast: false # one broken sample must not cancel the rest + matrix: + sample: ${{ fromJSON(needs.detect.outputs.samples) }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Derive language from sample path + id: lang + # matrix.sample is an untrusted, PR-controlled path — pass it via env and quote it, + # never interpolate ${{ }} straight into the script body (injection-safe). + env: + SAMPLE: ${{ matrix.sample }} + run: | + set -euo pipefail + rest="${SAMPLE#samples/}" + dir="${rest%%/*}" + case "$dir" in + csharp) language=csharp ;; + python) language=python ;; + typescript) language=typescript ;; + javascript) language=typescript ;; # JS validated via the TS default (node/npm) + java) language=java ;; + go) language=go ;; + *) language=unsupported ;; # not in the frozen 5-lang set — skip, don't error + esac + echo "language=$language" >> "$GITHUB_OUTPUT" + echo "sample=$SAMPLE dir=$dir -> language=$language" + + # Toolchain setup — GH-native setup-* actions, versions pinned to match the P1.4 audit. + # Each installs ONLY for the language this leg actually needs (barren-by-default). + - name: Set up .NET + if: steps.lang.outputs.language == 'csharp' + uses: actions/setup-dotnet@v4 + with: + dotnet-version: '8.0.x' + + - name: Set up Python + if: steps.lang.outputs.language == 'python' + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Set up Node.js + if: steps.lang.outputs.language == 'typescript' + uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Set up Java (Temurin) + if: steps.lang.outputs.language == 'java' + uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: '17' + + - name: Set up Go + if: steps.lang.outputs.language == 'go' + uses: actions/setup-go@v5 + with: + go-version: '1.22' + + - name: Install yq + if: steps.lang.outputs.language != 'unsupported' + run: | + sudo wget -qO /usr/local/bin/yq \ + https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 + sudo chmod +x /usr/local/bin/yq + yq --version + + - name: Skip unsupported language + if: steps.lang.outputs.language == 'unsupported' + env: + SAMPLE: ${{ matrix.sample }} + run: echo "::notice::Skipping ${SAMPLE} — language not covered by the L3 floor lane." + + - name: Validate sample (L3 / Load) + if: steps.lang.outputs.language != 'unsupported' + env: + SAMPLE: ${{ matrix.sample }} + LANGUAGE: ${{ steps.lang.outputs.language }} + # validate-sample.sh owns the verdict: exit 0 = pass, 1 = sample broken (FAIL), + # 2 = infra sick (ERROR). Both 1 and 2 make this leg red — the floor never goes + # green on an unresolved run — while the script's log distinguishes them. + run: bash .github/scripts/validate-sample.sh --language "$LANGUAGE" --sample-dir "$SAMPLE" diff --git a/samples/python/quickstart/chat-with-agent/quickstart-chat-with-agent.py b/samples/python/quickstart/chat-with-agent/quickstart-chat-with-agent.py index 0897dd974..5b8b0a078 100644 --- a/samples/python/quickstart/chat-with-agent/quickstart-chat-with-agent.py +++ b/samples/python/quickstart/chat-with-agent/quickstart-chat-with-agent.py @@ -1,3 +1,4 @@ +def broken(: # P2.1 probe: deliberate syntax error (known-broken -> expect L3 FAIL, exit 1) from azure.identity import DefaultAzureCredential from azure.ai.projects import AIProjectClient diff --git a/samples/python/quickstart/create-agent/quickstart-create-agent.py b/samples/python/quickstart/create-agent/quickstart-create-agent.py index 9243a734b..09e2c83cc 100644 --- a/samples/python/quickstart/create-agent/quickstart-create-agent.py +++ b/samples/python/quickstart/create-agent/quickstart-create-agent.py @@ -1,3 +1,4 @@ +# P2.1 probe: harmless comment to trigger detection (known-good -> expect L3 PASS) from azure.identity import DefaultAzureCredential from azure.ai.projects import AIProjectClient from azure.ai.projects.models import PromptAgentDefinition