Move the canonical SAM app template to aws-infra without moving app deployment

[alexeygrigorev]

Move the canonical SAM app template to aws-infra without moving app deployment

Status: blocked — aws-infra preparation exists but must be refreshed after #166 Phase D and cleanup
Tags: enhancement, infra, testing, P1
Depends on: #166
Blocks: None
Next owner: Product Manager after #166 cleanup, then Software Engineer in clean isolated worktrees

Current prepared state

• A local linked aws-infra worktree on branch issue-173 contains commit 5dd8a1a0d118ed116c1833538a499fbfde46b51b, which added sandbox/dataops/template.application.yaml. That commit is inert preparation only: it added no application deployment workflow, made no AWS call, and is not an approved DataOps pin.
• The prepared template was byte-identical to the DataOps source when copied, but it is no longer final. Current DataOps source contains later application-stack changes, including the accepted Tasks end-state work. The prepared snapshot must be refreshed from the exact post-Replace the transitional Tasks table with the final cardId schema #166 cleanup template and reviewed in aws-infra; do not merge or pin 5dd8a1a as the release template.
• Isolate SAM dependency installation from the shared developer workspace #185 is Tester/PM accepted and committed in DataOps as 247ea7cd1a83db0018418276f138218f372a9b1c. Its isolated, content-addressed Lambda builder is now a required accepted ancestor of Move the canonical SAM app template to aws-infra without moving app deployment #173. The cross-repository template resolver must feed SAM without reintroducing root dependency mutation, a checkout install in the deploy job, or a second build.
• Both shared checkouts contain unrelated work. Final implementation remains isolated-worktree work with explicit paths; no cleanup, checkout, stash, or commit of another session’s files is authorized.

Scope

Move the only tracked application-stack template from dataops/infra/template.full.yaml to aws-infra/sandbox/dataops/template.application.yaml. aws-infra owns the canonical CloudFormation/SAM definition, but it does not gain an application deployment workflow and a push there does not touch AWS.

DataTalksClub/dataops remains the sole app build and deploy authority. Its existing main-branch GitHub Actions workflow checks out a committed, immutable aws-infra revision, builds the DataOps source against that exact template, verifies the artifact, and deploys .aws-sam/build/template.yaml through the existing GitHub Actions OIDC role.

The concrete boundary is:

1. Add a single 40-character aws-infra commit pin to DataOps (for example .github/aws-infra.ref). Never follow main, a tag, or another mutable ref during a deployment.
2. Because DataTalksClub/aws-infra is private, both DataOps workflow jobs use a dedicated repository secret named AWS_INFRA_READ_TOKEN: a fine-grained token limited to DataTalksClub/aws-infra with repository Contents read-only. The checkout uses the committed SHA and persist-credentials: false. Do not reuse the runtime knowledge token and do not add AWS keys.
3. A DataOps-owned resolver verifies the pin format and materializes the pinned sandbox/dataops/template.application.yaml git blob under ignored .tmp/. CI checks out that exact revision first; local development may read the same git object from the sibling ../aws-infra clone without checking out, stashing, or modifying its current branch. The materialized file must be byte-identical to the pinned blob.
4. The moved template declares each Lambda CodeUri relative to the DataOps build base (infra/sam-build). make sam-build passes both the resolved template path and --base-dir set to the DataOps repository root, so the tiny makefile adapter continues to invoke the just-checked-out DataOps bundler. The accepted Isolate SAM dependency installation from the shared developer workspace #185 builder remains the sole content-addressed Lambda build: it treats the DataOps checkout as immutable input, installs/builds only inside its fingerprint-owned workspace, emits six independent artifacts, and may use a validated warm cache. The move must not package code from aws-infra or another DataOps revision, add a deploy-job install, or add a second SAM/backend build.
5. samconfig.toml no longer names the removed local source template. The deploy configuration continues to deploy only .aws-sam/build/template.yaml, which is the transformed output from the exact template/code pair just validated and built.

Acceptance Criteria

• aws-infra/sandbox/dataops/template.application.yaml is the sole tracked canonical application-stack template; dataops/infra/template.full.yaml is deleted and no generated or fallback copy is committed to DataOps.
• DataOps records one immutable 40-character aws-infra commit SHA, and validation fails before build/AWS credential exchange when the pin is malformed, unavailable, or does not contain the expected template path.
• The pinned revision is checked out read-only in both the checks and deploy jobs using AWS_INFRA_READ_TOKEN and persist-credentials: false; neither job follows an aws-infra branch head.
• [HUMAN] AWS_INFRA_READ_TOKEN exists in the DataOps repository and is limited to Contents read-only on the single private DataTalksClub/aws-infra repository.
• Local resolution reads the pinned git object from ../aws-infra without switching branches or changing/stashing unrelated dirty files; CI materializes the same bytes under DataOps .tmp/.
• SAM validation and the one shared esbuild build consume the same resolved template bytes. All six functions resolve infra/sam-build against the DataOps root and retain their declared bundled handlers.
• The paired DataOps release retains Isolate SAM dependency installation from the shared developer workspace #185 commit 247ea7cd1a83db0018418276f138218f372a9b1c in ancestry and preserves its isolated checkout-safe cold build, validated warm cache, six independent artifacts, one deploy SAM build, and no deploy-job npm ci.
• The deploy step still consumes .aws-sam/build/template.yaml, uses the existing dataops-github-actions-deploy OIDC role, and contains no static AWS credentials or manual app-deploy path.
• A push to aws-infra alone cannot trigger or perform a DataOps app deployment. A template change becomes deployable only after DataOps deliberately updates its pin and pushes that reviewed commit to main.
• Template-path tests, workflow contract tests, samconfig.toml, AGENTS.md, docs/local-development.md, the runtime architecture docs, and aws-infra/sandbox/dataops/README.md describe and enforce the new ownership/pin/build boundary with no stale local-template instructions.
• The DataOps infra/planning tests load the pinned canonical template rather than a fixture copy, and the aws-infra test suite covers the new canonical file at its owning path.
• make sam-validate, make sam-build, all packaged-artifact gates, backend tests/typecheck/build, DataOps infra/planning tests, and the affected aws-infra tests pass from clean worktrees.
• After the DataOps pin/deletion commit reaches main, On-Call evidence shows the existing OIDC workflow deployed the expected DataOps SHA plus pinned aws-infra SHA, CloudFormation reached UPDATE_COMPLETE, and production smoke passed.

Test Scenarios

Scenario: exact pinned cross-repository build

Given DataOps records an aws-infra SHA whose tree contains the canonical template
When the checks and deploy jobs run
Then each fetches that exact private commit, validates identical template bytes, and the deploy job packages the current DataOps source once against it before deploying the built template.

Scenario: app code changes without an infrastructure change

Given the pin is unchanged
When a DataOps application commit reaches main
Then the normal DataOps workflow builds and deploys that new app code using the same pinned template and existing OIDC role.

Scenario: infrastructure changes do not self-deploy

Given a new application-template commit reaches aws-infra/main
When no DataOps pin update is pushed
Then no DataOps workflow runs and no application stack changes.

Scenario: invalid or unavailable pin fails closed

Given the committed pin is malformed, missing from the private remote, or lacks sandbox/dataops/template.application.yaml
When validation begins
Then the workflow fails before SAM build and before AWS credential exchange; it never falls back to a branch head or a local DataOps template.

Scenario: local sibling repo has unrelated work

Given ../aws-infra is on another branch with unrelated modified/untracked files
When a developer runs the resolver and SAM checks
Then it reads the template blob at the pinned commit without checking out, cleaning, stashing, or editing that working tree.

Scenario: rollback a paired release

Given a newly pinned template/code pair fails deployment or smoke verification
When the DataOps release commit is reverted
Then the revert restores the previous compatible app code and aws-infra pin together, and the ordinary DataOps main-branch OIDC workflow redeploys that known-good pair. aws-infra history is not rewritten and no manual SAM deploy is introduced.

Cross-repository commit and release sequence

1. Complete Replace the transitional Tasks table with the final cardId schema #166 first so the final card/task schema is canonical before the template moves.
2. Create clean isolated worktrees for both repositories. The current shared DataOps and aws-infra checkouts contain unrelated work; do not clean, stash, overwrite, stage, or commit it.
3. After Replace the transitional Tasks table with the final cardId schema #166 cleanup, in the clean aws-infra worktree refresh the prepared template.application.yaml from the exact final DataOps source, add ownership documentation and owning tests, and obtain lifecycle review before committing and pushing it. Do not use the stale 5dd8a1a snapshot as the pin. The refreshed aws-infra commit is inert because aws-infra has no app deploy workflow.
4. Confirm the read-only cross-repository token is configured, then in the clean DataOps worktree record that exact already-pushed SHA, delete the local template, and update the resolver, Make/SAM/workflow paths, tests, and docs in one reviewed release increment.
5. Push DataOps main. Only this push triggers the normal OIDC app deployment. Record both repository SHAs in the issue/on-call evidence.
6. For every later template change, repeat the same order: push reviewed aws-infra template commit, then update the immutable pin in a reviewed DataOps commit. Coordinated app/template changes keep the code and pin in the same DataOps release commit.

Out of Scope

• Giving aws-infra CI/CD authority to deploy the DataOps application.
• Manual application deployment or changing the existing GitHub Actions OIDC AWS trust model.
• Copying or vendoring a tracked template back into DataOps, following aws-infra/main, or adding a fallback template.
• Reusing the content/knowledge GitHub token or granting write/admin/workflow access to the cross-repository checkout token.
• Changing Lambda behavior, events, IAM policies, environment variables, tables, indexes, persisted data, or resource names as part of the file move.
• Moving the DataOps bundler or infra/sam-build packaging adapter into aws-infra.
• Cleaning or committing either repository's unrelated existing worktree changes.
• Publicizing aws-infra or creating an aws-infra application-deploy workflow.

Dependencies

• Bundle the backend with esbuild to reduce deployment size and time #172 is complete and provides the single shared esbuild artifact plus the small infra/sam-build adapter.
• Isolate SAM dependency installation from the shared developer workspace #185 is satisfied by accepted DataOps commit 247ea7cd1a83db0018418276f138218f372a9b1c. The final Move the canonical SAM app template to aws-infra without moving app deployment #173 DataOps commit must retain that exact commit in ancestry and preserve its isolated cold build, validated warm cache, six independent artifacts, one deploy SAM build, and no deploy-job npm ci.
• Replace the transitional Tasks table with the final cardId schema #166 remains the engineering blocker. No production cutover phase has run. Finish the repaired A → B → C sequence, the in-sequence Gate canonical Card lifecycle on the empty production Cards table #182 final Cards preflight, Phase D, production evidence, and the accepted cleanup that removes every A/B/C/D guard and restores ordinary main-push OIDC deployment. Move the canonical SAM app template to aws-infra without moving app deployment #173 must consume the resulting steady-state application template and workflow; it must not move, pin, or teach the resolver about a transitional cutover template.
• Gate canonical Card lifecycle on the empty production Cards table #182 is not a separate template move or table replacement dependency. Its pass/fail decision occurs inside the final Replace the transitional Tasks table with the final cardId schema #166 activation sequence; a pass keeps the existing Cards table, while a failure stops and re-grooms Replace the transitional Tasks table with the final cardId schema #166 before Move the canonical SAM app template to aws-infra without moving app deployment #173 can resume.
• Engineering resume condition: Replace the transitional Tasks table with the final cardId schema #166 is closed, its post-success cleanup is committed, and the normal DataOps push workflow is restored. Then refresh sandbox/dataops/template.application.yaml from that exact final source in a clean aws-infra worktree and complete the two-repository implementation/review.
• Release resume condition: the refreshed, reviewed aws-infra application-template commit is already pushed, and a repository administrator confirms AWS_INFRA_READ_TOKEN exists with Contents read-only access limited to DataTalksClub/aws-infra. Only then may the paired DataOps pin/deletion commit be pushed. The aws-infra push is inert; the later DataOps main push remains the sole normal app deployment trigger.

[alexeygrigorev]

PM grooming complete. The selected boundary is an immutable private aws-infra commit pin plus a read-only cross-repository checkout; DataOps materializes only the pinned template blob under ignored .tmp/, builds current DataOps code with --base-dir at the app repo, and remains the only OIDC app deploy path. aws-infra pushes stay inert. #173 is blocked on finishing #166 and provisioning a single-repo Contents-read token; both currently dirty shared checkouts are explicitly protected by requiring clean isolated worktrees.

[alexeygrigorev]

Preparation checkpoint: the canonical application template is now added in the private DataTalksClub/aws-infra repository on branch issue-173, commit 5dd8a1a, at sandbox/dataops/template.application.yaml. It is byte-identical to the current DataOps template and passes sam validate --lint. No aws-infra deployment was added or run, and the shared dirty aws-infra worktree was not included. The final paired cutover remains blocked on #166 and the dedicated read credential specified in this issue; the template will be refreshed to the final #166 state before merge/cutover.