From f805a1a56d27cbc8211a4c1e42ecde077a04d8d7 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Sun, 17 May 2026 11:22:48 -0400 Subject: [PATCH 01/10] feat(deploy): kustomize manifests + sandbox cluster wiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Iterating on a manual deploy to the CfP sandbox cluster (Linode LKE). Replaces the Helm chart with a kustomize base + sandbox overlay per the direction in the parked specs/architecture.md spec amendment. What landed: - `deploy/kustomize/base/` — namespace-agnostic manifests: Deployment (single replica, Recreate strategy, non-root, readiness + liveness probes), Service, two PVCs (linode-block-storage-retain), ConfigMap with non-secret env, ServiceAccount, Ingress (nginx, letsencrypt-staging issuer, TLS to a per-overlay host). - `deploy/kustomize/overlays/sandbox/` — sandbox-specific Namespace, ingress host patch (codeforphilly-rewrite.codeforphilly.sandbox.k8s.phl.io), and SealedSecret manifests for: - codeforphilly-secrets (CFP_JWT_SIGNING_KEY, CFP_DATA_REMOTE) - codeforphilly-data-deploy-key (read-only ed25519 SSH key registered as a deploy key on the data repo) - `docs/operations/sandbox-deploy.md` — manual procedure, rotation steps, image-visibility note, branch-switching. Build-system fixes pulled in along the way: - `packages/shared` was `noEmit: true`, so the api's compiled JS that imports from `@cfp/shared/schemas` couldn't be resolved at runtime in a production image. Switched the package to emit to dist/, updated its exports map, set its package.json `main`/`types` to dist. - Root `build` script now enforces shared → api → web order (npm workspaces doesn't topo-sort by default). The previous `--workspaces --if-present` ran api before shared and exploded. - `Dockerfile`: dropped the per-workspace `node_modules` COPYs — npm workspaces hoists every dep to the root, so those paths didn't exist. Cluster state after `kubectl apply -k deploy/kustomize/overlays/sandbox`: - Both SealedSecrets decrypted into Secrets - Both PVCs bound (Linode block storage) - Ingress provisioned on the existing LB (45.79.246.168) - Deployment sitting in ImagePullBackOff waiting for the image push Blocked on user-side scope refresh: - `gh auth refresh -s write:packages` so docker push to GHCR succeeds - Make the ghcr.io/codeforphilly/codeforphilly-rewrite package public (one-time, on the package settings page) Co-Authored-By: Claude Opus 4.7 (1M context) --- Dockerfile | 15 ++- deploy/kustomize/base/configmap.yaml | 18 ++++ deploy/kustomize/base/deployment.yaml | 86 ++++++++++++++++ deploy/kustomize/base/ingress.yaml | 23 +++++ deploy/kustomize/base/kustomization.yaml | 25 +++++ deploy/kustomize/base/pvc-data.yaml | 10 ++ deploy/kustomize/base/pvc-private.yaml | 10 ++ deploy/kustomize/base/service.yaml | 13 +++ deploy/kustomize/base/serviceaccount.yaml | 4 + .../overlays/sandbox/kustomization.yaml | 39 ++++++++ .../kustomize/overlays/sandbox/namespace.yaml | 4 + .../sandbox/sealed-secret-deploy-key.yaml | 13 +++ .../overlays/sandbox/sealed-secret-env.yaml | 14 +++ docs/operations/sandbox-deploy.md | 99 +++++++++++++++++++ package.json | 2 +- packages/shared/package.json | 18 +++- packages/shared/tsconfig.json | 10 +- 17 files changed, 386 insertions(+), 17 deletions(-) create mode 100644 deploy/kustomize/base/configmap.yaml create mode 100644 deploy/kustomize/base/deployment.yaml create mode 100644 deploy/kustomize/base/ingress.yaml create mode 100644 deploy/kustomize/base/kustomization.yaml create mode 100644 deploy/kustomize/base/pvc-data.yaml create mode 100644 deploy/kustomize/base/pvc-private.yaml create mode 100644 deploy/kustomize/base/service.yaml create mode 100644 deploy/kustomize/base/serviceaccount.yaml create mode 100644 deploy/kustomize/overlays/sandbox/kustomization.yaml create mode 100644 deploy/kustomize/overlays/sandbox/namespace.yaml create mode 100644 deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml create mode 100644 deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml create mode 100644 docs/operations/sandbox-deploy.md diff --git a/Dockerfile b/Dockerfile index 27646e7..c999423 100644 --- a/Dockerfile +++ b/Dockerfile @@ -47,18 +47,17 @@ WORKDIR /app RUN apk add --no-cache git python3 make g++ +# npm workspaces hoists every dep to the root node_modules; the per-workspace +# node_modules dirs don't exist at this scale. Copy only the root. COPY --from=deps /app/node_modules ./node_modules -COPY --from=deps /app/apps/api/node_modules ./apps/api/node_modules -COPY --from=deps /app/apps/web/node_modules ./apps/web/node_modules -COPY --from=deps /app/packages/shared/node_modules ./packages/shared/node_modules COPY tsconfig.base.json package.json package-lock.json ./ COPY apps ./apps COPY packages ./packages -# Build both workspaces. Web is built first so api/dist references work; the -# workspace `build` script handles order via `--if-present`. -RUN npm run build --workspaces --if-present +# Build in dependency order: shared first (api + web both import from it), +# then api + web. The root `build` script enforces this ordering. +RUN npm run build # Drop devDependencies from node_modules to shrink the runtime image. We still # need workspace-local node_modules (better-sqlite3 native binding lives there). @@ -77,16 +76,14 @@ RUN apk add --no-cache git ca-certificates tini openssh-client WORKDIR /app -# Copy built artifacts + pruned node_modules. +# Copy built artifacts + pruned node_modules (hoisted at the root). COPY --from=build /app/package.json /app/package-lock.json ./ COPY --from=build /app/node_modules ./node_modules COPY --from=build /app/apps/api/package.json ./apps/api/ COPY --from=build /app/apps/api/dist ./apps/api/dist -COPY --from=build /app/apps/api/node_modules ./apps/api/node_modules COPY --from=build /app/apps/web/dist ./apps/web/dist COPY --from=build /app/packages/shared/package.json ./packages/shared/ COPY --from=build /app/packages/shared/dist ./packages/shared/dist -COPY --from=build /app/packages/shared/node_modules ./packages/shared/node_modules # Entrypoint script handles data-repo init/refresh before exec'ing node. COPY deploy/docker/entrypoint.sh /usr/local/bin/entrypoint.sh diff --git a/deploy/kustomize/base/configmap.yaml b/deploy/kustomize/base/configmap.yaml new file mode 100644 index 0000000..2558798 --- /dev/null +++ b/deploy/kustomize/base/configmap.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: codeforphilly-env +data: + CFP_DATA_REPO_PATH: "/app/data" + CFP_PRIVATE_STORAGE_PATH: "/app/private-storage" + CFP_WEB_DIST_PATH: "/app/apps/web/dist" + GIT_AUTHOR_EMAIL: "api@codeforphilly.org" + GIT_AUTHOR_NAME: "CodeForPhilly API" + NODE_ENV: "production" + PORT: "3001" + STORAGE_BACKEND: "filesystem" + CFP_DATA_BRANCH: "fixture" + # SSH key for the data repo deploy key (private branch reads). + # accept-new keeps first-connect simple; strict host-key checking via + # known_hosts ConfigMap is an overlay concern. + GIT_SSH_COMMAND: "ssh -i /etc/cfp-data-deploy-key/id_ed25519 -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null" diff --git a/deploy/kustomize/base/deployment.yaml b/deploy/kustomize/base/deployment.yaml new file mode 100644 index 0000000..8f288a7 --- /dev/null +++ b/deploy/kustomize/base/deployment.yaml @@ -0,0 +1,86 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: codeforphilly +spec: + # Single replica is a hard architectural constraint — the in-process write + # mutex serializes gitsheets commits. See specs/architecture.md. + replicas: 1 + strategy: + # Recreate, not RollingUpdate: two pods writing the same gitsheets repo + # would corrupt state. Old pod must release the lock before new starts. + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: codeforphilly + template: + metadata: + labels: + app.kubernetes.io/name: codeforphilly + spec: + serviceAccountName: codeforphilly + securityContext: + fsGroup: 1000 + containers: + - name: codeforphilly + image: ghcr.io/codeforphilly/codeforphilly-rewrite:sandbox + imagePullPolicy: IfNotPresent + ports: + - containerPort: 3001 + name: http + envFrom: + - configMapRef: + name: codeforphilly-env + - secretRef: + name: codeforphilly-secrets + env: + - name: HOST + value: "0.0.0.0" + volumeMounts: + - name: data + mountPath: /app/data + - name: private + mountPath: /app/private-storage + - name: deploy-key + mountPath: /etc/cfp-data-deploy-key + readOnly: true + # Readiness probe gates traffic until both stores have loaded. + readinessProbe: + httpGet: + path: /api/health/ready + port: http + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 30 + livenessProbe: + httpGet: + path: /api/health + port: http + initialDelaySeconds: 60 + periodSeconds: 30 + timeoutSeconds: 5 + resources: + requests: + cpu: 100m + memory: 384Mi + limits: + cpu: 1000m + memory: 768Mi + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] + volumes: + - name: data + persistentVolumeClaim: + claimName: codeforphilly-data + - name: private + persistentVolumeClaim: + claimName: codeforphilly-private + - name: deploy-key + secret: + secretName: codeforphilly-data-deploy-key + defaultMode: 0400 diff --git a/deploy/kustomize/base/ingress.yaml b/deploy/kustomize/base/ingress.yaml new file mode 100644 index 0000000..28af05d --- /dev/null +++ b/deploy/kustomize/base/ingress.yaml @@ -0,0 +1,23 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: codeforphilly + annotations: + # cert-manager will provision a TLS cert. Override the issuer per-overlay. + cert-manager.io/cluster-issuer: letsencrypt-staging +spec: + ingressClassName: nginx + tls: + - hosts: [PLACEHOLDER_HOST] + secretName: codeforphilly-tls + rules: + - host: PLACEHOLDER_HOST + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: codeforphilly + port: + number: 80 diff --git a/deploy/kustomize/base/kustomization.yaml b/deploy/kustomize/base/kustomization.yaml new file mode 100644 index 0000000..5de851c --- /dev/null +++ b/deploy/kustomize/base/kustomization.yaml @@ -0,0 +1,25 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# Base manifests for the codeforphilly-rewrite app. Environment-specific +# variation lives in overlays//. Apply with `kubectl apply -k +# deploy/kustomize/overlays/`. + +labels: + - pairs: + app.kubernetes.io/name: codeforphilly + app.kubernetes.io/managed-by: kustomize + includeSelectors: true + +resources: + - serviceaccount.yaml + - configmap.yaml + - pvc-data.yaml + - pvc-private.yaml + - service.yaml + - deployment.yaml + - ingress.yaml + +images: + - name: ghcr.io/codeforphilly/codeforphilly-rewrite + newTag: sandbox diff --git a/deploy/kustomize/base/pvc-data.yaml b/deploy/kustomize/base/pvc-data.yaml new file mode 100644 index 0000000..8e6db7e --- /dev/null +++ b/deploy/kustomize/base/pvc-data.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: codeforphilly-data +spec: + accessModes: [ReadWriteOnce] + storageClassName: linode-block-storage-retain + resources: + requests: + storage: 10Gi diff --git a/deploy/kustomize/base/pvc-private.yaml b/deploy/kustomize/base/pvc-private.yaml new file mode 100644 index 0000000..42e17be --- /dev/null +++ b/deploy/kustomize/base/pvc-private.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: codeforphilly-private +spec: + accessModes: [ReadWriteOnce] + storageClassName: linode-block-storage-retain + resources: + requests: + storage: 1Gi diff --git a/deploy/kustomize/base/service.yaml b/deploy/kustomize/base/service.yaml new file mode 100644 index 0000000..032d814 --- /dev/null +++ b/deploy/kustomize/base/service.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: codeforphilly +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 3001 + protocol: TCP + name: http + selector: + app.kubernetes.io/name: codeforphilly diff --git a/deploy/kustomize/base/serviceaccount.yaml b/deploy/kustomize/base/serviceaccount.yaml new file mode 100644 index 0000000..199d915 --- /dev/null +++ b/deploy/kustomize/base/serviceaccount.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: codeforphilly diff --git a/deploy/kustomize/overlays/sandbox/kustomization.yaml b/deploy/kustomize/overlays/sandbox/kustomization.yaml new file mode 100644 index 0000000..8573ab8 --- /dev/null +++ b/deploy/kustomize/overlays/sandbox/kustomization.yaml @@ -0,0 +1,39 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# Sandbox overlay — codeforphilly.sandbox.k8s.phl.io subdomain, filesystem +# private storage, letsencrypt-staging cert. Apply with: +# kubectl apply -k deploy/kustomize/overlays/sandbox + +namespace: codeforphilly-rewrite-sandbox + +# No namePrefix — namespace already isolates these resources. namePrefix +# would also rename Secrets that the SealedSecret CRD expands into, but +# kustomize can't rewrite Secret-style refs through the SealedSecret kind, +# so the Deployment's envFrom/volume references would silently break. + +resources: + - ../../base + - namespace.yaml + - sealed-secret-env.yaml + - sealed-secret-deploy-key.yaml + +# Image tag override is set by build/push pipelines; the base defaults to +# `sandbox` which matches the CI workflow for this overlay. +images: + - name: ghcr.io/codeforphilly/codeforphilly-rewrite + newTag: sandbox + +# Per-environment patches that aren't expressible as a simple value swap. +patches: + # Wire the real hostname into the ingress. + - target: + kind: Ingress + name: codeforphilly + patch: | + - op: replace + path: /spec/tls/0/hosts/0 + value: codeforphilly-rewrite.codeforphilly.sandbox.k8s.phl.io + - op: replace + path: /spec/rules/0/host + value: codeforphilly-rewrite.codeforphilly.sandbox.k8s.phl.io diff --git a/deploy/kustomize/overlays/sandbox/namespace.yaml b/deploy/kustomize/overlays/sandbox/namespace.yaml new file mode 100644 index 0000000..c9b7bc8 --- /dev/null +++ b/deploy/kustomize/overlays/sandbox/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: codeforphilly-rewrite-sandbox diff --git a/deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml b/deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml new file mode 100644 index 0000000..4eff4a6 --- /dev/null +++ b/deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml @@ -0,0 +1,13 @@ +--- +apiVersion: bitnami.com/v1alpha1 +kind: SealedSecret +metadata: + name: codeforphilly-data-deploy-key + namespace: codeforphilly-rewrite-sandbox +spec: + encryptedData: + id_ed25519: AgAfJZfYzNyySAAyeqVVS297WB/+sXIz9rU8OXzryC0Vp2AgS+al9ZzOqgB/GrNDub10Tdt2d/IuSLE8FKUXz0OiIv19WwJfsINZUJjbRX04C6TXnyRa5wRcOv/hP9Va/Hz2SxpfDtWxey2O6IBCH2b0+5pajC8YsXxw8VHvZY+bJJMuNv6piphgxIo67kcrGsx3pN7naUUObutRkm3aThsmZ5TWxd7fHuiVWi7+E3ek7JLqAmOXdA8JFv14CLtEKDgx524wLavJ1vzEBAxdhd5PBWgp0CY2FKmlSbxzYmp/wFUEc2hBvWQJWO1SjfIt/CgdjSanVR7/wQARCvQ7EBV3DYVi8TkoGthtz3yD9O58Et6q6V88m/lhB+hCjrevSnwk3EqbygZh8nkWM8UDhXzShvurowakWb1YWTKBvGn7HMiaC3coBcakPf1dgJkzr3BAijZL/2cvWOiSZQx9VX///vYmxyr72jSabIW2nArnz1K9q88mL0RjNPB/0vdjTt4MLKC+2UwQ4jtfB7NLp6UPELZe68MH6Xr3YiufEoZJ3wFpSK1X4deIz4pRJwCkX4BrCyDg2S+TgEZnI8O9HkwOkd8PxlkpB7WUTeR8wrR3RouBHilXh3iUGt949rjC6t8ww9qsvfTzrtSMCX87rdD4B9ewxgetIzMLzJWJTyllQNz8srVFCu5FFjaYZTgUbPxKWQ+RePchnWGPxgTjduwSODjPdKimfLZwIacT+XfcBe+0jMW2IarXNC9lLPrpT+iV/9UO7CufyWaduHAkWVHJUJmgf5a8ljcAEuaj+nfkLqCU2jLIAtnUDGHQIk7sKiuJn/e1xkK8PRL2m2OAZU7jUmgWlB/KwMNhoqipkP9+O7AIFHYaxfK+ATyoGAmHjF9ko94sJclaXwz/uPsKyFNUSUH1cxsPfco0LmEPatJHccDXO/yQmXjJaFEOekYQufHp8ThudYv5i1COpHEa8y0A21jL28Pt8vDEHNxJVFNOS+Og85ESvda/CmYefXUlRdLaESITQOukEdS4H/aHkFaympO4GAcZl0Qjzn+EWznbH9GmQipaL0PRUUhEmiiciuDgiNGveFfuNx05096adW8YAxVjNlwookDshzjxW+6OYIBboy1+FWFE+YQlugQusRmm1svXtBlL/Rrh4eM3hyBiuSobtPJ9s9wYyHlIT3D06bLkJdRWCfg7ZwDrqEdTWRmkhJldhNEfop5GAxoyhNX5X3IvxbTiWS1QLY+QP57a2ZtdnBgOsnrVko7kSaWPY1U= + template: + metadata: + name: codeforphilly-data-deploy-key + namespace: codeforphilly-rewrite-sandbox diff --git a/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml b/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml new file mode 100644 index 0000000..fe7ef63 --- /dev/null +++ b/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml @@ -0,0 +1,14 @@ +--- +apiVersion: bitnami.com/v1alpha1 +kind: SealedSecret +metadata: + name: codeforphilly-secrets + namespace: codeforphilly-rewrite-sandbox +spec: + encryptedData: + CFP_DATA_REMOTE: AgBdON1Ac/LBCPg3syYH1D4uNPnJOEP75PSRjaktkKpUhxVffGQgOyhCsfsV/sn7YRHU5cdtNJhyL9pJjJabVD6BfDLvtpHOSm/G3fiUqnjSVpKzcyXxKrPsBkYJfcTOyH5gN+AoDihIvxkrpgg1hdby9aC7IONHSh+DpEuUXgsiDWcu8bt4l18YCwByYLNgBLQLPNs4/Stv7FHzU99ec4coWmIWZedtAgsq4o7p5xMkNM2VfZ7L8V5Q5XUZOQShNfOXSqvfr/DGbigkzTZdGm9Hrq/D+SJSsNaRmN0OJaHHo0LBrGIOyWMzDZz9vO6lI/REjFmFijg8dpHMv1rVeF1y8xMqL9W/vASg1C0oceHHSBK13NTnZGCpPKvzRwaV1AhcHCdTObWFv/VnmCT56LtpfxZcjkfPSwiF3ZvUlhcu0HHpqUzR26eg1uKZt8OlUJE0gJo0tF+X2ulYMFSTyyXkl/Y+xu2qBmmknuoI2UVrDmKCr6PqL/O0CDjSZQ+OT2oSi7VXNg5LBwfzkneSGE0dnSssmxro4wVchdSze87HK8eZUdl8RJn8cMLAai+SxwelKvGVUbzD4GCrdiKE3cL5+4fgIFJZX24w/dFWxBjCXDuJD5KEfxFKbGyR/tD3cEiVXisABVeOtuBuprceNJJJtD9Tep/xGMJpTND1V8jqmVB0OKmeub0ruL3MBzHuGKsYdZz0oMBevzJCMuEiyyTzOy5/COl2o1pBymFwk00AGpKzkCsjAMP9hk3Opm6hLBgpYCs= + CFP_JWT_SIGNING_KEY: AgAy03/R7XN6y5qtsLe6o1HriQMQQh3FrHZkUL9+SWti9b2idHqZql6T1wXIk21axTB5MqiCQuURiGGW+V+7kSCZVJByqe6T/026WQcI6tNVEMLOiSZ5TfKQT4Cwota7CqiC1bqtuXh1oYDEn9U1kQ08sVy12kgmVDlwOsGXQrO7+wTVOMZOsaO/97SpN7L04a4uf2PKag4N7tg/WeRxvzDuNhfaPh6USCWVZmgijzGoqTrr/HJbXzbvK7cddoUnh4EYmxxnM3z/hrghgPLYg/QD1wWYicQq+BMdKv+JR+Zy98fuB+PTtmgPj7c2A8ot99RtgUTq4ZKQ3YOuIINsUY0iauikrU5DFg9JSTNgsE1Idt1Rpf8a5zy09x84llPcp/XGBJL0f2W0JH9r8TTdT+0MZ4uVuQTNaWTVET6ei6cw1Wohkrb3l4Qp3ix6Rw0gxt8M4534JPaDe3uXdcSYoNnoiq8fxrTd0Dt8CJ1xCx3902pAyV/xS/9um1azF7USl3mXad+UOhupa21Yf1fMnq7ZxH+wqOoFLdcv0TzQc5jmxDDxNKzRkG7UstfAA7C8N16uvmt5OiZrYnQe15wIJktYySL55UY9wmC3qxeCZvrljC25+nu5SlcUa4YhKADTQE/jizFaBtnuumRKf08lAaRm9n2f2i21gPq7UY0xdBXUmz9DKlVdfkPlIxm0pwrckCr3B6szwZQeoq4nUJ7iDcVJ5pT7XpS8G6im0G4UzjBb4kgOcx36KeGZYeWdoGleQCXQGdxc/mRIfONawuipCSYG + template: + metadata: + name: codeforphilly-secrets + namespace: codeforphilly-rewrite-sandbox diff --git a/docs/operations/sandbox-deploy.md b/docs/operations/sandbox-deploy.md new file mode 100644 index 0000000..f4861e5 --- /dev/null +++ b/docs/operations/sandbox-deploy.md @@ -0,0 +1,99 @@ +# Manual sandbox deploy + +This is the manual procedure for iterating on a deploy to the **CfP sandbox cluster** (Linode LKE, k8s.phl.io). GitOps wiring is a follow-up; this doc is the source of truth until that lands. + +## Cluster + +- **Kubeconfig:** `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` +- **Ingress:** nginx, wildcard DNS for `*.codeforphilly.sandbox.k8s.phl.io` → `45.79.246.168` +- **Storage class:** `linode-block-storage-retain` (default) +- **Sealed-secrets:** controller in `sealed-secrets` namespace +- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers (staging used here for fast iteration; flip to prod when ready) + +## Data repo + +The app reads its gitsheets data from a private GitHub repo cloned at boot: + +- **Repo:** `git@github.com:CodeForPhilly/codeforphilly-data.git` (private during cutover prep) +- **Branches** — each is an independent data scenario: + - `fixture` (default) — hand-/import-curated test data, used by sandbox + - `empty` — sheet configs only, no records + - `snapshot` — anonymized snapshot of prod (auto-produced post-cutover) +- A read-only **SSH deploy key** mounted into the pod authenticates the entrypoint's clone. + +## One-shot deploy steps (manual, while iterating) + +```bash +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml + +# 1. Build + push the image +docker build -t ghcr.io/codeforphilly/codeforphilly-rewrite:sandbox . +# NOTE: requires `write:packages` scope on your GitHub token. +# If `docker push` says "token does not match expected scopes": +# gh auth refresh -s write:packages +docker push ghcr.io/codeforphilly/codeforphilly-rewrite:sandbox + +# 2. Apply manifests (creates namespace, sealed-secrets, PVCs, deployment, service, ingress) +kubectl apply -k deploy/kustomize/overlays/sandbox + +# 3. Watch the rollout +kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly +kubectl -n codeforphilly-rewrite-sandbox logs -f deploy/codeforphilly +``` + +After the first successful rollout, the app is live at: + +- + +## Image visibility + +The Docker image is built from this repo and pushed to `ghcr.io/codeforphilly/codeforphilly-rewrite`. For the cluster to pull without an `imagePullSecret`, the package must be **public** on GHCR. After the first push: + +1. Visit +2. Under "Danger Zone" → "Change package visibility" → Public + +Until that's done, the deployment will sit in `ImagePullBackOff` with `403 Forbidden`. + +## Rotating the deploy key + +The SSH deploy key currently in the cluster was generated locally and added to the data repo via `gh repo deploy-key add`. To rotate: + +```bash +ssh-keygen -t ed25519 -f /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated -N "" -C "cfp-sandbox-rotated" +gh repo deploy-key add /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated.pub \ + --repo CodeForPhilly/codeforphilly-data \ + --title "cfp-sandbox cluster (rotated $(date +%Y-%m-%d))" +# Then re-seal the secret and re-apply +kubectl create secret generic codeforphilly-data-deploy-key \ + --namespace codeforphilly-rewrite-sandbox \ + --from-file=id_ed25519=/tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated \ + --dry-run=client -o yaml \ + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets -o yaml \ + > deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml +kubectl apply -k deploy/kustomize/overlays/sandbox +# Delete the old deploy key from GitHub after the rotation lands cleanly. +``` + +## Rotating the JWT signing key + +```bash +JWT_KEY=$(openssl rand -base64 48) +kubectl create secret generic codeforphilly-secrets \ + --namespace codeforphilly-rewrite-sandbox \ + --from-literal=CFP_JWT_SIGNING_KEY="$JWT_KEY" \ + --from-literal=CFP_DATA_REMOTE="git@github.com:CodeForPhilly/codeforphilly-data.git" \ + --dry-run=client -o yaml \ + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets -o yaml \ + > deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml +kubectl apply -k deploy/kustomize/overlays/sandbox +# Rotating the JWT signing key invalidates every issued session — users will +# need to re-auth. Acceptable in sandbox; coordinate before doing this in prod. +``` + +## Switching data branches + +The active branch is set in `deploy/kustomize/base/configmap.yaml` via `CFP_DATA_BRANCH`. To swap: + +1. Edit the ConfigMap (or add an overlay patch) +2. `kubectl apply -k deploy/kustomize/overlays/sandbox` +3. `kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly` — entrypoint re-clones the working tree against the new branch diff --git a/package.json b/package.json index 6f56438..9bdf061 100644 --- a/package.json +++ b/package.json @@ -13,7 +13,7 @@ ], "scripts": { "dev": "concurrently -n api,web -c blue,magenta \"npm run -w apps/api dev\" \"npm run -w apps/web dev\"", - "build": "npm run build --workspaces --if-present", + "build": "npm run -w packages/shared build && npm run -w apps/api build && npm run -w apps/web build", "type-check": "npm run type-check --workspaces --if-present", "lint": "eslint .", "test": "npm run test --workspaces --if-present" diff --git a/packages/shared/package.json b/packages/shared/package.json index 39c0117..44f9bc9 100644 --- a/packages/shared/package.json +++ b/packages/shared/package.json @@ -3,13 +3,23 @@ "version": "0.0.0", "private": true, "type": "module", - "main": "./src/index.ts", - "types": "./src/index.ts", + "main": "./dist/index.js", + "types": "./dist/index.d.ts", "exports": { - ".": "./src/index.ts", - "./schemas": "./src/schemas/index.ts" + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js", + "default": "./dist/index.js" + }, + "./schemas": { + "types": "./dist/schemas/index.d.ts", + "import": "./dist/schemas/index.js", + "default": "./dist/schemas/index.js" + } }, + "files": ["dist", "src"], "scripts": { + "build": "tsc -p tsconfig.json", "type-check": "tsc -p tsconfig.json --noEmit", "test": "vitest run", "generate-schemas": "tsx scripts/generate-json-schemas.ts", diff --git a/packages/shared/tsconfig.json b/packages/shared/tsconfig.json index 45635e3..655d568 100644 --- a/packages/shared/tsconfig.json +++ b/packages/shared/tsconfig.json @@ -1,11 +1,15 @@ { "extends": "../../tsconfig.base.json", "compilerOptions": { - "module": "ESNext", - "moduleResolution": "Bundler", + "module": "NodeNext", + "moduleResolution": "NodeNext", "target": "ES2023", "lib": ["ES2023"], - "noEmit": true + "outDir": "dist", + "rootDir": "src", + "declaration": true, + "declarationMap": true, + "sourceMap": true }, "include": ["src/**/*"] } From c55003589abf0d693ab67e70dd0520791e926cb3 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 02:03:58 -0400 Subject: [PATCH 02/10] feat(deploy): flip cert-manager issuer from staging to prod Per-overlay staging override is documented in the file's comment; sandbox now issues real TLS certs from letsencrypt-prod (rate limit: 50 certs/week per registered domain). Co-Authored-By: Claude Opus 4.7 (1M context) --- deploy/kustomize/base/ingress.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/deploy/kustomize/base/ingress.yaml b/deploy/kustomize/base/ingress.yaml index 28af05d..bd10717 100644 --- a/deploy/kustomize/base/ingress.yaml +++ b/deploy/kustomize/base/ingress.yaml @@ -3,8 +3,10 @@ kind: Ingress metadata: name: codeforphilly annotations: - # cert-manager will provision a TLS cert. Override the issuer per-overlay. - cert-manager.io/cluster-issuer: letsencrypt-staging + # cert-manager will provision a TLS cert via Let's Encrypt prod. + # Per-overlay can override to letsencrypt-staging for high-churn iteration + # (prod rate-limits to 50 certs/week per registered domain). + cert-manager.io/cluster-issuer: letsencrypt-prod spec: ingressClassName: nginx tls: From 11ffad6f9eb864b1dcff97eeff784574cf97086f Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 02:04:48 -0400 Subject: [PATCH 03/10] fix(deploy): rename GHCR image to codeforphilly-ng Match the GitHub repo name (CodeForPhilly/codeforphilly-ng). Kustomize image mappings + docs/operations/deploy.md updated; deployment.yaml itself + the sandbox-deploy runbook follow in the next commits along with the sandbox-iteration fixes. Co-Authored-By: Claude Opus 4.7 (1M context) --- deploy/kustomize/base/kustomization.yaml | 2 +- deploy/kustomize/overlays/sandbox/kustomization.yaml | 2 +- docs/operations/deploy.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/deploy/kustomize/base/kustomization.yaml b/deploy/kustomize/base/kustomization.yaml index 5de851c..513a117 100644 --- a/deploy/kustomize/base/kustomization.yaml +++ b/deploy/kustomize/base/kustomization.yaml @@ -21,5 +21,5 @@ resources: - ingress.yaml images: - - name: ghcr.io/codeforphilly/codeforphilly-rewrite + - name: ghcr.io/codeforphilly/codeforphilly-ng newTag: sandbox diff --git a/deploy/kustomize/overlays/sandbox/kustomization.yaml b/deploy/kustomize/overlays/sandbox/kustomization.yaml index 8573ab8..f7e4a04 100644 --- a/deploy/kustomize/overlays/sandbox/kustomization.yaml +++ b/deploy/kustomize/overlays/sandbox/kustomization.yaml @@ -21,7 +21,7 @@ resources: # Image tag override is set by build/push pipelines; the base defaults to # `sandbox` which matches the CI workflow for this overlay. images: - - name: ghcr.io/codeforphilly/codeforphilly-rewrite + - name: ghcr.io/codeforphilly/codeforphilly-ng newTag: sandbox # Per-environment patches that aren't expressible as a simple value swap. diff --git a/docs/operations/deploy.md b/docs/operations/deploy.md index f65b929..afb19b3 100644 --- a/docs/operations/deploy.md +++ b/docs/operations/deploy.md @@ -18,7 +18,7 @@ the runbook that implements it. | docker build / push v +----------------------+ -| GHCR image | ghcr.io/codeforphilly/codeforphilly-rewrite: +| GHCR image | ghcr.io/codeforphilly/codeforphilly-ng: +----------+-----------+ | helm upgrade --install v @@ -42,7 +42,7 @@ container. The single replica is a hard architectural constraint ### Build ```bash -docker build -t ghcr.io/codeforphilly/codeforphilly-rewrite:dev . +docker build -t ghcr.io/codeforphilly/codeforphilly-ng:dev . ``` Three stages — `deps` (full install), `build` (compile both workspaces, prune @@ -59,7 +59,7 @@ docker run --rm -p 3001:3001 \ -e CFP_JWT_SIGNING_KEY="$(openssl rand -base64 48)" \ -e GITHUB_OAUTH_CLIENT_ID=local \ -e GITHUB_OAUTH_CLIENT_SECRET=local \ - ghcr.io/codeforphilly/codeforphilly-rewrite:dev + ghcr.io/codeforphilly/codeforphilly-ng:dev curl http://localhost:3001/api/health # liveness curl http://localhost:3001/api/health/ready # readiness From df51de07c811956eeee25d828eb238eabe76430e Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 02:05:05 -0400 Subject: [PATCH 04/10] fix(deploy): harden sandbox iteration loop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes discovered standing up the first end-to-end sandbox deploy: - **Entrypoint self-heals PVC residue.** Block-storage PVCs survive pod restarts and can carry non-empty/non-git content from earlier iterations. The entrypoint now wipes a non-empty data dir before re-cloning and adds the data path to git's safe.directory list (uid mismatch is common when prior pods ran as root and the new pod runs as uid 1000). - **imagePullPolicy: Always for the sandbox image.** The `:sandbox` tag is mutable — re-pushed on each iteration. IfNotPresent would let kubelet reuse cached layers from an older digest. Production overlays should pin to a digest and set this back to IfNotPresent. - **`docs/operations/sandbox-deploy.md`** + image rename to codeforphilly-ng. Doc now notes `--platform=linux/amd64` is required on Apple Silicon — Linode LKE nodes are amd64 and won't pull an arm64-only manifest. Co-Authored-By: Claude Opus 4.7 (1M context) --- deploy/docker/entrypoint.sh | 14 ++++++++++++++ deploy/kustomize/base/deployment.yaml | 6 ++++-- docs/operations/sandbox-deploy.md | 10 ++++++---- 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/deploy/docker/entrypoint.sh b/deploy/docker/entrypoint.sh index cf30e61..85c15fe 100755 --- a/deploy/docker/entrypoint.sh +++ b/deploy/docker/entrypoint.sh @@ -27,6 +27,12 @@ log() { DATA_BRANCH="${CFP_DATA_BRANCH:-main}" +# Trust the data-repo working tree regardless of file ownership. PVCs survive +# pod restarts and may carry files owned by a different uid than this pod's +# runAsUser (e.g., an earlier iteration ran as root). Without this, git +# refuses every operation with "detected dubious ownership". +git config --global --add safe.directory "$CFP_DATA_REPO_PATH" + if [ -z "${CFP_DATA_REMOTE:-}" ]; then if [ -d "$CFP_DATA_REPO_PATH/.git" ]; then log "CFP_DATA_REMOTE unset; using existing working tree at $CFP_DATA_REPO_PATH" @@ -48,6 +54,14 @@ else git reset --hard "origin/$DATA_BRANCH" cd - >/dev/null else + # PVC may carry residue from a previous pod that bailed mid-clone or from + # an earlier iteration of the entrypoint. `git clone` refuses to clone into + # a non-empty directory, so wipe it first. Safe because the data repo is + # always re-cloneable from CFP_DATA_REMOTE. + if [ -n "$(ls -A "$CFP_DATA_REPO_PATH" 2>/dev/null)" ]; then + log "$CFP_DATA_REPO_PATH non-empty but lacks .git — wiping before clone" + find "$CFP_DATA_REPO_PATH" -mindepth 1 -maxdepth 1 -exec rm -rf {} + + fi log "cloning $CFP_DATA_REMOTE into $CFP_DATA_REPO_PATH (branch=$DATA_BRANCH)" # --depth=1 keeps the PVC footprint small; the push daemon will deepen as # needed when it next pushes (or we accept periodic re-clones). diff --git a/deploy/kustomize/base/deployment.yaml b/deploy/kustomize/base/deployment.yaml index 8f288a7..2e01faf 100644 --- a/deploy/kustomize/base/deployment.yaml +++ b/deploy/kustomize/base/deployment.yaml @@ -23,8 +23,10 @@ spec: fsGroup: 1000 containers: - name: codeforphilly - image: ghcr.io/codeforphilly/codeforphilly-rewrite:sandbox - imagePullPolicy: IfNotPresent + image: ghcr.io/codeforphilly/codeforphilly-ng:sandbox + # Always pull — sandbox tag is mutable (re-pushed on each iteration). + # Production overlays should pin to a digest and flip this to IfNotPresent. + imagePullPolicy: Always ports: - containerPort: 3001 name: http diff --git a/docs/operations/sandbox-deploy.md b/docs/operations/sandbox-deploy.md index f4861e5..e37f3a7 100644 --- a/docs/operations/sandbox-deploy.md +++ b/docs/operations/sandbox-deploy.md @@ -27,11 +27,13 @@ The app reads its gitsheets data from a private GitHub repo cloned at boot: export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml # 1. Build + push the image -docker build -t ghcr.io/codeforphilly/codeforphilly-rewrite:sandbox . +# --platform=linux/amd64 is required when building on Apple Silicon — the +# Linode LKE nodes are amd64 and won't pull an arm64-only manifest. +docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . # NOTE: requires `write:packages` scope on your GitHub token. # If `docker push` says "token does not match expected scopes": # gh auth refresh -s write:packages -docker push ghcr.io/codeforphilly/codeforphilly-rewrite:sandbox +docker push ghcr.io/codeforphilly/codeforphilly-ng:sandbox # 2. Apply manifests (creates namespace, sealed-secrets, PVCs, deployment, service, ingress) kubectl apply -k deploy/kustomize/overlays/sandbox @@ -47,9 +49,9 @@ After the first successful rollout, the app is live at: ## Image visibility -The Docker image is built from this repo and pushed to `ghcr.io/codeforphilly/codeforphilly-rewrite`. For the cluster to pull without an `imagePullSecret`, the package must be **public** on GHCR. After the first push: +The Docker image is built from this repo and pushed to `ghcr.io/codeforphilly/codeforphilly-ng`. For the cluster to pull without an `imagePullSecret`, the package must be **public** on GHCR. After the first push: -1. Visit +1. Visit 2. Under "Danger Zone" → "Change package visibility" → Public Until that's done, the deployment will sit in `ImagePullBackOff` with `403 Forbidden`. From 48d5b5e6681e4bdf0df32fc852d3bcf7e16d0260 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 02:10:14 -0400 Subject: [PATCH 05/10] feat(deploy): replace nginx Ingress with Gateway API + next-v2 hostname Cluster has Envoy Gateway (gatewayClassName: eg) at 139.144.241.4; DNS for *.sandbox.k8s.phl.io and next-v2.codeforphilly.org CNAMEs into it. The nginx Ingress lived at a separate IP (45.79.246.168) that nothing currently points at, so it was dead weight. - Add Gateway + HTTPRoute to kustomize/base; cert-manager annotation on the Gateway provisions the codeforphilly-gw-tls Secret via letsencrypt-prod. - Drop kustomize/base/ingress.yaml. - Sandbox overlay patches both resources to next-v2.codeforphilly.org. - docs/operations/sandbox-deploy.md updated to reflect the new ingress path and the production hostname. Co-Authored-By: Claude Opus 4.7 (1M context) --- deploy/kustomize/base/gateway.yaml | 23 +++++++++++++++++ deploy/kustomize/base/httproute.yaml | 17 +++++++++++++ deploy/kustomize/base/ingress.yaml | 25 ------------------- deploy/kustomize/base/kustomization.yaml | 3 ++- .../overlays/sandbox/kustomization.yaml | 25 +++++++++++++------ docs/operations/sandbox-deploy.md | 6 ++--- 6 files changed, 62 insertions(+), 37 deletions(-) create mode 100644 deploy/kustomize/base/gateway.yaml create mode 100644 deploy/kustomize/base/httproute.yaml delete mode 100644 deploy/kustomize/base/ingress.yaml diff --git a/deploy/kustomize/base/gateway.yaml b/deploy/kustomize/base/gateway.yaml new file mode 100644 index 0000000..d8c7bb0 --- /dev/null +++ b/deploy/kustomize/base/gateway.yaml @@ -0,0 +1,23 @@ +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: codeforphilly + annotations: + # cert-manager watches Gateways with this annotation and creates a + # Certificate that resolves into the listener's certificateRef Secret. + cert-manager.io/cluster-issuer: letsencrypt-prod +spec: + gatewayClassName: eg + listeners: + - name: https + hostname: PLACEHOLDER_HOST + port: 443 + protocol: HTTPS + allowedRoutes: + namespaces: + from: Same + tls: + mode: Terminate + certificateRefs: + - kind: Secret + name: codeforphilly-gw-tls diff --git a/deploy/kustomize/base/httproute.yaml b/deploy/kustomize/base/httproute.yaml new file mode 100644 index 0000000..8a9e3fb --- /dev/null +++ b/deploy/kustomize/base/httproute.yaml @@ -0,0 +1,17 @@ +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: codeforphilly +spec: + parentRefs: + - name: codeforphilly + hostnames: + - PLACEHOLDER_HOST + rules: + - matches: + - path: + type: PathPrefix + value: / + backendRefs: + - name: codeforphilly + port: 80 diff --git a/deploy/kustomize/base/ingress.yaml b/deploy/kustomize/base/ingress.yaml deleted file mode 100644 index bd10717..0000000 --- a/deploy/kustomize/base/ingress.yaml +++ /dev/null @@ -1,25 +0,0 @@ -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: codeforphilly - annotations: - # cert-manager will provision a TLS cert via Let's Encrypt prod. - # Per-overlay can override to letsencrypt-staging for high-churn iteration - # (prod rate-limits to 50 certs/week per registered domain). - cert-manager.io/cluster-issuer: letsencrypt-prod -spec: - ingressClassName: nginx - tls: - - hosts: [PLACEHOLDER_HOST] - secretName: codeforphilly-tls - rules: - - host: PLACEHOLDER_HOST - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: codeforphilly - port: - number: 80 diff --git a/deploy/kustomize/base/kustomization.yaml b/deploy/kustomize/base/kustomization.yaml index 513a117..84c8734 100644 --- a/deploy/kustomize/base/kustomization.yaml +++ b/deploy/kustomize/base/kustomization.yaml @@ -18,7 +18,8 @@ resources: - pvc-private.yaml - service.yaml - deployment.yaml - - ingress.yaml + - gateway.yaml + - httproute.yaml images: - name: ghcr.io/codeforphilly/codeforphilly-ng diff --git a/deploy/kustomize/overlays/sandbox/kustomization.yaml b/deploy/kustomize/overlays/sandbox/kustomization.yaml index f7e4a04..2c1dc68 100644 --- a/deploy/kustomize/overlays/sandbox/kustomization.yaml +++ b/deploy/kustomize/overlays/sandbox/kustomization.yaml @@ -1,8 +1,9 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization -# Sandbox overlay — codeforphilly.sandbox.k8s.phl.io subdomain, filesystem -# private storage, letsencrypt-staging cert. Apply with: +# Sandbox overlay — next-v2.codeforphilly.org (CNAME → sandbox.k8s.phl.io), +# filesystem private storage, letsencrypt-prod cert via cert-manager. +# Apply with: # kubectl apply -k deploy/kustomize/overlays/sandbox namespace: codeforphilly-rewrite-sandbox @@ -26,14 +27,22 @@ images: # Per-environment patches that aren't expressible as a simple value swap. patches: - # Wire the real hostname into the ingress. + # Wire the real hostname into the Gateway listener and HTTPRoute. - target: - kind: Ingress + group: gateway.networking.k8s.io + version: v1 + kind: Gateway name: codeforphilly patch: | - op: replace - path: /spec/tls/0/hosts/0 - value: codeforphilly-rewrite.codeforphilly.sandbox.k8s.phl.io + path: /spec/listeners/0/hostname + value: next-v2.codeforphilly.org + - target: + group: gateway.networking.k8s.io + version: v1 + kind: HTTPRoute + name: codeforphilly + patch: | - op: replace - path: /spec/rules/0/host - value: codeforphilly-rewrite.codeforphilly.sandbox.k8s.phl.io + path: /spec/hostnames/0 + value: next-v2.codeforphilly.org diff --git a/docs/operations/sandbox-deploy.md b/docs/operations/sandbox-deploy.md index e37f3a7..8997230 100644 --- a/docs/operations/sandbox-deploy.md +++ b/docs/operations/sandbox-deploy.md @@ -5,10 +5,10 @@ This is the manual procedure for iterating on a deploy to the **CfP sandbox clus ## Cluster - **Kubeconfig:** `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` -- **Ingress:** nginx, wildcard DNS for `*.codeforphilly.sandbox.k8s.phl.io` → `45.79.246.168` +- **Gateway:** Envoy Gateway (`gatewayClassName: eg`), wildcard DNS for `*.sandbox.k8s.phl.io` → `139.144.241.4`. The sandbox app is reachable at `next-v2.codeforphilly.org` via a CNAME to `sandbox.k8s.phl.io`. - **Storage class:** `linode-block-storage-retain` (default) - **Sealed-secrets:** controller in `sealed-secrets` namespace -- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers (staging used here for fast iteration; flip to prod when ready) +- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers. Sandbox uses prod; per-overlay can override to staging for high-churn iteration (prod rate-limits to 50 certs/week per registered domain). ## Data repo @@ -45,7 +45,7 @@ kubectl -n codeforphilly-rewrite-sandbox logs -f deploy/codeforphilly After the first successful rollout, the app is live at: -- +- ## Image visibility From e2e3a7d12b2c350a5538b6550985dbf19164c897 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 02:17:30 -0400 Subject: [PATCH 06/10] feat(deploy): seal GitHub OAuth credentials into sandbox env Secret Adds GITHUB_OAUTH_CLIENT_ID + GITHUB_OAUTH_CLIENT_SECRET alongside the existing CFP_JWT_SIGNING_KEY + CFP_DATA_REMOTE. The sandbox OAuth App "Code for Philly (sandbox)" is registered under the CodeForPhilly org with callback https://next-v2.codeforphilly.org/api/auth/github/callback. Verified end-to-end: GET /api/auth/github/start 302s to github.com/login/oauth/authorize with the correct client_id, redirect_uri, scopes (read:user user:email), and PKCE code_challenge. Co-Authored-By: Claude Opus 4.7 (1M context) --- deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml b/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml index fe7ef63..806bf15 100644 --- a/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml +++ b/deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml @@ -6,8 +6,10 @@ metadata: namespace: codeforphilly-rewrite-sandbox spec: encryptedData: - CFP_DATA_REMOTE: AgBdON1Ac/LBCPg3syYH1D4uNPnJOEP75PSRjaktkKpUhxVffGQgOyhCsfsV/sn7YRHU5cdtNJhyL9pJjJabVD6BfDLvtpHOSm/G3fiUqnjSVpKzcyXxKrPsBkYJfcTOyH5gN+AoDihIvxkrpgg1hdby9aC7IONHSh+DpEuUXgsiDWcu8bt4l18YCwByYLNgBLQLPNs4/Stv7FHzU99ec4coWmIWZedtAgsq4o7p5xMkNM2VfZ7L8V5Q5XUZOQShNfOXSqvfr/DGbigkzTZdGm9Hrq/D+SJSsNaRmN0OJaHHo0LBrGIOyWMzDZz9vO6lI/REjFmFijg8dpHMv1rVeF1y8xMqL9W/vASg1C0oceHHSBK13NTnZGCpPKvzRwaV1AhcHCdTObWFv/VnmCT56LtpfxZcjkfPSwiF3ZvUlhcu0HHpqUzR26eg1uKZt8OlUJE0gJo0tF+X2ulYMFSTyyXkl/Y+xu2qBmmknuoI2UVrDmKCr6PqL/O0CDjSZQ+OT2oSi7VXNg5LBwfzkneSGE0dnSssmxro4wVchdSze87HK8eZUdl8RJn8cMLAai+SxwelKvGVUbzD4GCrdiKE3cL5+4fgIFJZX24w/dFWxBjCXDuJD5KEfxFKbGyR/tD3cEiVXisABVeOtuBuprceNJJJtD9Tep/xGMJpTND1V8jqmVB0OKmeub0ruL3MBzHuGKsYdZz0oMBevzJCMuEiyyTzOy5/COl2o1pBymFwk00AGpKzkCsjAMP9hk3Opm6hLBgpYCs= - CFP_JWT_SIGNING_KEY: AgAy03/R7XN6y5qtsLe6o1HriQMQQh3FrHZkUL9+SWti9b2idHqZql6T1wXIk21axTB5MqiCQuURiGGW+V+7kSCZVJByqe6T/026WQcI6tNVEMLOiSZ5TfKQT4Cwota7CqiC1bqtuXh1oYDEn9U1kQ08sVy12kgmVDlwOsGXQrO7+wTVOMZOsaO/97SpN7L04a4uf2PKag4N7tg/WeRxvzDuNhfaPh6USCWVZmgijzGoqTrr/HJbXzbvK7cddoUnh4EYmxxnM3z/hrghgPLYg/QD1wWYicQq+BMdKv+JR+Zy98fuB+PTtmgPj7c2A8ot99RtgUTq4ZKQ3YOuIINsUY0iauikrU5DFg9JSTNgsE1Idt1Rpf8a5zy09x84llPcp/XGBJL0f2W0JH9r8TTdT+0MZ4uVuQTNaWTVET6ei6cw1Wohkrb3l4Qp3ix6Rw0gxt8M4534JPaDe3uXdcSYoNnoiq8fxrTd0Dt8CJ1xCx3902pAyV/xS/9um1azF7USl3mXad+UOhupa21Yf1fMnq7ZxH+wqOoFLdcv0TzQc5jmxDDxNKzRkG7UstfAA7C8N16uvmt5OiZrYnQe15wIJktYySL55UY9wmC3qxeCZvrljC25+nu5SlcUa4YhKADTQE/jizFaBtnuumRKf08lAaRm9n2f2i21gPq7UY0xdBXUmz9DKlVdfkPlIxm0pwrckCr3B6szwZQeoq4nUJ7iDcVJ5pT7XpS8G6im0G4UzjBb4kgOcx36KeGZYeWdoGleQCXQGdxc/mRIfONawuipCSYG + CFP_DATA_REMOTE: AgAj4UY/M3uFDp4jJLykFSzihy+C2uSQXHQuuyaOAd7ew14gpw6Vogu5oYMgM1y1yBv1sbJObbnh7ddexYqRfgFKJrgiwc60MoVzwbwt5JHQh1aJJIZ4xizvitbyT1h7GkLzLZbvLdJxAFH876wXn9zw0UAco62qsxkIQgoRGBN/Gq7SQDT6+HHv1Uc3LeDiqdnne49LIP/M32DSu8IQ0CffyA+pdAdTwCaRoMBh63rdhoLM4Ildvlt6jSWHP0dJFmYRP9gtKCKgLUycz7fXhU0xxYaAOc7T2Z0rVf6DkkZDaqwgKEFk52300kuz1xW3BufrZ2jCfcXs4AAwwUuWj17MNICq0dei7MDRurmGpgJJDdfaCvTD4+wggm/VUBvXWQIuhE9/PtIumx/brVCJQ05YmeyFSrL4D+COSmhlXS3vmngpy3lHB+2qEbwM8ZHnwB/Ff1XP4HVSPUdXejWU4vdFY6VQ/4hjCLDAgjQLGRTGBmaFX1aJqygjK1JzRnGts0aCniPTL/qF1s5OcHJeIAvkN0WleceEi5u7CnvEdAcSAfkYu0QiwnoXNeIhFzPekEJyP6c6KY9oaw+kEirWgvCwis5QUiGOIO4kFcGXfDVzCRu1GM4T7WV969n8nJSGcSQQSmVa4uPCAlZld8etvbR4xa8dSXaeZ6Nb6L3FIVqc05Zy0U9cnKnfSGKWvHEANtBA7ItPq9GUlE6DCTb9guwlUGWHpG7tk1iuTnUD5b0P5r7zALF7rAufCKHEPpNWn7Mfg/k= + CFP_JWT_SIGNING_KEY: AgClhcixIR+sgvwpmHiqkB3MOL9CTCgjRT7d7AcWkYe7tjXCdvj42x5RiTbHRJh7h8koWwvm/M9txEhCnkV8iT8O6vLAq1ZPBvSk+L+uh6dZDW92u8L0/WIpVxLF9CjLmD7YbLpt2LA7ntne3pbZM6XgH6Xs9yYVEdLhEZ4/vOgzwJF1o3rj6puVsx5pWdUFgl1AXsjUdwUEK7fYSyQIXOLeLY3TVquTRATC/BEXaJHTSqHJjGaDemr1+ZqdbysFCADtOOhjOUxbItfGsm0sIUFEFkPOf+rGrTt0Mqp8CAoJEgkwEITHjA8qXKsJN/zZmDDh5xutCw36k5KtABUMPcn+QT6EjK+N9EutM08Ul+DZL+KfHg5z9/B/3lL68xU/J7H2TbB93icSDgbvV/NEjiyZ7JnaZqt+fPDpl0Gs/QMAdjNxtqYCYp9gFj5MsM0s5qkx4mnrgSHyOACb2H96wqVI7LvIZqhjnpNqY5V8Qvhb7rjLedEsmLDfPSRJYrg7s/cpmdxaCyuBf/zqUSSCvjKo7L66lZ0t3lG+dtVy3fgRoabeOGB6hOY5Jj1GAK8avC4PZ0vLuYTTPIweOJOIakSL37Rxit4UgvuuoiB6oEqnsDD/9YBeZAJNjf3XNDDud7dScWhI1zRJ9N7qbHeAcSkWOO/CK+cFWXVB31ErBCrDIuiwiMnna5VurC96+xASRxGdeDh83I3f3uTM5OQCH0efUAHhw5ld3IAu/2wpfQK3ac2XziYn7L2QIyHOsULkAf0oRoJjodOX3V2Kmp3khY8w + GITHUB_OAUTH_CLIENT_ID: AgBZj/mQl8EKMl6Cey9OWK090/IecVx6YlonWqh7v63QOVwyFzj6ksObLC2Fw0H9OHaFtL3qGF1qGn35MyYI3uhGB1miLLYivGVA7jNHN8Wy4es/tDnZRHcsP4LI7NQMTsf1dz4tknLinpmZqPuMwwqcUCRrSasagmYLHREXZNvGm1ONSSniuoqmnQobj2v7/YNHhn7qK/kVqflOGb0n9Ai4bHvrgGuxXTHCvl/3N8Mb4p3/aWAJuxpFgc6n21MS7Hjn5mcMb/RwJ7TOn7F/BJnQ4Ii2nzZ5d6U0F37lnMBdN6/Bowp+MNynTttJ+SBUZkjidJ+9s/KXLUXJ8IQp+NR9Ycbbc0+osnHIAhZzBF7+4du0f0S7Te310vll6b4xd0oRoXSQ9FZcqlS5ysOCdgH98q9OWw5TJWF8KfwNq+j62YifICSz5u5FfUnbzTgxe7RctIkhW0elgHTJx07WJx2I53PrxbbrICnCqDSMQbWU5wcqyCXxmFXOBSMZG0Gycjg2rAWaPgot7Q7F752I8oOewB28uACYjHlULR/2BKrCf90+RZOFI4AhUctvTTMn4qR89QrDE5EIdJi5x/x7pcMiM6doWrL1MJdnQ/VEUa+AsJEDVWA0G1a4y/OIRI7rKyYZdpUSabmzQTBxk0CBMa07dX3N5IaX0+YDt2NosW+xtsjQugifYPze/AnKIFjdB5cHVQzdVWHIls4WKR9bOahstrScrQ== + GITHUB_OAUTH_CLIENT_SECRET: AgA5TBdNd4A6g04YyvGd+ZkpdTrQFeODGLY3MSA+M8STT6ejTBtlVmj0m4ooggPtOYXaUAYAytlANT51yRE7GGbAX5/SiHO04N4MJ2Qpw4Izb1j5OvqZSNRD/78D8Q2KUg+cp/AyqwGctG0v2UYg/qBw536+4q53EMSQzcS2HZADqw5sZNyuqBwA+vUfXFHXqU+ZZw+Dqa5E+2tdCLu6lZdsmM0oWM/iAbKfszT9aNRYJ8I2W6NPyUHLFmoRcDtxYKs1Wqk29Dhfuk7ZsanFrkN0C48a2F1167qhBJMf0dIZ7ZeenFpSoPv5XODOnB89G+ziA7pQVcccegZpA4/rhgoPV8jfFecuY8U+FqDQaAOxwVEXvFZ1kwbsYHO0Ggo1rPivJcCh1oYWufCWFvwLCzIQpZ+KV1wAzjrkcnNeM8kFSwnp3ptbI9ZzgxEbn5zcDC+mRP6O07urKta+4qW+TGepaUuDWN/Ocsz1UP6vEQMtr6fChAxkJFjfJo8O1270VlDLBa0vwF3rmtsUpsfR+OMk/kU7rjrTuJWLcXDiKWoZKprPivWqCcGGLh/7MVsDOYGo9Pc4f4fhrlv14cxIZLc9PhPhgY9MEkZhnqg3RCikRuB9otL6EkPrXx5+qiTl5JsP1Xau9mM77hE2eMY2lyWeokRvCWS7jkBR4q0gL5iuonHMck5ccsiFwBGHP7VFUQeo92p1FZdbX1g5gph7b942uRyuthZTKdfwD/qSBseg4r/06nLlK+Jh template: metadata: name: codeforphilly-secrets From 258da83c81b4e4fe6c1e2cb76237f052af6b4630 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 10:04:40 -0400 Subject: [PATCH 07/10] fix(api): look up session person from in-memory state by id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The session middleware (and a couple of sibling lookups in the auth and SAML routes) was reaching for the Person via `store.public.people.queryFirst({ id })`, which doesn't reflect commits made in the same process between transact and the next sheet refresh. Newly-created users (the create-fresh OAuth path) got a valid session JWT but the middleware's lookup returned null, so the SPA's /api/auth/me saw `accountLevel: user` with `person: null` and rendered as not-signed-in. Switched all three call sites to `fastify.inMemoryState.people.get(id)` — the canonical fast index already used by every other id→Person lookup in the codebase (projects-members, projects-buzz). Same source of truth that `stateApply.apply()` writes to immediately after the gitsheets transact. Co-Authored-By: Claude Opus 4.7 (1M context) --- apps/api/src/auth/middleware.ts | 10 +++++++--- apps/api/src/routes/auth.ts | 2 +- apps/api/src/routes/saml.ts | 4 +--- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/apps/api/src/auth/middleware.ts b/apps/api/src/auth/middleware.ts index 2093bd3..16a02cb 100644 --- a/apps/api/src/auth/middleware.ts +++ b/apps/api/src/auth/middleware.ts @@ -104,11 +104,15 @@ async function sessionMiddlewarePlugin(fastify: FastifyInstance): Promise return; } - // Look up person from public store - const person = await fastify.store.public.people.queryFirst({ id: claims.sub } as Record); + // Look up person from the in-memory state map (keyed by id). Other + // routes use the same path (`fastify.inMemoryState.people.get(personId)`) + // for id→Person resolution; this is the canonical fast index. The + // sheet-level `queryFirst({ id })` previously used here doesn't reflect + // in-process writes between commit and the next refresh. + const person = fastify.inMemoryState.people.get(claims.sub) ?? null; request.session = { - person: person ?? null, + person, accountLevel: claims.accountLevel, personId: claims.sub, jti: claims.jti, diff --git a/apps/api/src/routes/auth.ts b/apps/api/src/routes/auth.ts index db8f777..108d6e1 100644 --- a/apps/api/src/routes/auth.ts +++ b/apps/api/src/routes/auth.ts @@ -288,7 +288,7 @@ export async function authRoutes(fastify: FastifyInstance): Promise { throw new UnauthenticatedError('Refresh token revoked', 'refresh_token_revoked'); } - const person = await fastify.store.public.people.queryFirst({ id: claims.sub }); + const person = fastify.inMemoryState.people.get(claims.sub); if (!person) { throw new UnauthenticatedError('Person not found', 'refresh_token_revoked'); } diff --git a/apps/api/src/routes/saml.ts b/apps/api/src/routes/saml.ts index df7b736..d07c7d0 100644 --- a/apps/api/src/routes/saml.ts +++ b/apps/api/src/routes/saml.ts @@ -227,9 +227,7 @@ async function loadPersonAndProfile( fastify: FastifyInstance, personId: string, ): Promise<{ person: Person; profile: PrivateProfile }> { - const person = (await fastify.store.public.people.queryFirst({ id: personId })) as - | Person - | undefined; + const person = fastify.inMemoryState.people.get(personId); if (!person) { throw new UnauthenticatedError('Person not found', 'unauthenticated'); } From 5c98144070f8737bce5dbcde9a0f5b5a7de78d7e Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 10:05:14 -0400 Subject: [PATCH 08/10] feat(api): wire startPushDaemon at boot (closes #37) The deploy plan shipped the deploy-key mount + GIT_SSH_COMMAND wiring but never actually called `repo.startPushDaemon()`, so every commit produced by store.transact landed on the local branch and was wiped the next time the entrypoint refreshed against origin. Without this, accounts created via OAuth survived only until the next pod restart. - `openPublicStore` now returns `{ store, repo }` so the booted Repository handle is reachable from the plugin layer - `bootStores` plumbs the repo through as `publicRepo` - New `apps/api/src/plugins/push-daemon.ts` starts the daemon when `CFP_DATA_REMOTE` is set; emits push/retry/error logs; discriminates non-fast-forward (terminal) from transient (retried) failures per gitsheets 1.0.5+ - Fastify `onClose` hook stops the daemon on SIGTERM - `fastify.pushDaemon` decoration (typed `PushDaemon | null`) lets future admin/status routes surface daemon state - `CFP_DATA_BRANCH` lifted into the API env schema so the daemon pushes to the right branch (it was previously only consumed by the entrypoint) - Scripts/tests using `openPublicStore` updated to destructure the new return shape and reference the exported `PublicStore` type directly Verified: boot logs show `push-daemon started remote=origin branch=fixture` on the running sandbox pod. The follow-up problem of the entrypoint's unconditional `git reset --hard origin/` discarding local commits between pod-terminate and the daemon's next push tick is separate and will be addressed in a follow-up commit. Co-Authored-By: Claude Opus 4.7 (1M context) --- apps/api/scripts/cutover-dry-run.ts | 2 +- apps/api/scripts/cutover-mailout.ts | 8 +-- apps/api/scripts/reconcile.ts | 6 +- apps/api/src/app.ts | 4 ++ apps/api/src/env.ts | 3 + apps/api/src/plugins/push-daemon.ts | 87 ++++++++++++++++++++++++++ apps/api/src/plugins/store.ts | 6 +- apps/api/src/store/boot.ts | 11 ++-- apps/api/src/store/public.ts | 12 +++- apps/api/tests/cutover-mailout.test.ts | 6 +- apps/api/tests/reconcile.test.ts | 12 ++-- 11 files changed, 132 insertions(+), 25 deletions(-) create mode 100644 apps/api/src/plugins/push-daemon.ts diff --git a/apps/api/scripts/cutover-dry-run.ts b/apps/api/scripts/cutover-dry-run.ts index 6ee545d..6999769 100644 --- a/apps/api/scripts/cutover-dry-run.ts +++ b/apps/api/scripts/cutover-dry-run.ts @@ -327,7 +327,7 @@ export async function runDryRun(opts: DryRunOptions): Promise { let smokeChecks: SmokeCheckResult[] = []; if (opts.target) { - const publicStore = await openPublicStore(opts.dataRepo); + const { store: publicStore } = await openPublicStore(opts.dataRepo); const people = await publicStore.people.queryAll(); const projects = await publicStore.projects.queryAll(); const liveProjects = projects.filter((p) => !p.deletedAt); diff --git a/apps/api/scripts/cutover-mailout.ts b/apps/api/scripts/cutover-mailout.ts index e959cdd..93a72a8 100644 --- a/apps/api/scripts/cutover-mailout.ts +++ b/apps/api/scripts/cutover-mailout.ts @@ -22,7 +22,7 @@ import { writeFile } from 'node:fs/promises'; import { resolve } from 'node:path'; -import { openPublicStore } from '../src/store/public.js'; +import { openPublicStore, type PublicStore } from '../src/store/public.js'; import { FilesystemPrivateStore, S3PrivateStore, @@ -50,7 +50,7 @@ export interface MailoutReport { } export interface MailoutOptions { - readonly publicStore: Awaited>; + readonly publicStore: PublicStore; readonly privateStore: PrivateStore; readonly mode: 'dry-run' | 'send'; readonly from?: string; @@ -64,7 +64,7 @@ export interface MailoutOptions { // --------------------------------------------------------------------------- export async function collectRecipients( - publicStore: Awaited>, + publicStore: PublicStore, privateStore: PrivateStore, ): Promise<{ recipients: MailoutRecipient[]; skipped: Array<{ personId: string; reason: string }> }> { const people = await publicStore.people.queryAll(); @@ -289,7 +289,7 @@ async function main(): Promise { process.exit(2); } - const publicStore = await openPublicStore(requireEnv('CFP_DATA_REPO_PATH')); + const { store: publicStore } = await openPublicStore(requireEnv('CFP_DATA_REPO_PATH')); const privateStore = buildPrivateStore(); await privateStore.load(); diff --git a/apps/api/scripts/reconcile.ts b/apps/api/scripts/reconcile.ts index 6cd06ea..388f92a 100644 --- a/apps/api/scripts/reconcile.ts +++ b/apps/api/scripts/reconcile.ts @@ -41,7 +41,7 @@ import { PrivateProfileSchema, type PrivateProfile, } from '@cfp/shared/schemas'; -import { openPublicStore } from '../src/store/public.js'; +import { openPublicStore, type PublicStore } from '../src/store/public.js'; import { FilesystemPrivateStore, S3PrivateStore, @@ -118,7 +118,7 @@ function buildPrivateStore(): PrivateStore { // --------------------------------------------------------------------------- export interface ReconcileOptions { - readonly publicStore: Awaited>; + readonly publicStore: PublicStore; readonly privateStore: PrivateStore; readonly fix?: boolean; readonly now?: string; @@ -305,7 +305,7 @@ async function main(): Promise { const args = parseArgs(process.argv.slice(2)); const repoPath = requireEnv('CFP_DATA_REPO_PATH'); - const publicStore = await openPublicStore(repoPath); + const { store: publicStore } = await openPublicStore(repoPath); const privateStore = buildPrivateStore(); await privateStore.load(); diff --git a/apps/api/src/app.ts b/apps/api/src/app.ts index be15fb6..7d13bd1 100644 --- a/apps/api/src/app.ts +++ b/apps/api/src/app.ts @@ -29,6 +29,7 @@ import { envJsonSchema, type Env } from './env.js'; import { mapError } from './lib/errors.js'; import traceIdPlugin from './plugins/trace-id.js'; import storePlugin from './plugins/store.js'; +import pushDaemonPlugin from './plugins/push-daemon.js'; import servicesPlugin from './plugins/services.js'; import rateLimitPlugin from './plugins/rate-limit.js'; import idempotencyPlugin from './plugins/idempotency.js'; @@ -110,6 +111,9 @@ export async function buildApp(opts: BuildAppOptions = {}): Promise { + if (!fastify.config.CFP_DATA_REMOTE) { + fastify.log.info( + 'push-daemon disabled: CFP_DATA_REMOTE unset (local commits stay local)', + ); + fastify.decorate('pushDaemon', null); + return; + } + + const daemon = await fastify.publicRepo.startPushDaemon({ + remote: 'origin', + branch: fastify.config.CFP_DATA_BRANCH, + backoff: 'exponential', + }); + + daemon.on('push', ({ commit, durationMs }: { commit: string; durationMs: number }) => { + fastify.log.info({ commit, durationMs }, 'pushed commit to origin'); + }); + daemon.on('retry', ({ attempt, nextDelayMs }: { attempt: number; nextDelayMs: number }) => { + fastify.log.info({ attempt, nextDelayMs }, 'push-daemon retrying'); + }); + daemon.on( + 'error', + ({ + err, + attempt, + reason, + }: { + err: unknown; + attempt: number; + reason: 'non-fast-forward' | 'unknown'; + }) => { + if (reason === 'non-fast-forward') { + fastify.log.error( + { err: String(err), attempt }, + 'push rejected non-fast-forward — manual reconciliation required', + ); + } else { + fastify.log.warn({ err: String(err), attempt, reason }, 'push attempt failed'); + } + }, + ); + + fastify.decorate('pushDaemon', daemon); + + fastify.addHook('onClose', async () => { + fastify.log.info('stopping push-daemon'); + await daemon.stop(); + }); + + fastify.log.info( + { remote: 'origin', branch: fastify.config.CFP_DATA_BRANCH ?? 'HEAD' }, + 'push-daemon started', + ); +} + +export default fp(pushDaemonPlugin, { + name: 'push-daemon', + fastify: '5.x', + dependencies: ['store'], +}); diff --git a/apps/api/src/plugins/store.ts b/apps/api/src/plugins/store.ts index 020556d..2d6f5e7 100644 --- a/apps/api/src/plugins/store.ts +++ b/apps/api/src/plugins/store.ts @@ -9,17 +9,20 @@ */ import type { FastifyInstance } from 'fastify'; import fp from 'fastify-plugin'; +import type { Repository } from 'gitsheets'; import { bootStores } from '../store/boot.js'; import type { Store } from '../store/store.js'; declare module 'fastify' { interface FastifyInstance { store: Store; + /** Underlying gitsheets repo for the public data store — used by the push-daemon plugin. */ + publicRepo: Repository; } } async function storePlugin(fastify: FastifyInstance): Promise { - const store = await bootStores({ + const { store, publicRepo } = await bootStores({ CFP_DATA_REPO_PATH: fastify.config.CFP_DATA_REPO_PATH, STORAGE_BACKEND: fastify.config.STORAGE_BACKEND, CFP_PRIVATE_STORAGE_PATH: fastify.config.CFP_PRIVATE_STORAGE_PATH, @@ -31,6 +34,7 @@ async function storePlugin(fastify: FastifyInstance): Promise { }); fastify.decorate('store', store); + fastify.decorate('publicRepo', publicRepo); } export default fp(storePlugin, { diff --git a/apps/api/src/store/boot.ts b/apps/api/src/store/boot.ts index a82e597..e3ee9c2 100644 --- a/apps/api/src/store/boot.ts +++ b/apps/api/src/store/boot.ts @@ -1,3 +1,4 @@ +import type { Repository } from 'gitsheets'; import { FilesystemPrivateStore } from './private/filesystem.js'; import { S3PrivateStore } from './private/s3.js'; import { openPublicStore } from './public.js'; @@ -21,7 +22,9 @@ export interface Env { } /** - * Boot both stores and return a combined Store. + * Boot both stores and return a combined Store + the underlying public-repo + * handle. The repo handle is consumed by the push-daemon plugin to push + * commits to origin; everything else only needs `store`. * * Fails loudly (throws) if either store is unreachable. The API must not * serve traffic until this resolves — private profiles are required for login. @@ -31,8 +34,8 @@ export interface Env { * 2. Private store data * 3. (FTS index is built by the caller from the loaded public data) */ -export async function bootStores(env: Env): Promise { - const publicStore = await openPublicStore(env.CFP_DATA_REPO_PATH).catch((err) => { +export async function bootStores(env: Env): Promise<{ store: Store; publicRepo: Repository }> { + const { store: publicStore, repo: publicRepo } = await openPublicStore(env.CFP_DATA_REPO_PATH).catch((err) => { throw new Error(`Failed to open public gitsheets store at ${env.CFP_DATA_REPO_PATH}: ${String(err)}`, { cause: err }); }); @@ -46,7 +49,7 @@ export async function bootStores(env: Env): Promise { throw new Error(`Failed to load private store (${env.STORAGE_BACKEND}): ${String(err)}`, { cause: err }); }); - return new Store(publicStore, privateStore); + return { store: new Store(publicStore, privateStore), publicRepo }; } function buildPrivateStore(env: Env): FilesystemPrivateStore | S3PrivateStore { diff --git a/apps/api/src/store/public.ts b/apps/api/src/store/public.ts index e800675..160e1e9 100644 --- a/apps/api/src/store/public.ts +++ b/apps/api/src/store/public.ts @@ -1,5 +1,5 @@ import { openRepo, openStore } from 'gitsheets'; -import type { StandardSchemaV1, Store, StoreTx, ValidatorMap } from 'gitsheets'; +import type { Repository, StandardSchemaV1, Store, StoreTx, ValidatorMap } from 'gitsheets'; import { HelpWantedInterestExpressionSchema, HelpWantedRoleSchema, @@ -63,8 +63,13 @@ export type PublicStoreTx = StoreTx; * Reads `.gitsheets/.toml` for each declared sheet in `repoPath`. * In-memory secondary indices are built by the caller (boot.ts) after this * returns, since they require iterating over all records. + * + * Returns both the typed store and the underlying Repository handle — the + * latter is needed by the push-daemon plugin to push commits to origin. */ -export async function openPublicStore(repoPath: string): Promise { +export async function openPublicStore( + repoPath: string, +): Promise<{ store: PublicStore; repo: Repository }> { const repo = await openRepo({ gitDir: `${repoPath}/.git`, workTree: repoPath }); repo.requireExplicitTransactions(); @@ -82,5 +87,6 @@ export async function openPublicStore(repoPath: string): Promise { revocations: asValidator(RevocationSchema), }; - return openStore(repo, { validators }) as Promise; + const store = (await openStore(repo, { validators })) as PublicStore; + return { store, repo }; } diff --git a/apps/api/tests/cutover-mailout.test.ts b/apps/api/tests/cutover-mailout.test.ts index 51ead17..4ad0e3b 100644 --- a/apps/api/tests/cutover-mailout.test.ts +++ b/apps/api/tests/cutover-mailout.test.ts @@ -62,7 +62,7 @@ describe('cutover-mailout', () => { await seedPerson(repo.path, { id: danId, slug: 'dan' }); await seedPerson(repo.path, { id: eveId, slug: 'eve', deletedAt: NOW }); - const publicStore = await openPublicStore(repo.path); + const { store: publicStore } = await openPublicStore(repo.path); await privateStore.putProfile({ personId: aliceId, @@ -138,7 +138,7 @@ describe('cutover-mailout', () => { const personId = uuid(7); await seedPerson(repo.path, { id: personId, slug: 'frank' }); - const publicStore = await openPublicStore(repo.path); + const { store: publicStore } = await openPublicStore(repo.path); await privateStore.putProfile({ personId, email: 'frank@example.com', @@ -177,7 +177,7 @@ describe('cutover-mailout', () => { const personId = uuid(8); await seedPerson(repo.path, { id: personId, slug: 'gail' }); - const publicStore = await openPublicStore(repo.path); + const { store: publicStore } = await openPublicStore(repo.path); await privateStore.putProfile({ personId, email: 'gail@example.com', diff --git a/apps/api/tests/reconcile.test.ts b/apps/api/tests/reconcile.test.ts index 9cb1dd1..e95e572 100644 --- a/apps/api/tests/reconcile.test.ts +++ b/apps/api/tests/reconcile.test.ts @@ -15,7 +15,7 @@ import { describe, expect, it } from 'vitest'; import { openRepo } from 'gitsheets'; import { reconcile } from '../scripts/reconcile.js'; -import { openPublicStore } from '../src/store/public.js'; +import { openPublicStore, type PublicStore } from '../src/store/public.js'; import { FilesystemPrivateStore } from '../src/store/private/filesystem.js'; import { createFullDataRepo, createPrivateStorageDir } from './helpers/test-full-repo.js'; @@ -29,7 +29,7 @@ function uuid(n: number): string { interface Fixture { repo: Awaited>; priv: Awaited>; - publicStore: Awaited>; + publicStore: PublicStore; privateStore: FilesystemPrivateStore; } @@ -40,7 +40,7 @@ async function bootFixture(): Promise { CFP_PRIVATE_STORAGE_PATH: priv.path, }); await privateStore.load(); - const publicStore = await openPublicStore(repo.path); + const { store: publicStore } = await openPublicStore(repo.path); return { repo, priv, publicStore, privateStore }; } @@ -77,7 +77,7 @@ describe('reconcile', () => { const personId = uuid(1); await seedPerson(f.repo.path, { id: personId, slug: 'alice' }); // Re-open the store so it sees the new commit. - const publicStore = await openPublicStore(f.repo.path); + const { store: publicStore } = await openPublicStore(f.repo.path); const report = await reconcile({ publicStore, @@ -125,7 +125,7 @@ describe('reconcile', () => { try { const personId = uuid(3); await seedPerson(f.repo.path, { id: personId, slug: 'bob' }); - const publicStore = await openPublicStore(f.repo.path); + const { store: publicStore } = await openPublicStore(f.repo.path); await f.privateStore.putProfile({ personId, email: 'bob@example.com', @@ -168,7 +168,7 @@ describe('reconcile', () => { slug: 'carol', githubUserId: 99001, }); - const publicStore = await openPublicStore(f.repo.path); + const { store: publicStore } = await openPublicStore(f.repo.path); await f.privateStore.putProfile({ personId, email: 'carol@example.com', From 7b79c748f048bf98c5e7c4fb838a2f080a26cb21 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 10:34:56 -0400 Subject: [PATCH 09/10] fix(deploy): smart entrypoint reconciliation with conflict escape hatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous entrypoint did `git fetch && git reset --hard origin/` on every boot, which discarded any commits the API made locally that the push daemon hadn't yet pushed (the structural cause of the post-OAuth account-loss bug). With the push daemon now wired (5c98144 / #37), the entrypoint can be smarter: Boot-time states + actions, in order of cheapness: - in sync with origin → no-op - behind origin → `git merge --ff-only` - ahead of origin → push (push-daemon retries on failure) - diverged + clean rebase → rebase onto origin, then push - diverged + conflicting → escape hatch: 1. abort rebase 2. create `conflicts/` branch at the pre-rebase HEAD 3. push that branch to origin (loudly logged for operators) 4. hard-reset local to origin so the pod boots from known-good state Safety properties: - Work is never silently dropped; a conflict produces a named branch. - Fetch failures (network blips) are non-fatal; entrypoint falls through to API start with whatever's locally available. - Push failures during reconciliation are non-fatal; the running push daemon retries. - Drops `--depth=1` from initial clone — rebase needs the merge-base. Existing shallow PVCs are unshallowed on first boot of this image. Pseudonymous identity (api@users.noreply.codeforphilly.org) is used for the entrypoint's own git operations; commit authors are preserved on rebase. Co-Authored-By: Claude Opus 4.7 (1M context) --- deploy/docker/entrypoint.sh | 187 ++++++++++++++++++++++++++++-------- 1 file changed, 145 insertions(+), 42 deletions(-) diff --git a/deploy/docker/entrypoint.sh b/deploy/docker/entrypoint.sh index 85c15fe..1f419c5 100755 --- a/deploy/docker/entrypoint.sh +++ b/deploy/docker/entrypoint.sh @@ -1,21 +1,36 @@ #!/bin/sh # CodeForPhilly API entrypoint. # -# Per specs/architecture.md, on pod start: -# 1. Runs `git clone` / `git fetch && git reset --hard origin/` -# against CFP_DATA_REMOTE to populate the data-repo working tree. -# 2. exec node apps/api/dist/index.js +# On pod start: +# 1. Ensures a workable clone of CFP_DATA_REMOTE exists at CFP_DATA_REPO_PATH. +# 2. Reconciles local commits (made by the previous pod's runtime that the +# push daemon hadn't yet pushed) with origin: +# - in sync → no-op +# - behind → fast-forward +# - ahead → push pending commits to origin +# - diverged + clean rebase → rebase + push +# - diverged + conflicts → push a `conflicts/` branch +# to origin for operator review, then hard-reset local to origin so +# the pod boots from a known-good state. Never silently drops work. +# 3. exec the API. # # Required env: # CFP_DATA_REPO_PATH — local working-tree path (mounted PVC in k8s) -# CFP_DATA_REMOTE — git URL to clone/fetch from # Optional env: -# CFP_DATA_BRANCH — branch to track (default: main) -# GIT_SSH_COMMAND — set by Helm when an SSH deploy key is mounted; usually -# `ssh -i /etc/cfp/git-deploy-key -o StrictHostKeyChecking=accept-new` +# CFP_DATA_REMOTE — git URL to clone/fetch/push. If unset, the entrypoint +# assumes an offline-style dev setup and uses whatever +# working tree is already at CFP_DATA_REPO_PATH. +# CFP_DATA_BRANCH — branch to track (default: main). +# GIT_SSH_COMMAND — set when an SSH deploy key is mounted. # -# Failure modes: any non-zero exit causes the container to crash. K8s restarts -# it. Readiness probe stays 503 until /api/health/ready returns 200. +# Failure modes: +# - Fetch failures are non-fatal — log + continue with local state. The +# push-daemon retries on its schedule. +# - Push failures during reconciliation are non-fatal — the push-daemon +# retries once the API starts. +# - Rebase conflicts trigger the escape hatch (conflict branch + hard reset). +# The API still boots; the operator investigates the named branch. +# - Anything else (clone failure, etc.) crashes the container; k8s restarts. set -eu @@ -29,13 +44,119 @@ DATA_BRANCH="${CFP_DATA_BRANCH:-main}" # Trust the data-repo working tree regardless of file ownership. PVCs survive # pod restarts and may carry files owned by a different uid than this pod's -# runAsUser (e.g., an earlier iteration ran as root). Without this, git -# refuses every operation with "detected dubious ownership". +# runAsUser (e.g., an earlier iteration ran as root). git config --global --add safe.directory "$CFP_DATA_REPO_PATH" +# Identity for any direct git operations made by the entrypoint (rebase +# preserves authors of existing commits; this just covers the committer when +# rebase actually rewrites a commit). API mutations supply their own GIT_AUTHOR_* +# via gitsheets transaction options. +: "${GIT_AUTHOR_NAME:=CodeForPhilly API}" +: "${GIT_AUTHOR_EMAIL:=api@users.noreply.codeforphilly.org}" +: "${GIT_COMMITTER_NAME:=$GIT_AUTHOR_NAME}" +: "${GIT_COMMITTER_EMAIL:=$GIT_AUTHOR_EMAIL}" +export GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL + +# --------------------------------------------------------------------------- +# Reconcile against origin. Returns 0 on success or a soft failure; only +# unrecoverable filesystem/clone errors propagate via `set -e`. +# --------------------------------------------------------------------------- +reconcile() { + cd "$CFP_DATA_REPO_PATH" + + git config user.name "$GIT_AUTHOR_NAME" + git config user.email "$GIT_AUTHOR_EMAIL" + git remote set-url origin "$CFP_DATA_REMOTE" + + # Unshallow if a previous clone used --depth=1; the reconciliation logic + # below needs the merge-base to be reachable. + if [ -f .git/shallow ]; then + log "unshallowing existing clone (needed for rebase)" + git fetch --unshallow origin "$DATA_BRANCH" 2>&1 | sed 's/^/ /' || \ + log "WARN: --unshallow failed; continuing with shallow history" + fi + + if ! git fetch --prune origin "$DATA_BRANCH" 2>&1 | sed 's/^/ /'; then + log "WARN: fetch failed; skipping reconciliation, using local state" + return 0 + fi + + # Ensure we're on the branch. + if git rev-parse --verify "refs/heads/$DATA_BRANCH" >/dev/null 2>&1; then + git checkout "$DATA_BRANCH" 2>&1 | sed 's/^/ /' + else + git checkout -b "$DATA_BRANCH" "origin/$DATA_BRANCH" 2>&1 | sed 's/^/ /' + fi + + LOCAL=$(git rev-parse HEAD) + REMOTE=$(git rev-parse "origin/$DATA_BRANCH") + if ! BASE=$(git merge-base HEAD "origin/$DATA_BRANCH" 2>/dev/null); then + log "WARN: no merge-base with origin/$DATA_BRANCH; resetting to origin" + git reset --hard "origin/$DATA_BRANCH" 2>&1 | sed 's/^/ /' + return 0 + fi + + if [ "$LOCAL" = "$REMOTE" ]; then + log "in sync with origin/$DATA_BRANCH" + return 0 + fi + + if [ "$LOCAL" = "$BASE" ]; then + log "behind origin/$DATA_BRANCH — fast-forwarding" + git merge --ff-only "origin/$DATA_BRANCH" 2>&1 | sed 's/^/ /' + return 0 + fi + + if [ "$REMOTE" = "$BASE" ]; then + AHEAD=$(git rev-list --count "origin/$DATA_BRANCH..HEAD") + log "ahead of origin/$DATA_BRANCH by ${AHEAD} commit(s) — pushing" + if git push origin "$DATA_BRANCH" 2>&1 | sed 's/^/ /'; then + log "push succeeded" + else + log "WARN: push failed; push-daemon will retry once API starts" + fi + return 0 + fi + + # Diverged: local has commits that origin doesn't AND origin has commits + # that local doesn't. Attempt a rebase; if it conflicts, escape-hatch. + AHEAD=$(git rev-list --count "origin/$DATA_BRANCH..HEAD") + BEHIND=$(git rev-list --count "HEAD..origin/$DATA_BRANCH") + log "diverged from origin/$DATA_BRANCH (ahead=${AHEAD}, behind=${BEHIND}) — rebasing" + + if git rebase "origin/$DATA_BRANCH" 2>&1 | sed 's/^/ /'; then + log "rebase clean — pushing" + if git push origin "$DATA_BRANCH" 2>&1 | sed 's/^/ /'; then + log "push succeeded" + else + log "WARN: push failed; push-daemon will retry once API starts" + fi + return 0 + fi + + # Conflict — escape hatch. + CONFLICT_BRANCH="conflicts/$(date -u +%Y-%m-%dT%H-%M-%SZ)" + log "ERROR: rebase conflict on $DATA_BRANCH — invoking escape hatch" + git rebase --abort 2>&1 | sed 's/^/ /' || true + log "preserving pre-rebase HEAD ($LOCAL) at $CONFLICT_BRANCH" + git branch "$CONFLICT_BRANCH" "$LOCAL" + if git push origin "$CONFLICT_BRANCH" 2>&1 | sed 's/^/ /'; then + log "pushed $CONFLICT_BRANCH to origin — operator must investigate" + else + log "WARN: failed to push $CONFLICT_BRANCH; diverged commits preserved only in this PVC's reflog" + fi + log "resetting $DATA_BRANCH to origin/$DATA_BRANCH" + git reset --hard "origin/$DATA_BRANCH" 2>&1 | sed 's/^/ /' + return 0 +} + if [ -z "${CFP_DATA_REMOTE:-}" ]; then if [ -d "$CFP_DATA_REPO_PATH/.git" ]; then log "CFP_DATA_REMOTE unset; using existing working tree at $CFP_DATA_REPO_PATH" + cd "$CFP_DATA_REPO_PATH" + git config user.name "$GIT_AUTHOR_NAME" + git config user.email "$GIT_AUTHOR_EMAIL" + cd - >/dev/null else log "ERROR: CFP_DATA_REMOTE is unset and $CFP_DATA_REPO_PATH is not a git repo" exit 1 @@ -44,44 +165,26 @@ else mkdir -p "$CFP_DATA_REPO_PATH" if [ -d "$CFP_DATA_REPO_PATH/.git" ]; then - log "refreshing existing data repo at $CFP_DATA_REPO_PATH (branch=$DATA_BRANCH)" - cd "$CFP_DATA_REPO_PATH" - - # Re-point origin in case CFP_DATA_REMOTE was rotated. - git remote set-url origin "$CFP_DATA_REMOTE" - git fetch --prune --depth=1 origin "$DATA_BRANCH" - git checkout -B "$DATA_BRANCH" "origin/$DATA_BRANCH" - git reset --hard "origin/$DATA_BRANCH" - cd - >/dev/null + log "reconciling existing data repo at $CFP_DATA_REPO_PATH (branch=$DATA_BRANCH)" + reconcile + cd - >/dev/null || true else - # PVC may carry residue from a previous pod that bailed mid-clone or from - # an earlier iteration of the entrypoint. `git clone` refuses to clone into - # a non-empty directory, so wipe it first. Safe because the data repo is - # always re-cloneable from CFP_DATA_REMOTE. + # PVC may carry residue from a previous pod that bailed mid-clone. + # `git clone` refuses to clone into a non-empty directory, so wipe it + # first. Safe because the data repo is always re-cloneable. if [ -n "$(ls -A "$CFP_DATA_REPO_PATH" 2>/dev/null)" ]; then log "$CFP_DATA_REPO_PATH non-empty but lacks .git — wiping before clone" find "$CFP_DATA_REPO_PATH" -mindepth 1 -maxdepth 1 -exec rm -rf {} + fi log "cloning $CFP_DATA_REMOTE into $CFP_DATA_REPO_PATH (branch=$DATA_BRANCH)" - # --depth=1 keeps the PVC footprint small; the push daemon will deepen as - # needed when it next pushes (or we accept periodic re-clones). - git clone --depth=1 --branch "$DATA_BRANCH" "$CFP_DATA_REMOTE" "$CFP_DATA_REPO_PATH" + # Full history (no --depth) so subsequent reconciliations can rebase. + git clone --branch "$DATA_BRANCH" "$CFP_DATA_REMOTE" "$CFP_DATA_REPO_PATH" + cd "$CFP_DATA_REPO_PATH" + git config user.name "$GIT_AUTHOR_NAME" + git config user.email "$GIT_AUTHOR_EMAIL" + cd - >/dev/null fi fi -# Identity for any commits the API makes (the gitsheets writer commits per -# mutation). Override via env in Helm values if you want per-environment -# identities. -: "${GIT_AUTHOR_NAME:=CodeForPhilly API}" -: "${GIT_AUTHOR_EMAIL:=api@codeforphilly.org}" -: "${GIT_COMMITTER_NAME:=$GIT_AUTHOR_NAME}" -: "${GIT_COMMITTER_EMAIL:=$GIT_AUTHOR_EMAIL}" -export GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL - -cd "$CFP_DATA_REPO_PATH" -git config user.name "$GIT_AUTHOR_NAME" -git config user.email "$GIT_AUTHOR_EMAIL" -cd - >/dev/null - log "data repo ready; starting API" exec "$@" From da0f86f2ad979d5a40b7a4f4295a61d92d68ddc5 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Mon, 18 May 2026 11:08:11 -0400 Subject: [PATCH 10/10] ci: build @cfp/shared before type-check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `packages/shared` exports compiled output from `dist/` (since the Docker build fix in f805a1a). Tests, type-check, and lint resolve `@cfp/shared` via the exports map, which means `dist/` must exist before any of them run — but CI was running type-check before build. Add an explicit `npm run -w packages/shared build` step right after `npm ci`. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/ci.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1cfa5dd..b262edb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,6 +21,11 @@ jobs: - name: Install dependencies run: npm ci + # @cfp/shared's exports map points at dist/. Build it first so type-check, + # lint, and test can resolve `@cfp/shared` / `@cfp/shared/schemas`. + - name: Build @cfp/shared + run: npm run -w packages/shared build + - name: Type check run: npm run type-check