Rotate the private knowledge token in the runtime secret

[alexeygrigorev]

Rotate the private knowledge token in the runtime secret

Status: blocked on HUMAN credentialed operator — the repository Actions secret is already rotated and validation dispatches are running; only an in-place update of the existing Secrets Manager value and sanitized portal read proof remain
Tags: bug, human, infra, docs, P0
Depends on: authorized credentialed AWS operator access through the normal AWS Gate; the already-created replacement value supplied through the approved private/no-echo handoff
Blocks: reliable production portal reads from the private knowledge repository
Next owner: [HUMAN] credentialed AWS operator for the exact value update, then On-Call for sanitized read verification and PM closeout
Resume condition: a named operator can access the deployed account/region without static AWS keys and can consume the replacement value without printing, logging, committing, or copying it into GitHub

Outcome

Update only the SecretString value of the existing Secrets Manager secret selected by the deployed stack's GitHubTokenSecretName. Then prove an authenticated production portal read reaches the private knowledge repository and record only safe metadata.

The GitHub token itself and the repository Actions secret DATAOPS_CONTENT_GITHUB_TOKEN were already rotated on 2026-08-13. Existing validation workflow dispatches exercise that Actions secret. Their result does not prove the runtime secret changed: CI and the application use separate secret stores.

No repository code, infrastructure template, CloudFormation update, new secret, runtime/admin rotation endpoint, migration machinery, compatibility token, manual application deploy, or static AWS credential is expected.

Known current state

• The replacement token exists only in the approved ignored local operator environment. It must never be read into this issue, command output, shell tracing, logs, screenshots, clipboard history, commits, or CI artifacts.
• The Actions secret update completed at 2026-08-13T19:37:56Z; the already-running validation dispatches should be allowed to finish. Do not rotate it again or create another dispatch solely for this issue.
• The application uses the existing Secrets Manager resource named by GitHubTokenSecretName; the deployed value shape is the token as a plain SecretString, not JSON.
• A prior local attempt failed closed at the AWS credential gate. That attempt changed no AWS value and is not operator failure evidence.
• The runtime resolves the Secrets Manager value and caches it inside an execution environment. An already-warm environment can temporarily retain the revoked value; no application deploy or configuration mutation is authorized to force recycling.

Safety and no-action boundary

PM grooming authorizes no .env or token read, AWS call, secret mutation, repository edit, test, workflow dispatch, deployment, runtime restart, GitHub-token regeneration, or portal write.

During execution:

• never enable shell tracing and never pass the token directly in command arguments, shell history, environment dumps, process listings, issue text, or screenshots;
• use the approved no-echo secure handoff and an operator-owned private input mechanism that preserves the exact token bytes; do not add JSON quoting or a trailing newline;
• target the one existing deployed secret only; do not call create-secret, change its name/policy/tags/replication/KMS configuration, create a second compatibility secret, or change CloudFormation;
• use only short-lived operator credentials obtained through the normal AWS Gate; do not create or store access keys;
• do not call get-secret-value merely to verify and never print old/new secret contents;
• do not manually deploy the application, update Lambda configuration, publish a version, or invoke a write route to force a cold start;
• do not test document create/edit/delete. This issue requires read proof only.

Phase 1 — preflight

• [HUMAN] Name the credentialed operator and On-Call verifier; privately confirm the intended AWS account, region, existing deployed secret identifier, and secure replacement-value source.
• Confirm the Actions secret was updated at the recorded timestamp. Capture links and terminal status for the already-running validation dispatches without exposing checkout URLs, token headers, or private content.
• Confirm the target secret already exists through metadata-only inspection and is the value referenced by the deployed application. Abort on a missing/ambiguous target, unexpected account/region, pending deletion, or replication/KMS state that requires broader action.
• Confirm the operator input will contain exactly the replacement token as the plain SecretString. No issue comment or command transcript may include its length, prefix, suffix, digest, or value.
• Disable shell tracing/history capture as appropriate and prepare a private least-exposure input path. Any temporary material is permission-restricted and has a named cleanup step.

Preflight authorizes only an in-place PutSecretValue against the resolved existing secret. Any need for IAM, secret policy, KMS, replication, tag, stack, Lambda, or GitHub token changes stops and is separately groomed.

Phase 2 — in-place value update [HUMAN]

• Use the credentialed operator session to create one new version on the existing secret with the replacement plain SecretString; do not expose the token on the command line or in output.
• Require a successful service response. Record only timestamp, account/region confirmation, existing-secret identity in its already-public/sanitized form, and success/failure. Do not paste an ARN containing account details, request payload, secret value, or full command transcript.
• Verify through metadata only that the new version owns AWSCURRENT. Do not retrieve or compare secret contents and do not publish the version identifier if local policy treats it as private.
• Destroy any operator-owned temporary input immediately after the service accepts it and confirm it was not added to shell history, repository state, logs, or artifacts.

If the call fails, make no second resource and no policy/IAM/template change. Record only the sanitized error class and return to the credential/target owner.

Phase 3 — bounded runtime read proof

• Wait for the already-running validation dispatches to finish. Both validation workflows must pass; this proves the separately rotated Actions secret/private checkout only.
• Through the normal authenticated production portal, perform a read-only Process Docs listing/read request that must reach the private knowledge repository. Discard the response body and record only route class, UTC timestamp, HTTP status, and pass/fail; do not record titles, paths, counts, private links, text, headers, cookies, or screenshots.
• A successful 2xx private-knowledge response after the value update is the runtime proof. Confirm no credential-shaped value appears in application/portal output used for the check.
• Because warm execution environments cache the old value, a first authentication failure may be observed after rotation. Retry only within a predeclared bounded observation window for a naturally fresh environment. Do not deploy, mutate configuration, increase traffic, invoke writes, or use AWS control-plane changes to manufacture a cold start.
• If no successful read occurs within that window, leave the issue open and report a sanitized failure/status class. Reconfirm the replacement token's repository selection/permissions and in-place version metadata privately; do not rotate again, repoint to the revoked prior value, or broaden runtime behavior without new grooming.

Acceptance criteria

• The GitHub Actions secret was updated without exposing the replacement value.
• The already-running validation dispatches finish successfully with public-safe evidence; no extra dispatch is required.
• A HUMAN operator updates only the existing runtime secret's plain SecretString using short-lived credentialed access and a no-echo input path.
• Metadata confirms the replacement version is AWSCURRENT; no secret value was retrieved, printed, logged, committed, or copied to GitHub.
• No new secret, static key, IAM/policy/KMS/tag/replication/template change, runtime endpoint, compatibility path, migration, application deployment, or Lambda configuration change occurs.
• A normal authenticated production portal read returns 2xx from the private knowledge source after the update, with evidence limited to route class, timestamp, status, and pass/fail.
• Any temporary secret material is destroyed and repository status remains unaffected.
• On-Call posts sanitized PASS and PM accepts; only then may the HUMAN owner close the issue.

Rollback and failure handling

PutSecretValue retains version history, but the previous token was rotated/revoked and must not be treated as an automatic rollback target. On a bad update or failed read:

1. stop without redeploying or creating another secret;
2. preserve sanitized failure class and version-stage metadata only;
3. privately verify the intended replacement value and its GitHub repository access;
4. if the supplied replacement was wrong, a HUMAN Security/operator owner may place the correct replacement as a new version on the same existing secret through the same controls;
5. never restore a known-revoked value merely to make the portal appear healthy.

Any need to change secret policy, IAM, KMS, replication, resource identity, application caching, or GitHub token permissions is a separately authorized issue.

Lifecycle

• PM grooming
• GitHub Actions secret rotation
• Existing validation dispatches complete
• [HUMAN] Existing Secrets Manager value update
• On-Call sanitized portal read proof
• PM acceptance
• [HUMAN] close issue

No commit is expected. If implementation unexpectedly requires a repository change, stop this operator issue and run that change through PM → SWE → Tester → PM → explicit-path commit → push → On-Call before resuming.

Out of scope

Reading or publishing .env/token content; token regeneration; changing the Actions secret again; new workflow dispatches; a new/renamed secret; secret policy, KMS, replication, or tag changes; IAM work; static AWS keys; runtime/admin rotation endpoints; compatibility secrets; migrations; repository code/docs/tests; manual application deploy; Lambda configuration/version changes; write-path/editor testing; AWS/data imports, exports, restores, or production mutations beyond the one authorized secret value update.

[alexeygrigorev]

PM GROOMING PASS — narrow HUMAN runtime-secret update

The Actions side is already complete: DATAOPS_CONTENT_GITHUB_TOKEN was rotated, and the existing validation dispatches are running. Their eventual result proves only CI's private checkout; it cannot prove the application's separate Secrets Manager value.

The remaining scope is now exact:

1. a named HUMAN operator enters through the normal short-lived AWS credential gate;
2. metadata-only preflight resolves the one existing deployed runtime secret;
3. the operator places the replacement token as one new plain SecretString version without command-line/history/log exposure;
4. metadata—not GetSecretValue—confirms the new version is AWSCURRENT;
5. On-Call performs a normal authenticated portal read and records only route class, UTC timestamp, HTTP status, and PASS/FAIL.

The runbook accounts for the runtime's per-execution-environment token cache: bounded waiting for a naturally fresh environment is allowed, but deploys, Lambda/config mutations, traffic forcing, and write requests are not. The revoked previous value is not an automatic rollback target.

No code or commit is expected. New/renamed secrets, static AWS keys, IAM/KMS/policy/tag changes, runtime rotation endpoints, compatibility paths, migration machinery, manual app deploys, token regeneration, editor writes, and any token/content logging are explicit stop conditions.

needs grooming is removed; human remains because the only state-changing step requires the credentialed operator. I did not inspect .env or any token, call AWS, read private content, edit repository files, run tests/workflows, deploy, or mutate data.

[alexeygrigorev]

Sanitized CI evidence after the Actions-secret rotation:

• Validate DataOps Content: run 31737002485 — success on public main at 7940763; private knowledge checkout and all validation steps completed.
• Validate Planning Docs: run 31737004774 — success on public main at 7940763; private knowledge checkout and all validation steps completed.

No token value was logged or retrieved. This proves only DATAOPS_CONTENT_GITHUB_TOKEN; the separate HUMAN runtime Secrets Manager rotation remains pending exactly as groomed.