Cosign verify-blob-attestation flaky

[crazy-max]

Encounter this error one time when invoking cosign verify-blob-attestation: https://github.com/docker/github-builder-experimental/actverify-blob-attestationons/runs/19393939019/job/55491360947

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Error: invalid character ':' after top-level value
error during command execution: invalid character ':' after top-level value

Re-running the workflow looks good: https://github.com/docker/github-builder-experimental/actions/runs/19393939019/job/55491405358#step:6:17

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Verified OK
/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_arm64/provenance.sigstore.json linux_arm64/hello.txt
Verified OK

@haydentherapper Is there a flag for cosign to print stacktraces if command fails so I can create a good bug report on cosign repo if this error happens again? Maybe -d, --verbose?

[crazy-max]

Happened again after multiple runs: https://github.com/docker/github-builder-experimental/actions/runs/19404268711/job/55516496233#step:6:17

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Error: invalid character '"' after top-level value
error during command execution: invalid character '"' after top-level value

[Hayden-IO]

We'll need to add some more logging to chase down exactly where this occurs. Would you be able to provide the JSON provenance?

[Hayden-IO]

Looking through the build logs, I think I found the correct provenance, from https://github.com/docker/github-builder-experimental/actions/runs/19404268711/job/55520341689. Verification is working for me locally. These types of unexpected errors usually are from GitHub Action flakes, though I wouldn't know why in this case.

[crazy-max]

These types of unexpected errors usually are from GitHub Action flakes, though I wouldn't know why in this case.

Yes looks like it but I got another error during verification: https://github.com/docker/github-builder-experimental/actions/runs/19764434110/job/56633826692#step:6:18

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/bake.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Error: failed to verify log inclusion: transparency log signature does not match

Re-running the job with the exact same bundle payload succeeds: https://github.com/docker/github-builder-experimental/actions/runs/19764434110/job/56634562108#step:6:18

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/bake.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Verified OK

In this case I think this might be something flaky on rekor?

Artifact: bake-output-linux-amd64.zip

[crazy-max]

Got a new one: https://github.com/docker/github-builder-experimental/actions/runs/19823401695/job/56790894053#step:6:19

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Error: failed to verify signature: could not verify envelope: accepted signatures do not match threshold, Found: 0, Expected 1

Second re-run new error with the exact same bundle payload: https://github.com/docker/github-builder-experimental/actions/runs/19823401695/job/56791124123#step:6:18

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Error: invalid character '2' after top-level value

Good after third re-run with the exact same bundle payload: https://github.com/docker/github-builder-experimental/actions/runs/19823401695/job/56791296447#step:6:17

/home/runner/.cosign/cosign verify-blob-attestation --new-bundle-format --certificate-oidc-issuer https://token.actions.githubusercontent.com/ --certificate-identity-regexp ^https://github.com/docker/github-builder-experimental/.github/workflows/build.yml.*$ --bundle linux_amd64/provenance.sigstore.json linux_amd64/hello.txt
Verified OK

Artifact: build-output-linux-amd64.zip

[Hayden-IO]

Sorry for the delayed response. I think this is something with GitHub Actions artifacts. Verification is an offline process, so flakiness in any Sigstore component (fulcio or rekor) would be ruled out. There is a call to Google Cloud Storage to retrieve a TUF repo, but you'd get a different error if anything in this repo was corrupted. I'm concerned there's some corruption happening with the downloaded artifacts. What's weird to me is that the errors differ each time.

If you can log the bundle whenever there's an error, I could try to debug this locally. I've been unable to reproduce this locally, repeatedly redownloading and verifying the artifact from https://github.com/docker/github-builder-experimental/actions/runs/19823401695/job/56791121790#step:14:27.

[crazy-max]

@haydentherapper I don't think this is an issue with GitHub Actions artifacts on download, as the checksum validation matches in both the failing and successful cases:

• https://github.com/docker/github-builder-experimental/actions/runs/19823401695/job/56791124123#step:5:22
• https://github.com/docker/github-builder-experimental/actions/runs/19823401695/job/56791296447#step:5:22

That being said I will look at running the verify command multiple times in the test workflow to see if we can repro.

[crazy-max]

I think this is something with GitHub Actions artifacts.

Hum might be similar to

• [bug] Matching names and merge-multiple causes data corruption actions/download-artifact#298
• [bug] merge-multiple may silently result in a corrupt file when multiple of the same name exist actions/download-artifact#395

[crazy-max]

Doesn't seem to repro anymore with multiple runs in #62 after switching from merging with download-artifact in verify workflow to instead merging using upload-artifact/merge in finalize job.

[Hayden-IO]

Good find on the fix!