Skip to content

Pull: digest-based references produce malformed localRef (missing registry host and wrong separator) #1082

@jeanlaurent

Description

@jeanlaurent

Summary

  • Context: The Pull function in pkg/remote/pull.go downloads agent artifacts from OCI registries and stores them locally using a content store.
  • Bug: When pulling an artifact using a digest-based reference (e.g., myregistry.io/org/app@sha256:abc...), the code constructs an invalid local reference format.
  • Actual vs. expected: The code produces org/app:sha256:abc... (missing registry host, wrong separator) instead of preserving the original reference myregistry.io/org/app@sha256:abc....
  • Impact: Pulling any artifact by digest fails with a parsing error from crane.Save(), making digest-based references completely unusable.

Code with bug

localRef := ref.Context().RepositoryStr() + ":" + ref.Identifier()  // <-- BUG 🔴 Creates malformed reference for digest-based refs

The bug is on line 32 of pkg/remote/pull.go. The code has two problems:

  1. ref.Context().RepositoryStr() returns only the repository path (e.g., org/app) without the registry host (e.g., myregistry.io)
  2. The hardcoded : separator is incorrect for digest-based references, which should use @

Evidence

Example

Consider pulling an artifact by digest:

Input reference: myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef

Step-by-step execution:

  1. Line 17: name.ParseReference() successfully parses the digest-based reference
  2. Line 32: The code constructs localRef:
    • ref.Context().RepositoryStr() returns "org/app" (missing myregistry.io)
    • ref.Identifier() returns "sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"
    • Result: localRef = "org/app:sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"
  3. Line 57: Calls store.StoreArtifact(img, localRef) with the malformed reference
  4. In pkg/content/store.go line 78: crane.Save(img, reference, tarPath) attempts to parse the reference
  5. Failure: crane.Save() fails because "org/app:sha256:..." is not a valid OCI reference format (digest-based references must use @ separator, not :)

Expected behavior: The code should use ref.String() which returns the complete, correctly-formatted reference: "myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"

Failing test

Test script

package remote

import (
	"testing"

	"github.com/google/go-containerregistry/pkg/name"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
)

// TestLocalRefConstruction tests that localRef is constructed correctly for both tag and digest references
func TestLocalRefConstruction(t *testing.T) {
	tests := []struct {
		name             string
		registryRef      string
		expectedLocalRef string
	}{
		{
			name:             "tag-based reference",
			registryRef:      "myregistry.io/org/app:latest",
			expectedLocalRef: "myregistry.io/org/app:latest",
		},
		{
			name:             "digest-based reference",
			registryRef:      "myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
			expectedLocalRef: "myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
		},
		{
			name:             "tag with registry port",
			registryRef:      "myregistry.io:5000/org/app:v1.0.0",
			expectedLocalRef: "myregistry.io:5000/org/app:v1.0.0",
		},
		{
			name:             "digest with registry port",
			registryRef:      "myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234",
			expectedLocalRef: "myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234",
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			ref, err := name.ParseReference(tt.registryRef)
			require.NoError(t, err)

			// This is the buggy code from pull.go line 32
			localRefBuggy := ref.Context().RepositoryStr() + ":" + ref.Identifier()

			// Show what the code currently produces
			t.Logf("Input:         %s", tt.registryRef)
			t.Logf("Expected:      %s", tt.expectedLocalRef)
			t.Logf("Current code:  %s", localRefBuggy)
			t.Logf("ref.String():  %s", ref.String())

			// The bug: using RepositoryStr() loses the registry host
			// and using ":" separator is wrong for digest-based references
			assert.Equal(t, tt.expectedLocalRef, localRefBuggy,
				"localRef should match the original reference format")
		})
	}
}

Test output

=== RUN   TestLocalRefConstruction
=== RUN   TestLocalRefConstruction/tag-based_reference
    pull_digest_test.go:49: Input:         myregistry.io/org/app:latest
    pull_digest_test.go:50: Expected:      myregistry.io/org/app:latest
    pull_digest_test.go:51: Current code:  org/app:latest
    pull_digest_test.go:52: ref.String():  myregistry.io/org/app:latest
    pull_digest_test.go:56:
        	Error Trace:	/home/user/cagent/pkg/remote/pull_digest_test.go:56
        	Error:      	Not equal:
        	            	expected: "myregistry.io/org/app:latest"
        	            	actual  : "org/app:latest"

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-myregistry.io/org/app:latest
        	            	+org/app:latest
        	Test:       	TestLocalRefConstruction/tag-based_reference
        	Messages:   	localRef should match the original reference format
=== RUN   TestLocalRefConstruction/digest-based_reference
    pull_digest_test.go:49: Input:         myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
    pull_digest_test.go:50: Expected:      myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
    pull_digest_test.go:51: Current code:  org/app:sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
    pull_digest_test.go:52: ref.String():  myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
    pull_digest_test.go:56:
        	Error Trace:	/home/user/cagent/pkg/remote/pull_digest_test.go:56
        	Error:      	Not equal:
        	            	expected: "myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"
        	            	actual  : "org/app:sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-myregistry.io/org/app@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
        	            	+org/app:sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
        	Test:       	TestLocalRefConstruction/digest-based_reference
        	Messages:   	localRef should match the original reference format
=== RUN   TestLocalRefConstruction/tag_with_registry_port
    pull_digest_test.go:49: Input:         myregistry.io:5000/org/app:v1.0.0
    pull_digest_test.go:50: Expected:      myregistry.io:5000/org/app:v1.0.0
    pull_digest_test.go:51: Current code:  org/app:v1.0.0
    pull_digest_test.go:52: ref.String():  myregistry.io:5000/org/app:v1.0.0
    pull_digest_test.go:56:
        	Error Trace:	/home/user/cagent/pkg/remote/pull_digest_test.go:56
        	Error:      	Not equal:
        	            	expected: "myregistry.io:5000/org/app:v1.0.0"
        	            	actual  : "org/app:v1.0.0"

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-myregistry.io:5000/org/app:v1.0.0
        	            	+org/app:v1.0.0
        	Test:       	TestLocalRefConstruction/tag_with_registry_port
        	Messages:   	localRef should match the original reference format
=== RUN   TestLocalRefConstruction/digest_with_registry_port
    pull_digest_test.go:49: Input:         myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234
    pull_digest_test.go:50: Expected:      myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234
    pull_digest_test.go:51: Current code:  org/app:sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234
    pull_digest_test.go:52: ref.String():  myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234
    pull_digest_test.go:56:
        	Error Trace:	/home/user/cagent/pkg/remote/pull_digest_test.go:56
        	Error:      	Not equal:
        	            	expected: "myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234"
        	            	actual  : "org/app:sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234"

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-myregistry.io:5000/org/app@sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234
        	            	+org/app:sha256:abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234
        	Test:       	TestLocalRefConstruction/digest_with_registry_port
        	Messages:   	localRef should match the original reference format
--- FAIL: TestLocalRefConstruction (0.00s)
    --- FAIL: TestLocalRefConstruction/tag-based_reference (0.00s)
    --- FAIL: TestLocalRefConstruction/digest-based_reference (0.00s)
    --- FAIL: TestLocalRefConstruction/tag_with_registry_port (0.00s)
    --- FAIL: TestLocalRefConstruction/digest_with_registry_port (0.00s)
FAIL

Full context

The Pull function is the primary mechanism for downloading agent artifacts from OCI registries. It's used in two main contexts:

  1. Command-line pulls: When users run cagent pull docker.io/user/agent@sha256:... to download an agent artifact locally
  2. Automatic pulls for OCI sources: When an agent configuration references an OCI artifact (via pkg/config/sources.go), the system automatically pulls it before loading

The function performs these steps:

  1. Parses the registry reference
  2. Resolves the remote digest
  3. Checks if a local copy exists (using the malformed localRef)
  4. Pulls the image from the registry
  5. Stores it locally with the malformed reference

The bug occurs at step 3 and 5, where the code constructs an incorrect local reference. This causes the storage operation to fail at pkg/content/store.go:78 when crane.Save() attempts to parse the malformed reference.

The content store uses reference strings as lookup keys by hashing them (pkg/content/store.go:255-256). When the reference is malformed during storage, subsequent attempts to retrieve the artifact using the original (correct) reference will fail because the hashes don't match.

External documentation

type Reference interface {
    Context() Repository
    Identifier() string
    Name() string
    Scope(string) string
    String() string
}

String returns the complete reference as a string, including registry, repository, and either tag or digest.
References can be by tag or digest:
- By tag: registry.example.com/repository/image:tag
- By digest: registry.example.com/repository/image@sha256:digest

The @ separator is required for digest-based references.
func Save(img v1.Image, src, path string) error

Save writes the v1.Image img as a tarball at path with tag src.
The src parameter must be a valid reference string.

Why has this bug gone undetected?

This bug has gone undetected because:

  1. No digest-based reference testing: All existing tests in pkg/remote/pull_test.go use tag-based references (e.g., test:latest, test-app:latest). No tests verify pulling by digest.

  2. Limited real-world usage: The feature appears to be relatively new (added in commit 64ade5e on July 10, 2025). Most users likely pull agents by tag rather than by digest.

  3. Bug affects all reference types, but manifests differently:

    • For digest-based references: Complete failure - crane.Save() rejects the malformed reference format org/app:sha256:... immediately
    • For tag-based references: Partial failure - The reference org/app:latest is technically valid, so crane.Save() succeeds, but the artifact is stored without the registry host. This causes subtle issues when trying to pull from registries with the same repository paths but different hosts (e.g., registry1.io/org/app:latest vs registry2.io/org/app:latest would collide)
  4. Error may be attributed to network issues: When the pull command fails, users might assume it's a network problem or authentication issue rather than a bug in reference construction.

Recommended fix

Replace line 32 with:

localRef := ref.String()  // <-- FIX 🟢 Preserves complete reference with correct format

The ref.String() method returns the complete, correctly-formatted reference string that preserves:

  • The registry host (e.g., myregistry.io)
  • The repository path (e.g., org/app)
  • The correct separator (: for tags, @ for digests)
  • The identifier (tag or digest)

This ensures that:

  1. crane.Save() can successfully parse the reference
  2. The artifact can be retrieved later using the same reference format
  3. Multiple registries with the same repository paths don't collide

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions