Docker Bake: Add `transposelist()`

[polarathene]

Description

I would like the ability to convert a list from rows into columns, AFAIK the typical term for this functionality is transpose.

It can currently be implemented via a user-defined function but may be worthwhile functionality for Docker Bake to include in it's stdlib functions:

function transposelist {
  params = [nested_list]
  result = [
    for i in range(0, length(nested_list[0])):
    nested_list[*][i]
  ]
}

FWIW, Terraform has similar functionality for maps of lists as a transpose collection function.

---

Reference

For context, I'm aware of some related functionality in HCL and Docker Bake.

Related functionality:

• This functionality is similar to the matrix field in target blocks, but could be leveraged elsewhere such as via user-defined functions or global scope attributes.
• HCL also has the splat syntax such that you can get a new list by a sublist index across all elements, or from a common key of each object element:

  # Splat with indices:
  example = [["a", "1"], ["a", "2"], ["a", "3"]]
  formatlist("Left: %s, Right: %s", example[*][0], example[*][1])
  # This is equivalent to `example...`, it does not spread a splat of all indices:
  formatlist("Left: %s, Right: %s", example[*]...)
  
  map_example = [{a: "1"}, {"a": "2"}, {"a": "3"}]
  formatlist("Value of key `a`: %s", map_example[*].a)

AFAIK the splat syntax lacks a way to do full transposition of the list, so you would need to repeat it for each element unless it could take a range.

There is also the ... spread operator that can only be used as the last parameter to a function, expanding a list of values into additional args.

Without a key or index applied after a splat, the list will remain unchanged, making the use of ... unhelpful towards transposition as example[*]... / example[*][...] / example[*...] are not valid forms to express transpose all element indices of a list.

Instead you either splat each index explicitly (as shown above), or via a for expression:

# Produces: [["a", "1"], ["a", "2"], ["a", "3"], ["b", "1"], ["b", "2"], ["b", "3"]]
matrix = setproduct(["a", "b"], ["1", "2", "3"])

# Currently to transpose a list you would need to know the fixed range
# to iterate and splat each column into a new collection:
# Transpose to: [["a", "a", "a", "b", "b", "b"], ["1", "2", "3", "1", "2", "3"]]
transposed = [for i,v in matrix[0]: matrix[*][i]]

# Proposed:
# This enables a list produce like `setproduct()` as the input without
# reliance on a reference like the the `for` expression needs.
transposed = transposelist(
  setproduct(["a", "b"], ["1", "2", "3"])
)

This approach is compatible with ... to expand list elements into args:

# Current workaround (assuming a reference is available for the input):
formatlist("Left: %s, Right: %s", [for i,v in matrix[0]: matrix[*][i]]...)

# Proposed is capable of inlining the value instead of depending upon a reference:
formatlist("Left: %s, Right: %s", transposelist(matrix)...)

[polarathene]

For a real-world example as context, I'll be comparing two approachs (A, B) to transposelist().

NOTE: The details of the variables and methods used in the below examples isn't too important, but if it helps I've provided them here:

Helper functions

function with_image_names {
  params = [image_name]
  variadic_params = registries
  # When no registries are provided, fallback to without a registry prefix:
  # (`variadic_params` will be `[]` when no value is passed in)
  result = coalescelist(
    formatlist("%s/${image_name}", registries),
    [ image_name ]
  )
}

function with_tags {
  params = [is_primary_release, tags] # <bool, list(string)>
  variadic_params = suffixes # <list(string)>
  # Flatten nested lists and compact to remove any empty / null elements:
  result = compact(flatten([
    [for suffix in suffixes: [
      # `latest` represents the bare version tags without suffix applied
      suffix == "latest" ? tags : formatlist("%s-${suffix}", tags),
      # Skip adding the bare suffix tag for other releases:
      is_primary_release ? suffix : null
    ]]
  ]))
}

function with_semver {
  params = [VERSION]
  result = {
    raw: VERSION
    major: regex("([0-9]+).*$", VERSION)[0]
    minor: regex("([0-9]+\\.[0-9]+).*$", VERSION)[0]
    primary: VERSION == PRIMARY_VERSION
  }
}

function tags_from_semver {
  params = [version]
  # [2.10.2, 2.10, 2?]
  result = compact([
    version.raw,
    version.minor,
    version.primary ? version.major : null
  ])
}

With some variable and target for usage:

group "default" {
  targets = ["linux"]
}

group "linux" {
  targets = ["linux-runtime", "linux-builder"]
}

variable "IMAGE_NAME" {
  default = "hello/world"
}

variable "REGISTRIES" {
  default = ["docker.io", "ghcr.io"]
}

variable "PRIMARY_VERSION" {
  default = "2.10.2"
}

variable "VERSIONS" {
  default = ["2.10.2", "2.11.0-beta.1"]
}

TAGS = [ for v in VERSIONS: with_semver(v) ]

target "linux-runtime" {
  matrix = {
    version = TAGS
  }

  name = "${ sanitize(version.minor) }-alpine"
  context = "${ version.minor }/alpine"
  tags = create_tags(IMAGE_NAME, version, "latest", "alpine")
}

target "linux-builder" {
  matrix = {
    version = TAGS
  }

  name = "${ sanitize(version.minor) }-builder-alpine"
  context = "${ version.minor }/builder"
  tags = create_tags(IMAGE_NAME, version, "builder", "builder-alpine")
}

Approach A: flatten() + for expression

function create_tags {
  params = [image_name, version]  # <string, object>
  variadic_params = suffixes
  result = format_image_tags({
    names: with_image_names(image_name, REGISTRIES...),
    tags: with_tags(version.primary, tags_from_semver(version), suffixes...),
  })
}

# HCL lacks convenient temporary storage for scoped variables,
# this separate function helps with readability.
function format_image_tags {
  params = [image_data]
  result = flatten([for name in image_data.names:
    formatlist("${name}:%s", image_data.tags)
  ])
}

As a single function

Using formatlist() within for expressions effectively requires flatten() still, even when condensing format_image_tags() into create_tags():

function create_tags {
  params = [image_name, version]
  variadic_params = suffixes
  result = flatten([
    # Workaround for lack of variables, use a `for` expression's context
    # with a single element:
    for image in [{
      names: with_image_names(image_name, REGISTRIES...),
      tags: with_tags(version.primary, tags_from_semver(version), suffixes...),
    }]:

    # Nested for loop (non-issue due to being contained within a `flatten()` call:
    [for name in image.names: formatlist("${name}:%s", image.tags)]
  ])
}

Approach B: Splat

function create_tags {
  params = [image_name, version]
  variadic_params = suffixes
  result = format_image_tags(setproduct(
    with_image_names(image_name, REGISTRIES...),
    with_tags(version.primary, tags_from_semver(version), suffixes...)
  ))
}

# Helper method to split the param value into two lists by their inner indices using splat syntax:
function format_image_tags {
  params = [tag_list]
  result = formatlist("%s:%s",
    tag_list[*][0], # `ghcr.io/hello/world` or `hello/world`
    tag_list[*][1]  # `:2.10-alpine`
  )
}

As a single function

Likewise we could fold the two functions into one, but it's not really an improvement:

function create_tags {
  params = [image_name, version]
  variadic_params = suffixes
  result = formatlist("%s:%s", [
    for image in [setproduct(
      with_image_names(image_name, REGISTRIES...),
      with_tags(version.primary, tags_from_semver(version), suffixes...),
    )]:

    # Splat the indices to transpose:
    [image[*][0], image[*][1]]
  ][0]...)
}

Instead of flatten(), the transposed result is spread via ... (and since list comprehension via the for expression was needed as a workaround for the image variable reference, the result was wrapped in an extra list layer, hence the [0] index).

Approach C: With transposelist()

# Only change here is `setproduct()` was moved into the helper function
function create_tags {
  params = [image_name, version]
  variadic_params = suffixes
  result = format_image_tags(
    with_image_names(image_name, REGISTRIES...),
    with_tags(version.primary, tags_from_semver(version), suffixes...)
  )
}

function format_image_tags {
  params = [names, tags]
  result = formatlist("%s:%s", transposelist(setproduct(names, tags))...)
}

If this were global scope instead, we would have the benefit of assignment via global attributes:

# Sadly no way to accomplish similar to this within user-defined functions (or elsewhere):
names = with_image_names(image_name, REGISTRIES...)
tags = with_tags(version.primary, tags_from_semver(version), suffixes...)

result = formatlist("%s:%s", transposelist(setproduct(names, tags))...)

---

Rubber Ducky

In writing up this comment I realized without using formatlist(), that flatten() could be avoided via setproduct() + basic string interpolation with indices (similar to the splat approach):

function format_image_tags {
  params = [names, tags]
  result = [for image in setproduct(names, tags):
    "${image[0]}:${image[1]}"
  ]
}

If we then leverage the for expression workaround for variable binding I showed in the "single function" variants, this would result in a fairly clean and easy to grok version of the create_tags() function:

function create_tags {
  params = [image_name, version]
  variadic_params = suffixes
  result = [
    for image in setproduct(
      with_image_names(image_name, REGISTRIES...),
      with_tags(version.primary, tags_from_semver(version), suffixes...),
    ):

    "${image[0]}:${image[1]}"
  ]
}

Still I hope that the transposelist() example illustrates some benefit 😅