Update array value

[ghost]

Hello!

I am using shell and yq (version 4.17.2) to update a yml file. I have posted my question on stackoverflow but I am also posting it here in hopes of finding an answer.

https://stackoverflow.com/questions/70932362/use-yq-to-update-array-value

[ghost]

Below is my stackoverflow post:

I have two yaml files I am using. The first yaml file looks like this:

spring.yml

spring:
  cloud:
    gateway:
      routes:
      - id: someid
	    uri: someUri
	    predicates:
	    - Path=/somePath
	    filters:
	    - RewritePath=/someOtherPath

I have another file that just contains routes and looks like this:

routes.yml

routes:
  - id: someid
    uri: someOtherUri
    predicates:
    - Path=/somePath
    filters:
    - RewritePath=/someNewPath

My goal is to update the route in the first file with the value of the route in the second file. Note that the first file in reality will have many routes but for demonstration purposes I am only showing the first in this example. I have the following script which loops through to update the routes as necessary when the id's match:

#!/bin/sh
OVERRIDE_ROUTE_IDS=$(yq eval '.routes.[].id' routes.yml)
GENERATED_ROUTE_IDS=$(yq eval '.spring.cloud.gateway.routes.[].id' spring.yml)

SAVEIFS=$IFS   # Save current IFS (Internal Field Separator)
IFS=$'\n'      # Change IFS to newline char
OVERRIDE_ROUTE_IDS=($OVERRIDE_ROUTE_IDS) # split the `OVERRIDE_ROUTE_IDS` string into an array by the same name
GENERATED_ROUTE_IDS=($GENERATED_ROUTE_IDS) # split the `GENERATED_ROUTE_IDS` string into an array by the same name
IFS=$SAVEIFS   # Restore original IFS

for (( i=0; i<${#OVERRIDE_ROUTE_IDS[@]}; i++ ))
do
   if [[ "${GENERATED_ROUTE_IDS[*]}" =~ "${OVERRIDE_ROUTE_IDS[$i]}" ]]
   then
      echo "route ID ${OVERRIDE_ROUTE_IDS[$i]} exists in generated routes"
      for (( j=0; j<${#GENERATED_ROUTE_IDS[@]}; j++ ))
      do
         if [[ "${GENERATED_ROUTE_IDS[$j]}" == "${OVERRIDE_ROUTE_IDS[$i]}" ]]
         then
            echo "index of route ${GENERATED_ROUTE_IDS[$j]} is $j"
            echo "$i"
            ROUTE_TO_USE=$(yq eval ".routes.[$i]" routes.yml)
            $(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" spring.yml)
         fi
      done
   else
      echo "no match so add to top of routes"
   fi
done

My assumption is this command should update spring.yml file with the new route in place of the one that was identified with the same id:

$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" application.yml)

But I am getting the following error

Error: Parsing expression: Lexer error: could not match text starting at 1:37 failing at 1:39 unmatched text: "id"

I'm stumped on this and not sure what I'm doing wrong at this point. For reference I am using yq version 4.17.2.

[mikefarah]

Oh yep - I can see in your description you mentioned matching id 👍🏼 - sorry I missed it initially

[ghost]

Awesome, thank you for that! Also another question. For my use case if the id doesn't match, I need to add the value of the route from routes.yaml to the top of the array in spring.yaml. Is that possible?

[mikefarah]

Tricky, but I think I got it:

yq ea '
  (select(fi==0) | .spring.cloud.gateway.routes.[].id) as $currentIds |
  (select(fi==1) | [.routes.[] | select( [$currentIds != .id] | all )] ) as $newRoutes |
  ( $newRoutes + .spring.cloud.gateway.routes + .routes) as $routesToMerge |

  (
    (($routesToMerge | .[] | {.id: .}) as $item ireduce ({}; . * $item )) as $uniqueMap
    | ( $uniqueMap  | to_entries | .[]) as $item ireduce([]; . + $item.value)
  ) as $mergedArray
  | select(fi == 0) | .spring.cloud.gateway.routes = $mergedArray  

' spring.yaml routes.yaml

yq will maintain the order of the array when merging, so the key to solving this was to first create the array to merge in the desired order.
I filtered out the second list to only include entries that are not in the original, then create a new array with those entries at the start.

[ghost]

This works perfect! Thank you!

[mikefarah]

No prob - could you please update your stackoverflow too so that if others stumble onto it they have the answer too :)