Primary instance won't restart when switching during migration from in-tree to plugin

[staypekos]

Followed the migration guide, but after Cluster edit in Step 2 it got stucked on switchover on (former) primary instance.

Primary instance:    test-pekos-repo-db-cnpg-cluster-1 (switching to test-pekos-repo-db-cnpg-cluster-2)
Primary start time:  2026-05-11 11:09:04 +0000 UTC (uptime 2h9m8s)
Status:              Waiting for the instances to become active Some instances are not yet active. Please wait.

but all instances seems active - pods 3 and 2 were restarted.

Instances status
Name                               Current LSN  Replication role  Status  QoS        Manager Version 
----                               -----------  ----------------  ------  ---        ---------------
test-pekos-repo-db-cnpg-cluster-1  0/19000000   Primary           OK      Burstable  1.28.1
test-pekos-repo-db-cnpg-cluster-2  0/19000000   Standby (async)   OK      Burstable  1.28.1
test-pekos-repo-db-cnpg-cluster-3  0/19000000   Standby (async)   OK      Burstable  1.28.1

Plugins status
Name                            Version  Status  Reported Operator Capabilities
----                            -------  ------  ------------------------------
barman-cloud.cloudnative-pg.io  0.12.0   N/A     Reconciler Hooks, Lifecycle Service, TYPE_INSTANCE_SIDECAR_INJECTION

(former) primary logs:

{"level":"info","ts":"2026-05-11T12:26:13.782387507Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:26:13.782 UTC","process_id":"25","session_id":"6a01b8d0.19","session_line_num":"35","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"checkpoint starting: time","backend_type":"checkpointer","query_id":"0"}}
{"level":"error","ts":"2026-05-11T12:26:17.482168146Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"wal archive plugin is not available: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/proc.go:285"}
{"level":"info","ts":"2026-05-11T12:26:17.508476589Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:26:17.508 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"1","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000017","backend_type":"archiver","query_id":"0"}}
{"level":"error","ts":"2026-05-11T12:26:18.538421588Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"wal archive plugin is not available: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/proc.go:285"}
{"level":"info","ts":"2026-05-11T12:26:18.540914907Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:26:18.540 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"2","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000017","backend_type":"archiver","query_id":"0"}}
{"level":"error","ts":"2026-05-11T12:26:19.568897143Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"wal archive plugin is not available: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/proc.go:285"}
{"level":"info","ts":"2026-05-11T12:26:19.571999382Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:26:19.571 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"3","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000017","backend_type":"archiver","query_id":"0"}}
{"level":"info","ts":"2026-05-11T12:26:19.57213517Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:26:19.571 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"4","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"WARNING","sql_state_code":"01000","message":"archiving write-ahead log file \"000000010000000000000017\" failed too many times, will try again later","backend_type":"archiver","query_id":"0"}}
{"level":"info","ts":"2026-05-11T12:26:30.090082849Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:26:30.089 UTC","process_id":"25","session_id":"6a01b8d0.19","session_line_num":"36","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"checkpoint complete: wrote 152 buffers (0.9%); 0 WAL file(s) added, 0 removed, 0 recycled; write=15.478 s, sync=0.174 s, total=16.308 s; sync files=22, longest=0.036 s, average=0.008 s; distance=13189 kB, estimate=26267 kB; lsn=0/18000060, redo lsn=0/171142C8","backend_type":"checkpointer","query_id":"0"}}
{"level":"error","ts":"2026-05-11T12:26:42.941273162Z","msg":"Error loading plugins, retrying","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"c206ffef-527c-44aa-833c-be90cca3b0c9","instance":"test-pekos-repo-db-cnpg-cluster-1","cluster":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).Reconcile\n\tinternal/management/controller/instance_controller.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2026-05-11T12:26:42.941440628Z","msg":"Reconciler error","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"c206ffef-527c-44aa-833c-be90cca3b0c9","error":"while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
...
{"level":"error","ts":"2026-05-11T12:27:19.658517143Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"wal archive plugin is not available: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/proc.go:285"}
{"level":"info","ts":"2026-05-11T12:27:19.660942788Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:27:19.660 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"5","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000017","backend_type":"archiver","query_id":"0"}}
{"level":"error","ts":"2026-05-11T12:27:20.688147322Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"wal archive plugin is not available: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/proc.go:285"}
{"level":"info","ts":"2026-05-11T12:27:20.690595677Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:27:20.690 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"6","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000017","backend_type":"archiver","query_id":"0"}}
{"level":"error","ts":"2026-05-11T12:27:20.89215728Z","msg":"Error loading plugins, retrying","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"b7e7fef3-dbe2-4c4d-b76b-85ab312fd808","instance":"test-pekos-repo-db-cnpg-cluster-1","cluster":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).Reconcile\n\tinternal/management/controller/instance_controller.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2026-05-11T12:27:20.892262168Z","msg":"Reconciler error","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"b7e7fef3-dbe2-4c4d-b76b-85ab312fd808","error":"while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"}
{"level":"error","ts":"2026-05-11T12:27:21.733032628Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","error":"wal archive plugin is not available: barman-cloud.cloudnative-pg.io","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.2/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.7/x64/src/runtime/proc.go:285"}
{"level":"info","ts":"2026-05-11T12:27:21.73703792Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:27:21.736 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"7","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"archive command failed with exit code 1","detail":"The failed archive command was: /controller/manager wal-archive --log-destination /controller/log/postgres.json pg_wal/000000010000000000000017","backend_type":"archiver","query_id":"0"}}
{"level":"info","ts":"2026-05-11T12:27:21.737072687Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-1","record":{"log_time":"2026-05-11 12:27:21.736 UTC","process_id":"30","session_id":"6a01b8d0.1e","session_line_num":"8","session_start_time":"2026-05-11 11:09:04 UTC","transaction_id":"0","error_severity":"WARNING","sql_state_code":"01000","message":"archiving write-ahead log file \"000000010000000000000017\" failed too many times, will try again later","backend_type":"archiver","query_id":"0"}}

target-primary logs:

{"level":"info","ts":"2026-05-11T12:27:50.82314402Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","record":{"log_time":"2026-05-11 12:27:50.822 UTC","process_id":"22","session_id":"6a01cb44.16","session_line_num":"6","session_start_time":"2026-05-11 12:27:48 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system is ready to accept read-only connections","backend_type":"postmaster","query_id":"0"}}
{"level":"info","ts":"2026-05-11T12:27:50.870702483Z","logger":"postgres","msg":"record","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","record":{"log_time":"2026-05-11 12:27:50.870 UTC","process_id":"65","session_id":"6a01cb46.41","session_line_num":"1","session_start_time":"2026-05-11 12:27:50 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"started streaming WAL from primary at 0/18000000 on timeline 1","backend_type":"walreceiver","query_id":"0"}}
{"level":"info","ts":"2026-05-11T12:27:57.655368909Z","msg":"Setting myself as primary","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"444c7060-a9ae-4558-bcb6-aa1176dd6cbf","phase":"Waiting for the instances to become active","currentTimestamp":"2026-05-11T12:27:57.655330Z","targetPrimaryTimestamp":"2026-05-11T12:27:57.601661Z","currentPrimaryTimestamp":"2026-05-11T11:09:04.493598Z","msPassedSinceTargetPrimaryTimestamp":53,"msPassedSinceCurrentPrimaryTimestamp":4733161,"msDifferenceBetweenCurrentAndTargetPrimary":-4733108,"logging_pod":"test-pekos-repo-db-cnpg-cluster-2"}
{"level":"info","ts":"2026-05-11T12:27:57.655447141Z","msg":"I'm the target primary, wait for the wal_receiver to be terminated","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"444c7060-a9ae-4558-bcb6-aa1176dd6cbf","logging_pod":"test-pekos-repo-db-cnpg-cluster-2"}
{"level":"info","ts":"2026-05-11T12:27:57.658824485Z","msg":"WAL receiver is still active, waiting","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"444c7060-a9ae-4558-bcb6-aa1176dd6cbf","logging_pod":"test-pekos-repo-db-cnpg-cluster-2"}
{"level":"info","ts":"2026-05-11T12:27:58.663742452Z","msg":"WAL receiver is still active, waiting","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"444c7060-a9ae-4558-bcb6-aa1176dd6cbf","logging_pod":"test-pekos-repo-db-cnpg-cluster-2"}
{"level":"info","ts":"2026-05-11T12:27:59.667687436Z","msg":"WAL receiver is still active, waiting","logger":"instance-manager","logging_pod":"test-pekos-repo-db-cnpg-cluster-2","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"444c7060-a9ae-4558-bcb6-aa1176dd6cbf","logging_pod":"test-pekos-repo-db-cnpg-cluster-2"}

cnpg-cloudnative-pg manager log:

{"level":"info","ts":"2026-05-11T13:21:34.606389866Z","msg":"There is a switchover or a failover in progress, waiting for the operation to complete","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"},"namespace":"test-pekos-repo","name":"test-pekos-repo-db-cnpg-cluster","reconcileID":"99d31232-ca6d-40ba-a8d1-be7f1f3dd78a","currentPrimary":"test-pekos-repo-db-cnpg-cluster-1","targetPrimary":"test-pekos-repo-db-cnpg-cluster-2"}

plugin-barman-cloud barman-cloud logs:

{"level":"debug","ts":"2026-05-11T12:27:57.504938797Z","msg":"parsing cluster definition","caller":"/workspace/internal/cnpgi/operator/reconciler.go:80"}
{"level":"debug","ts":"2026-05-11T12:27:57.505306715Z","msg":"parsing barman object configuration","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo","caller":"/workspace/internal/cnpgi/operator/reconciler.go:94"}
{"level":"info","ts":"2026-05-11T12:27:57.505384398Z","msg":"Pre hook reconciliation completed","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"}
{"level":"info","ts":"2026-05-11T12:27:57.594298454Z","logger":"lifecycle","msg":"Lifecycle hook reconciliation start"}
{"level":"info","ts":"2026-05-11T12:27:57.594867037Z","logger":"lifecycle","msg":"Reconciling pod"}
{"level":"debug","ts":"2026-05-11T12:27:57.595461091Z","logger":"plugin-barman-cloud-lifecycle","msg":"generated patch","podName":"test-pekos-repo-db-cnpg-cluster-3","caller":"/workspace/internal/cnpgi/operator/lifecycle.go:343","content":"[{\"op\":\"add\",\"path\":\"/spec/volumes/4\",\"value\":{\"name\":\"plugins\",\"emptyDir\":{}}},{\"op\":\"add\",\"path\":\"/spec/initContainers/1\",\"value\":{\"name\":\"plugin-barman-cloud\",\"image\":\"ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.12.0\",\"args\":[\"instance\"],\"env\":[{\"name\":\"PGDATA\",\"value\":\"/var/lib/postgresql/data/pgdata\"},{\"name\":\"POD_NAME\",\"value\":\"test-pekos-repo-db-cnpg-cluster-3\"},{\"name\":\"NAMESPACE\",\"value\":\"test-pekos-repo\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"test-pekos-repo-db-cnpg-cluster\"},{\"name\":\"PSQL_HISTORY\",\"value\":\"/controller/tmp/.psql_history\"},{\"name\":\"PGPORT\",\"value\":\"5432\"},{\"name\":\"PGHOST\",\"value\":\"/controller/run\"},{\"name\":\"TMPDIR\",\"value\":\"/controller/tmp\"},{\"name\":\"AWS_REQUEST_CHECKSUM_CALCULATION\",\"value\":\"when_required\"},{\"name\":\"AWS_RESPONSE_CHECKSUM_VALIDATION\",\"value\":\"when_required\"},{\"name\":\"AWS_NO_CHUNKED_ENCODING\",\"value\":\"True\"},{\"name\":\"SPOOL_DIRECTORY\",\"value\":\"/controller/wal-restore-spool\"},{\"name\":\"CUSTOM_CNPG_GROUP\",\"value\":\"postgresql.cnpg.io\"},{\"name\":\"CUSTOM_CNPG_VERSION\",\"value\":\"v1\"}],\"resources\":{},\"restartPolicy\":\"Always\",\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"pg-wal\",\"mountPath\":\"/var/lib/postgresql/wal\"},{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}],\"startupProbe\":{\"exec\":{\"command\":[\"/manager\",\"healthcheck\",\"unix\"]},\"timeoutSeconds\":10,\"failureThreshold\":10},\"imagePullPolicy\":\"IfNotPresent\",\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/volumeMounts/5\",\"value\":{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}}]"}
{"level":"info","ts":"2026-05-11T12:27:57.597264186Z","logger":"lifecycle","msg":"Lifecycle hook reconciliation start"}
{"level":"info","ts":"2026-05-11T12:27:57.597602598Z","logger":"lifecycle","msg":"Reconciling pod"}
{"level":"debug","ts":"2026-05-11T12:27:57.598095046Z","logger":"plugin-barman-cloud-lifecycle","msg":"generated patch","podName":"test-pekos-repo-db-cnpg-cluster-2","caller":"/workspace/internal/cnpgi/operator/lifecycle.go:343","content":"[{\"op\":\"add\",\"path\":\"/spec/volumes/4\",\"value\":{\"name\":\"plugins\",\"emptyDir\":{}}},{\"op\":\"add\",\"path\":\"/spec/initContainers/1\",\"value\":{\"name\":\"plugin-barman-cloud\",\"image\":\"ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.12.0\",\"args\":[\"instance\"],\"env\":[{\"name\":\"PGDATA\",\"value\":\"/var/lib/postgresql/data/pgdata\"},{\"name\":\"POD_NAME\",\"value\":\"test-pekos-repo-db-cnpg-cluster-2\"},{\"name\":\"NAMESPACE\",\"value\":\"test-pekos-repo\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"test-pekos-repo-db-cnpg-cluster\"},{\"name\":\"PSQL_HISTORY\",\"value\":\"/controller/tmp/.psql_history\"},{\"name\":\"PGPORT\",\"value\":\"5432\"},{\"name\":\"PGHOST\",\"value\":\"/controller/run\"},{\"name\":\"TMPDIR\",\"value\":\"/controller/tmp\"},{\"name\":\"AWS_REQUEST_CHECKSUM_CALCULATION\",\"value\":\"when_required\"},{\"name\":\"AWS_RESPONSE_CHECKSUM_VALIDATION\",\"value\":\"when_required\"},{\"name\":\"AWS_NO_CHUNKED_ENCODING\",\"value\":\"True\"},{\"name\":\"SPOOL_DIRECTORY\",\"value\":\"/controller/wal-restore-spool\"},{\"name\":\"CUSTOM_CNPG_GROUP\",\"value\":\"postgresql.cnpg.io\"},{\"name\":\"CUSTOM_CNPG_VERSION\",\"value\":\"v1\"}],\"resources\":{},\"restartPolicy\":\"Always\",\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"pg-wal\",\"mountPath\":\"/var/lib/postgresql/wal\"},{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}],\"startupProbe\":{\"exec\":{\"command\":[\"/manager\",\"healthcheck\",\"unix\"]},\"timeoutSeconds\":10,\"failureThreshold\":10},\"imagePullPolicy\":\"IfNotPresent\",\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/volumeMounts/5\",\"value\":{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}}]"}
{"level":"info","ts":"2026-05-11T12:27:57.599533408Z","logger":"lifecycle","msg":"Lifecycle hook reconciliation start"}
{"level":"info","ts":"2026-05-11T12:27:57.599940946Z","logger":"lifecycle","msg":"Reconciling pod"}
{"level":"debug","ts":"2026-05-11T12:27:57.600441479Z","logger":"plugin-barman-cloud-lifecycle","msg":"generated patch","podName":"test-pekos-repo-db-cnpg-cluster-1","caller":"/workspace/internal/cnpgi/operator/lifecycle.go:343","content":"[{\"op\":\"add\",\"path\":\"/spec/volumes/4\",\"value\":{\"name\":\"plugins\",\"emptyDir\":{}}},{\"op\":\"add\",\"path\":\"/spec/initContainers/1\",\"value\":{\"name\":\"plugin-barman-cloud\",\"image\":\"ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.12.0\",\"args\":[\"instance\"],\"env\":[{\"name\":\"PGDATA\",\"value\":\"/var/lib/postgresql/data/pgdata\"},{\"name\":\"POD_NAME\",\"value\":\"test-pekos-repo-db-cnpg-cluster-1\"},{\"name\":\"NAMESPACE\",\"value\":\"test-pekos-repo\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"test-pekos-repo-db-cnpg-cluster\"},{\"name\":\"PSQL_HISTORY\",\"value\":\"/controller/tmp/.psql_history\"},{\"name\":\"PGPORT\",\"value\":\"5432\"},{\"name\":\"PGHOST\",\"value\":\"/controller/run\"},{\"name\":\"TMPDIR\",\"value\":\"/controller/tmp\"},{\"name\":\"AWS_REQUEST_CHECKSUM_CALCULATION\",\"value\":\"when_required\"},{\"name\":\"AWS_RESPONSE_CHECKSUM_VALIDATION\",\"value\":\"when_required\"},{\"name\":\"AWS_NO_CHUNKED_ENCODING\",\"value\":\"True\"},{\"name\":\"SPOOL_DIRECTORY\",\"value\":\"/controller/wal-restore-spool\"},{\"name\":\"CUSTOM_CNPG_GROUP\",\"value\":\"postgresql.cnpg.io\"},{\"name\":\"CUSTOM_CNPG_VERSION\",\"value\":\"v1\"}],\"resources\":{},\"restartPolicy\":\"Always\",\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"pg-wal\",\"mountPath\":\"/var/lib/postgresql/wal\"},{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}],\"startupProbe\":{\"exec\":{\"command\":[\"/manager\",\"healthcheck\",\"unix\"]},\"timeoutSeconds\":10,\"failureThreshold\":10},\"imagePullPolicy\":\"IfNotPresent\",\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/volumeMounts/5\",\"value\":{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}}]"}
{"level":"info","ts":"2026-05-11T12:27:57.664269419Z","msg":"Pre hook reconciliation start"}
{"level":"debug","ts":"2026-05-11T12:27:57.664488549Z","msg":"parsing cluster definition","caller":"/workspace/internal/cnpgi/operator/reconciler.go:80"}
{"level":"debug","ts":"2026-05-11T12:27:57.664847632Z","msg":"parsing barman object configuration","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo","caller":"/workspace/internal/cnpgi/operator/reconciler.go:94"}
{"level":"info","ts":"2026-05-11T12:27:57.664925511Z","msg":"Pre hook reconciliation completed","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"}
{"level":"info","ts":"2026-05-11T12:27:57.763202152Z","msg":"Pre hook reconciliation start"}
{"level":"debug","ts":"2026-05-11T12:27:57.763442907Z","msg":"parsing cluster definition","caller":"/workspace/internal/cnpgi/operator/reconciler.go:80"}
{"level":"debug","ts":"2026-05-11T12:27:57.763862754Z","msg":"parsing barman object configuration","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo","caller":"/workspace/internal/cnpgi/operator/reconciler.go:94"}
{"level":"info","ts":"2026-05-11T12:27:57.763986498Z","msg":"Pre hook reconciliation completed","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"}
{"level":"info","ts":"2026-05-11T12:27:57.808688662Z","msg":"Pre hook reconciliation start"}
{"level":"debug","ts":"2026-05-11T12:27:57.809009008Z","msg":"parsing cluster definition","caller":"/workspace/internal/cnpgi/operator/reconciler.go:80"}
{"level":"debug","ts":"2026-05-11T12:27:57.80943313Z","msg":"parsing barman object configuration","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo","caller":"/workspace/internal/cnpgi/operator/reconciler.go:94"}
{"level":"info","ts":"2026-05-11T12:27:57.809541965Z","msg":"Pre hook reconciliation completed","name":"test-pekos-repo-db-cnpg-cluster","namespace":"test-pekos-repo"}
{"level":"info","ts":"2026-05-11T12:27:58.812223318Z","msg":"Pre hook reconciliation start"}
...

Environment
CloudNativePG operator: v1.28.1 (installed via helm)
CNPG-Cluster: 0.3.1 (installed via helm)
plugin-barman-cloud: v0.12.0 (tried installed via helm and yaml manifest)
PostgreSQL: 16 (ghcr.io/cloudnative-pg/postgresql:16)
Kubernetes: v1.32 (RKE v1.32.5+rke2r1)

Cluster configuration:

spec:
  affinity:
    podAntiAffinityType: preferred
    topologyKey: topology.kubernetes.io/zone
  bootstrap:
    initdb:
      database: test_pekos_repo
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: test_pekos_repo
  env:
  - name: AWS_REQUEST_CHECKSUM_CALCULATION
    value: when_required
  - name: AWS_RESPONSE_CHECKSUM_VALIDATION
    value: when_required
  - name: AWS_NO_CHUNKED_ENCODING
    value: "True"
  failoverDelay: 0
  instances: 3
  maxSyncReplicas: 0
  minSyncReplicas: 0
  plugins:
  - enabled: true
    isWALArchiver: true
    name: barman-cloud.cloudnative-pg.io
    parameters:
      barmanObjectName: test-pekos-repo-backup-store
    syncReplicaElectionConstraint:
      enabled: false
  primaryUpdateMethod: switchover
  primaryUpdateStrategy: unsupervised
  replicationSlots:
    highAvailability:
      enabled: true
      slotPrefix: _cnpg_
    synchronizeReplicas:
      enabled: true
    updateInterval: 30

Cluster status

  instancesStatus:
    healthy:
    - test-pekos-repo-db-cnpg-cluster-1
    - test-pekos-repo-db-cnpg-cluster-2
    - test-pekos-repo-db-cnpg-cluster-3
  lastSuccessfulBackupByMethod:
    barmanObjectStore: "2026-05-11T11:10:10Z"
  phase: Waiting for the instances to become active
  phaseReason: Some instances are not yet active. Please wait.
  pluginStatus:
  - capabilities:
    - TYPE_RECONCILER_HOOKS
    - TYPE_LIFECYCLE_SERVICE
    - TYPE_INSTANCE_SIDECAR_INJECTION
    name: barman-cloud.cloudnative-pg.io
    version: 0.12.0