THREESCALE-14651 Fix HPA reconciler to skip API delete calls when HPA already absent

[urbanikb]

Summary

Fixes THREESCALE-14651: once an HPA was successfully deleted, ReconcileHpa() was calling DeleteResource() directly on every subsequent reconcile cycle, hitting a 404 indefinitely.

• Replace DeleteResource() with TagObjectToDelete + ReconcileResource, which does a cache-backed Get first and is a no-op when the object is already absent — consistent with how PDB and monitoring resources are handled
• Affects all three HPA targets: backend-listener, backend-worker, apicast-production
• Add table-driven tests covering enabled, disabled, and async-disable annotation scenarios for all three targets

Test plan

• Unit tests pass (make test-unit)
• Manually verified on fyre cluster: disabled all three HPAs via spec.backend.listenerSpec.hpa, spec.backend.workerSpec.hpa, spec.apicast.productionSpec.hpa — no repeated delete attempts in operator logs after HPAs removed

Manual testing instructions

Tested with following initial setup:

export NAMESPACE=3scale-test
make NAMESPACE=$NAMESPACE cluster/prepare/local

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: s3-credentials
  namespace: $NAMESPACE
data:
  AWS_ACCESS_KEY_ID: c29tZXRoaW5nCg==
  AWS_BUCKET: c29tZXRoaW5nCg==
  AWS_REGION: dXMtd2VzdC0xCg==
  AWS_SECRET_ACCESS_KEY: c29tZXRoaW5nCg==
type: Opaque
EOF

DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
cat << EOF | oc create -f -
kind: APIManager
apiVersion: apps.3scale.net/v1alpha1
metadata:
  name: 3scale
  namespace: $NAMESPACE
spec:
  wildcardDomain: $DOMAIN
  system:
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: s3-credentials
  externalComponents:
    backend:
      redis: true
    system:
      database: true
      redis: true
EOF

While running locally with make run.

Once system was provisioned, there were no HPAs:

% oc get hpa -n 3scale-test
No resources found in 3scale-test namespace.

Test scenarios:

1. enable HPAs on listener/worker/prod apicast

borisurbanik@MacBookPro 3scale-operator % oc patch apimanager 3scale -n 3scale-test --type=merge \
    -p '{"spec":{"backend":{"listenerSpec":{"hpa":true},"workerSpec":{"hpa":true}},"apicast":{"productionSpec":{"hpa":true}}}}'
apimanager.apps.3scale.net/3scale patched
borisurbanik@MacBookPro 3scale-operator % oc get hpa -n 3scale-test
NAME                 REFERENCE                       TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
apicast-production   Deployment/apicast-production   memory: 50%/85%, cpu: 0%/85%   1         5         1          79s
backend-listener     Deployment/backend-listener     memory: 13%/85%, cpu: 0%/85%   1         5         1          80s
backend-worker       Deployment/backend-worker       memory: 19%/85%, cpu: 0%/85%   1         5         1          79s

2. annotate with disable-async - as expected the backend HPAs have been removed

borisurbanik@MacBookPro 3scale-operator % oc patch apimanager 3scale -n 3scale-test --type=merge \
    -p '{"metadata":{"annotations":{"apps.3scale.net/disable-async":"true"}}}'
apimanager.apps.3scale.net/3scale patched
borisurbanik@MacBookPro 3scale-operator % oc get hpa -n 3scale-test
NAME                 REFERENCE                       TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
apicast-production   Deployment/apicast-production   memory: 50%/85%, cpu: 0%/85%   1         5         1          8m44s

3. remove annotation and hpa flag - all HPAs removed

borisurbanik@MacBookPro 3scale-operator %   oc patch apimanager 3scale -n 3scale-test --type=json \
    -p '[{"op":"remove","path":"/metadata/annotations/apps.3scale.net~1disable-async"},{"op":"replace","path":"/spec/backend/listenerSpec/hpa","value":false},{"op":"replace","path":"/spec/backend/workerSpec/hpa","value":false},{"op":"replace","path":"/spec/apicast/productionSpec/hpa","value":false}]'
apimanager.apps.3scale.net/3scale patched
borisurbanik@MacBookPro 3scale-operator % oc get hpa -n 3scale-test
No resources found in 3scale-test namespace.

🤖 Co-authored with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.03%. Comparing base (4963add) to head (d8fbaaf).
⚠️ Report is 18 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1175      +/-   ##
==========================================
+ Coverage   41.84%   44.03%   +2.18%     
==========================================
  Files         203      204       +1     
  Lines       20859    20923      +64     
==========================================
+ Hits         8729     9213     +484     
+ Misses      11350    10913     -437     
- Partials      780      797      +17     

Flag	Coverage Δ
unit	44.03% <100.00%> (+2.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
apis/apps/v1alpha1 (u)	63.56% <ø> (+5.01%)	⬆️
apis/capabilities/v1alpha1 (u)	3.50% <ø> (ø)
apis/capabilities/v1beta1 (u)	20.21% <ø> (ø)
controllers (i)	12.08% <88.23%> (+2.76%)	⬆️
pkg (u)	63.75% <81.90%> (+2.03%)	⬆️

Files with missing lines	Coverage Δ
...e/amp/operator/base_apimanager_logic_reconciler.go	56.80% <100.00%> (+4.17%)	⬆️

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Missing test cases

[tkan145]

/retest

[tkan145]

/retest

[tkan145]

/lgtm
/approve

Thanks. CI is failing however, and it seems the prow is currently a bit unstable. I will approve this PR but please postpone the merge until Monday.

[tkan145]

/retest

[tkan145]

Integration tests are failing because the route never available, similar results when running local tests.

Waiting for availability of Route with host 'api-3scale-apicast-production.test1.127.0.0.1.nip.io'
Waiting for availability of Route with host 'api-3scale-apicast-production.test1.127.0.0.1.nip.io'
Waiting for availability of Route with host 'api-3scale-apicast-production.test1.127.0.0.1.nip.io'
Waiting for availability of Route with host 'api-3scale-apicast-production.test1.127.0.0.1.nip.io'
Waiting for availability of Route with host 'api-3scale-apicast-production.test1.127.0.0.1.nip.io'
Waiting for availability of Route with host 'api-3scale-apicast-production.test1.127.0.0.1.nip.io'

Can you check whether it's something from us or zync is actually broken

[urbanikb]

It did create routes locally - it seems the timout of 900s might be too short

/retest

[openshift-ci]

@urbanikb: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-integration	61ef18d	link	true	/test test-integration

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[tkan145]

The latest porta image is not backward compatible, I ran integration tested locally with 2.16 porta image and the test passed.
Feel free to merge.