ResourceGroupTest fails in CI: HTTP 403 Forbidden on AICore.resourceGroups CREATE

[Schmarvinius]

Summary

ResourceGroupTest (in integration-tests/spring) consistently errors out in CI with HTTP/1.1 403 Forbidden when creating resource groups against the AICore service. 4 of 5 tests in the class fail this way; the 5th (list_resourceGroups) passes because it only reads. This is a separate, pre-existing failure from the recommendation issue tracked in #22 — it has been red since the integration-tests pipeline was first wired up, and it survives the fix in #23.

Failing tests

• ResourceGroupTest.create_andRead_resourceGroup (line 47)
• ResourceGroupTest.create_withTenantLabel_andFilterByTenant (line 68)
• ResourceGroupTest.create_withLabels (line 93)
• ResourceGroupTest.delete_resourceGroup (line 122) — the underlying CREATE setup also 403s, then DELETE 404s on the missing group

Failure (CI run 26304484099, Java 17)

[ERROR] ResourceGroupTest.create_andRead_resourceGroup:47 » ContextualizedService java.io.IOException: Failed to process response: HTTP/1.1 403 Forbidden (service 'AICore', event 'CREATE', entity 'AICore.resourceGroups')

Caused by: com.sap.cloud.sdk.services.openapi.apache.core.OpenApiRequestException: java.io.IOException: Failed to process response: HTTP/1.1 403 Forbidden
    at com.sap.ai.sdk.core.client.ResourceGroupApi.create(ResourceGroupApi.java:118)
    at com.sap.cds.feature.aicore.core.handler.ResourceGroupHandler.onCreate(ResourceGroupHandler.java:114)

The same call also produces follow-up HTTP/1.1 404 Not Found errors for the matching DELETE because the resource group was never created.

Likely cause

The CI job binds an ai-core CF service via the cf-bind action. The resulting service key apparently does not carry the scopes required to CREATE/DELETE resource groups — only to LIST/READ. Either:

1. The service plan / role collection used in the workflow secret is read-only (e.g. the sap-internal plan instance bound in CI exposes a key without aicore.resourceGroup.create permissions), or
2. The bound key targets a tenant where the calling identity does not have admin rights on /v2/admin/resourceGroups.

This is an environment / CF-service-key configuration issue, not a code bug.

Repro

CI: every PR run touching Integration Tests (Java 17|21) against the cap-java/cds-ai cap-java-ai CF space exhibits this — see runs 26303582733 (main) and 26304484099 (PR #23).

Local: cds bind --to ai-core --on cf -k AICore-btp ... followed by cds bind --exec -- mvn -pl integration-tests/spring test -Dtest=ResourceGroupTest. Whether this 403s locally depends on which user / CF role you bind under — workflow secret vs developer login may differ.

Suggested fix path

1. Check the role collection / scopes attached to the service key created by the CI workflow. The action is .github/actions/cf-bind; it logs in with ${{ secrets.CF_USERNAME }} (cap.calesi.munich@sap.com per the local CF target) and creates / reads a service key on ${{ secrets.CF_ORG_AWS }}/${{ secrets.CF_SPACE_AWS }}.
2. Either grant the technical user the AICore_Admin (or equivalent) role collection in the bound subaccount, or re-create the bound ai-core instance with a plan that exposes admin scopes.
3. If admin scopes cannot be granted to CI for policy reasons, mark the four failing methods with @DisabledIfEnvironmentVariable / a CI-skip tag, and document that resource-group CRUD is verified manually.

Workaround

Until this is sorted, the Integration Tests and SonarQube Scan jobs will remain red even with everything else passing. Reviewers should look at the surefire output rather than the job's exit code, and confirm only ResourceGroupTest.* is failing.

[Schmarvinius]

Documentation findings — confirms tenant-admin scope requirement

I dug through the SAP AI Core service guide PDF (f17fa8568d0448c685f2a0301061a6ee.pdf) to verify whether the CI 403 is recoverable through service-key configuration alone, or whether it really needs an IAM role assignment. Findings:

/v2/admin/resourceGroups CRUD requires tenant administrator privilege

• §3.2 Resource Groups (p. 58): "Tenant administrators can create or delete additional resource groups using the AI API. Tenants can map resource groups based on corresponding usage scenarios." The verb "Tenants use" applies to read-only consumption; CREATE/DELETE is gated to administrators.
• §6.3 Manage Resource Groups (p. 95): "As an administrator, you create, edit, or delete resource groups, based on your service consumers and usage scenarios."
• §11.3 Authentication and Administration (p. 158): authorization is XSUAA-based, with credentials taken from the service key. The specific scope name (something like aicore.resourceGroup.create) is enforced by the AI API XSUAA app definition, not user-configurable.

No documented way to request admin scopes when creating a service key

• §4.1.6 Create a Service Key (p. 77): the only documented JSON config parameter accepted by cf create-service-key (or the BTP cockpit equivalent) is for x.509 credential type ({"xsuaa":{"credential-type":"x509","x509":{...}}}). There is no documented role-collection, scopes, or admin parameter.

  → This rules out any "just recreate the key with admin scopes" workaround in .github/actions/cf-bind. The scopes attached to a service key are determined by the role-collection assignment of the technical user/binding identity in the BTP subaccount, which only a global-account admin can change in the cockpit.

Service plan is not the issue

• §4.1.4.1 Service Plans (p. 69): documents the public Free / Standard / Extended plans. The CI binding uses the SAP-internal sap-internal plan, which is not in the user-facing docs but functionally lines up with Standard (multiple resource groups allowed). The 403 is not because the plan disallows resource groups; it's because the bound key's identity lacks the admin role.

Local hybrid testing won't differ from CI

The local CF target on my dev machine is the same subaccount/space/user as CI:

api: cf.eu12.hana.ondemand.com
user: cap.calesi.munich@sap.com
org: cdsruntime
space: cap-java-ai
plan: sap-internal

So cds bind --exec -- mvn -pl integration-tests/spring test -Dtest=ResourceGroupTest against this binding would 403 identically to CI. There's no useful local repro distinct from the CI failure to capture.

Conclusion + next step

The 403 is exactly what the docs predict: the technical user doesn't have the AI Core tenant-admin role on the bound subaccount, and no PR-level change can grant it. To unblock the pipeline, PR #25 marks the 4 mutating tests @Disabled with a reason citing this issue. Re-enabling them requires either:

1. Granting the technical user cap.calesi.munich@sap.com the AICore tenant-admin role collection on the cdsruntime subaccount (BTP cockpit action, global-account admin), or
2. Migrating CI to a different binding identity that already has admin privileges.

Until then, the read-only readAll_returnsResourceGroups stays enabled and gives smoke coverage of the binding's GET path. PR: #25.

[Schmarvinius]

Correction: actual root cause is quota leakage, not missing admin scopes

My previous comment on this issue was based on the AI Core service guide and concluded the CI key was missing tenant-admin scopes. That conclusion was wrong. Empirical investigation against the live binding shows:

Both keys (cap-java's ai-core-key and cap-js's ai-core-key) carry identical scopes

Decoded the JWTs from both bound service keys. Both have 47 identical scopes, including the one needed for resource-group CRUD:

xsuaa_std!b318061.resourcegroup.write
xsuaa_std!b318061.resourcegroup.read
xsuaa_std!b318061.resourcegroup.delete

So the admin scopes I claimed were missing are actually present. The technical user cap.calesi.munich@sap.com already has tenant-admin role on the bound subaccount.

The 403 was actually a 400 wrapped — quota exhaustion

A direct POST /v2/admin/resourceGroups returned:

{"code":"05060007","message":"Quota for resource groups exceeded for tenant <id>","requestId":"..."}

The tenant cdsmunich had 50 / 50 resource groups (hard limit per AI Core docs §3.2). 48 of those 50 were leaked itest-rg-* groups from prior CI runs. CREATE only fails when the quota is full — explains why some PRs initially passed, then started failing as leaks accumulated.

Why cleanup never ran

ResourceGroupCleanupExtension.deleteTestResourceGroups() had this guard:

String envKey = System.getenv("AICORE_SERVICE_KEY");
if (envKey == null || envKey.isBlank()) {
  logger.debug("No AI Core binding available, skipping resource group cleanup.");
  return;
}

In CI (and under cds bind --exec), credentials are injected via VCAP_SERVICES, not AICORE_SERVICE_KEY. So the guard always tripped and cleanup silently no-op'd. Production code in AICoreServiceConfiguration.hasAICoreBinding correctly checks both VCAP_SERVICES and the env var; the test extension only checked the env var. That's the bug.

Fix (PR #25 reworked)

1. Manually wiped 48 leaked itest-rg-* groups (HTTP 202 each).
2. Reverted @Disabled annotations — the tests can run.
3. Removed the broken env-var guard from ResourceGroupCleanupExtension. The ResourceGroupApi() no-arg constructor auto-discovers from VCAP_SERVICES (same pattern as production AICoreServiceImpl).
4. Added a belt-and-suspenders cleanup-resource-groups.cjs Node script invoked with if: always() in the integration-tests action — mirrors cap-js's pattern and runs even if the JVM crashes mid-test.

PR #25 will be force-pushed with the corrected approach.