testing branch for new girder_worker_api_url environment variable

.env.default

@@ -25,7 +25,11 @@ COMPOSE_PROFILES=gpu
 #RABBITMQ_DEFAULT_VHOST=default
 #GIRDER_WORKER_BROKER=amqp://guest:guest@rabbit/default
 #GIRDER_WORKER_BACKEND=rpc://guest:guest@localhost/
+# Server-side: stamped into jobs at schedule time (worker.api_url setting)
 #GIRDER_SETTING_WORKER_API_URL=http://girder:8080/api/v1
+# Worker-side: per-worker override when this process cannot reach the stamped URL
+# (e.g. remote workers: http://WEB_HOST:8010/api/v1). Leave unset on single-node.
+#GIRDER_WORKER_API_URL=
 #GIRDER_NOTIFICATION_REDIS_URL=redis://redis:6379
 
 # Docker image tag for kitware/viame-web (workers use cpu/gpu tags in compose)
@@ -66,7 +70,7 @@ COMPOSE_PROFILES=gpu
 #ACME_EMAIL=changeme@domain.com
 #ACME_CA_SERVER=https://acme-v02.api.letsencrypt.org/directory
 #LOG_LEVEL=ERROR
-# Split or standalone worker: remote Girder API (replaces legacy WORKER_API_URL)
-#GIRDER_SETTING_WORKER_API_URL=https://viame.kitware.com/api/v1
+# Split / remote worker: override stamped API URL on the worker process
+#GIRDER_WORKER_API_URL=https://viame.kitware.com/api/v1
 #SOCK_PATH=/var/run/docker.sock
 WATCHTOWER_API_TOKEN="customtokenstring"

docs/Deployment-Docker-Compose.md

@@ -119,32 +119,63 @@ It's possible to split your web server and task runner between multiple nodes.
 * Make two cloud VM instances, one with NVIDIA drivers and container toolkit, and one without.  This is still a special case of scenario 1 from the [Provisioning Guide](Deployment-Provision.md)
 * Clone the dive repository on both VMs. The `.env` files are **not** identical — the web VM keeps the internal Compose service names, while the worker VM points at the web VM's IP or hostname (referred to below as `WEB_HOST`).
 
+### Worker API URL settings
+
+Two related variables control how workers call back to Girder. They apply at different layers:
+
+| Variable | Where to set it | Role |
+|----------|-----------------|------|
+| `GIRDER_SETTING_WORKER_API_URL` | **Girder web server** process | Sets the Girder system setting `worker.api_url`. That value is **stamped into every job** when it is scheduled (Celery headers / `jobInfoSpec`). |
+| `GIRDER_WORKER_API_URL` | **Worker** process only | Optional **per-worker override**. When set, the worker uses this URL for API callbacks instead of the URL stamped at schedule time (including job status/log updates). |
+
+!!! tip "When to set `GIRDER_WORKER_API_URL`"
+
+    Use it when a worker cannot reach the API URL stamped by the server — typically remote or external workers that cannot resolve Docker-internal names like `girder:8080`.
+
+    Typical mixed setup:
+
+    * Web VM + `localworker` on the Compose network keep the default stamped URL `http://girder:8080/api/v1`.
+    * Remote worker VMs set `GIRDER_WORKER_API_URL=http://WEB_HOST:8010/api/v1` so they reach Girder through the host-published Traefik port.
+
+    Leave `GIRDER_WORKER_API_URL` **unset** on single-node stacks and on any worker that can already reach the stamped URL.
+
 !!! warning
 
-    `GIRDER_SETTING_WORKER_API_URL` is applied by the **web server** as the `worker.api_url` Girder system setting and is then stamped into every job that is dispatched to workers. It must be set on the **web VM**. Setting it only on the worker VM has no effect on the callback URL that workers actually use.
+    `GIRDER_SETTING_WORKER_API_URL` only affects the **web server** process that schedules jobs. Setting it on a worker container does **not** change the callback URL that worker uses at runtime — use `GIRDER_WORKER_API_URL` for that.
+
+    Port `8080` is Girder's in-container port and is **not** published on the host. Traefik publishes the API on host port `8010` (base `docker-compose.yml`) or on `80`/`443` when you also use `docker-compose.prod.yml`. Remote workers should use the published port — never `8080` unless you publish it yourself.
 
 ### Web VM `.env`
 
-The web VM runs RabbitMQ, Redis, and Mongo as Compose services, so it keeps the internal service names — with one exception: `GIRDER_SETTING_WORKER_API_URL` must be an address the **remote worker** can reach, not the internal `girder:8080`.
+The web VM runs RabbitMQ, Redis, and Mongo as Compose services and should keep internal service names. For a mixed local + remote worker deployment, leave the server stamping the internal URL (the Compose default):
 
 ```
-GIRDER_SETTING_WORKER_API_URL=http://WEB_HOST:8010/api/v1
+# Optional — this is already the Compose default for a single-node / mixed stack
+#GIRDER_SETTING_WORKER_API_URL=http://girder:8080/api/v1
 ```
 
-!!! note
+`localworker` on the same Compose network can reach `girder:8080` and does not need `GIRDER_WORKER_API_URL`.
 
-    Port `8080` is Girder's in-container port and is **not** published on the host. Traefik publishes the API on host port `8010` (base `docker-compose.yml`) or on `80`/`443` when you also use `docker-compose.prod.yml`. Use the published port here — never `8080`.
+!!! note
 
-    This only applies when workers run on a **separate** host. For a single-node (all-in-one) stack, leave `GIRDER_SETTING_WORKER_API_URL` unset — the default `http://girder:8080/api/v1` is correct because the worker resolves `girder` and reaches port `8080` over the internal Compose network.
+    If **every** worker (including `localworker`) can reach a single public API URL — for example via host hairpin to `WEB_HOST:8010` — you may instead set `GIRDER_SETTING_WORKER_API_URL=http://WEB_HOST:8010/api/v1` on the web VM and omit `GIRDER_WORKER_API_URL`. Prefer `GIRDER_WORKER_API_URL` on remote workers when local and remote workers need different reachable URLs.
 
 ### Worker VM `.env`
 
 On the worker VM, uncomment and set these to point at `WEB_HOST`:
 
 * `GIRDER_WORKER_BROKER` — RabbitMQ URL on the web VM (e.g. `amqp://guest:guest@WEB_HOST/default`)
-* `GIRDER_SETTING_WORKER_API_URL` — Girder API URL on the web VM (e.g. `http://WEB_HOST:8010/api/v1`; match the web VM's value)
+* `GIRDER_WORKER_API_URL` — Girder API URL reachable from this worker (e.g. `http://WEB_HOST:8010/api/v1`)
 * `GIRDER_NOTIFICATION_REDIS_URL` — Redis URL on the web VM (e.g. `redis://WEB_HOST:6379`); required so workers can publish job/UI status notifications
 
+Example:
+
+```
+GIRDER_WORKER_BROKER=amqp://guest:guest@WEB_HOST/default
+GIRDER_WORKER_API_URL=http://WEB_HOST:8010/api/v1
+GIRDER_NOTIFICATION_REDIS_URL=redis://WEB_HOST:6379
+```
+
 ### Required connectivity
 
 The worker VM must be able to reach the web VM on these ports (port `8080` does **not** need to be open between VMs):
@@ -169,13 +200,24 @@ In this split setup, `localworker` on the web server handles the `local` queue,
 
 ### Verify the configuration
 
-After the stack is up, confirm the web server is handing out the correct callback URL (not the internal `girder:8080` default). From the web VM's Swagger page or via curl:
+After the stack is up:
+
+1. Confirm what the web server stamps into jobs (Swagger or curl on the web VM):
 
 ```
 GET /api/v1/system/setting?key=worker.api_url
 ```
 
-This must return your `http://WEB_HOST:8010/api/v1` value. If it still shows `http://girder:8080/api/v1`, the web VM's `GIRDER_SETTING_WORKER_API_URL` was not applied — jobs will appear to dispatch (and `GET /worker/status` will look healthy, since that only reflects broker connectivity) but will stall because remote workers cannot reach `girder:8080`.
+For the recommended mixed setup this is `http://girder:8080/api/v1`. That value is correct for `localworker`; remote workers rely on `GIRDER_WORKER_API_URL` instead.
+
+2. On each remote worker container, confirm the override is present:
+
+```bash
+docker compose exec girder_worker_default printenv GIRDER_WORKER_API_URL
+# -> http://WEB_HOST:8010/api/v1
+```
+
+If remote workers stall while `GET /worker/status` looks healthy (broker only), they usually cannot reach the stamped URL and `GIRDER_WORKER_API_URL` is missing or wrong.
 
 ## Addon management
 
@@ -233,7 +275,7 @@ This image contains both the backend and client.
 | GIRDER_ADMIN_PASS | `letmein` | admin password |
 | GIRDER_WORKER_BROKER | `amqp://guest:guest@rabbit/default` | RabbitMQ connection string (Celery broker) |
 | GIRDER_WORKER_BACKEND | `rpc://guest:guest@localhost/` | Celery result backend (RPC) |
-| GIRDER_SETTING_WORKER_API_URL | `http://girder:8080/api/v1` | Girder REST API URL used by workers. Default is correct for a single-node stack; override with the web host's published URL (e.g. `http://WEB_HOST:8010/api/v1`) only for [split deployments](#splitting-services). |
+| GIRDER_SETTING_WORKER_API_URL | `http://girder:8080/api/v1` | Girder system setting `worker.api_url` stamped into jobs at schedule time. Default is correct for single-node and for `localworker` on the Compose network. See [Worker API URL settings](#worker-api-url-settings). |
 | GIRDER_NOTIFICATION_REDIS_URL | `redis://redis:6379` | Redis URL for notification fan-out |
 | GIRDER_STATIC_ROOT_DIR | `/opt/dive/clients/girder` | Built web client static files (set in image/Compose) |
 
@@ -260,7 +302,8 @@ This image contains a celery worker to run VIAME pipelines and transcoding jobs.
 | WORKER_CONCURRENCY | `# of CPU cores` | max concurrnet jobs. **Lower this if you run training** |
 | WORKER_GPU_UUID | null | leave empty to use all GPUs.  Specify UUID to use specific device |
 | GIRDER_WORKER_BROKER | `amqp://guest:guest@rabbit/default` | RabbitMQ connection string. Ignored in standalone mode. |
-| GIRDER_SETTING_WORKER_API_URL | `http://girder:8080/api/v1` | Girder API URL (split/multi-node deployments). Ignored in standalone mode when using `DIVE_API_URL`. |
+| GIRDER_WORKER_API_URL | unset | Optional per-worker override of the Girder API base URL used for callbacks. Set on remote/external workers that cannot reach the URL stamped by the server (e.g. `http://WEB_HOST:8010/api/v1`). See [Worker API URL settings](#worker-api-url-settings). Ignored in standalone mode when using `DIVE_API_URL`. |
+| GIRDER_SETTING_WORKER_API_URL | `http://girder:8080/api/v1` | Only meaningful on the **web server** (stamped at schedule time). Setting it on a worker process does not override callback URLs at runtime — use `GIRDER_WORKER_API_URL` instead. |
 | GIRDER_NOTIFICATION_REDIS_URL | `redis://redis:6379` | Redis for notifications when running workers in Compose |
 | KWIVER_DEFAULT_LOG_LEVEL | `warn` | Log level for VIAME pipeline jobs (env name unchanged; used by the Kwiver logging stack) |
 | DIVE_USERNAME | null | Username to start private queue processor. Providing this enables standalone mode. |

docs/Deployment-Girder-5-Upgrade.md

@@ -21,12 +21,15 @@ Update your `.env` (and any external orchestration) to use Girder 5 names:
 | Girder 4 / legacy name | Girder 5 name | Purpose |
 |------------------------|---------------|---------|
 | `CELERY_BROKER_URL` | `GIRDER_WORKER_BROKER` | RabbitMQ URL for Celery (message broker). |
-| `WORKER_API_URL` | `GIRDER_SETTING_WORKER_API_URL` | Girder REST API URL workers call (e.g. `http://girder:8080/api/v1`). |
+| `WORKER_API_URL` | `GIRDER_SETTING_WORKER_API_URL` | Server-side Girder setting `worker.api_url` stamped into jobs at schedule time (e.g. `http://girder:8080/api/v1`). |
+| *(not used)* | `GIRDER_WORKER_API_URL` | Optional **worker-side** override of the stamped API URL for callbacks. Set on remote workers that cannot reach the server-stamped URL. |
 | *(not used)* | `GIRDER_WORKER_BACKEND` | Celery result backend (Compose default: `rpc://guest:guest@localhost/`). |
 | *(not used)* | `GIRDER_NOTIFICATION_REDIS_URL` | Redis URL for Girder notification fan-out (e.g. `redis://redis:6379`). |
 | *(not used)* | `GIRDER_STATIC_ROOT_DIR` | Path to built web client inside the Girder container (set in Compose). |
 
-`GIRDER_WORKER_BACKEND` is **not** a replacement for `WORKER_API_URL`. Do not point it at `http://…/api/v1`; use `GIRDER_SETTING_WORKER_API_URL` for that.
+`GIRDER_WORKER_BACKEND` is **not** a replacement for `WORKER_API_URL`. Do not point it at `http://…/api/v1`.
+
+Use `GIRDER_SETTING_WORKER_API_URL` on the **web server** (what gets stamped into jobs). Use `GIRDER_WORKER_API_URL` on **worker** processes when they need a different reachable API base than the stamped value. Details: [Worker API URL settings](Deployment-Docker-Compose.md#worker-api-url-settings).
 
 ## Recommended upgrade steps
 
@@ -35,7 +38,7 @@ Update your `.env` (and any external orchestration) to use Girder 5 names:
 3. **Set image tag** if needed (for example `TAG=latest` in `.env`), then pull images.
 4. **Add Redis** — use the `redis` service from `docker-compose.yml` or an external Redis instance and set `GIRDER_NOTIFICATION_REDIS_URL` on `girder` and all worker services.
 5. **Rename variables** in `.env` per the table above.
-6. **Multi-node deployments:** set `GIRDER_WORKER_BROKER` to your RabbitMQ URL and `GIRDER_SETTING_WORKER_API_URL` to your Girder API URL on every worker host; workers must reach Redis used by the web tier if they rely on notification-related features configured in Compose.
+6. **Multi-node deployments:** set `GIRDER_WORKER_BROKER` to your RabbitMQ URL and `GIRDER_WORKER_API_URL` to a Girder API URL reachable from that worker (for example `http://WEB_HOST:8010/api/v1`). Keep `GIRDER_SETTING_WORKER_API_URL` on the web VM at the value appropriate for `localworker` (often the Compose default `http://girder:8080/api/v1`). Workers must reach Redis used by the web tier if they rely on notification-related features configured in Compose.
 7. **Rebuild or pull** images, then restart the stack:
 
 ```bash

server/pyproject.toml

@@ -40,11 +40,11 @@ classifiers = [
 requires-python = ">=3.10,<3.12"
 dependencies = [
   "click>=8.1.3",
-  "girder==5.0.10",
-  "girder-client==5.0.10",
-  "girder_jobs==5.0.10",
-  "girder_worker==5.0.10",
-  "girder-plugin-worker==5.0.10",
+  "girder==5.0.14",
+  "girder-client==5.0.14",
+  "girder_jobs==5.0.14",
+  "girder_worker==5.0.14",
+  "girder-plugin-worker==5.0.14",
   "girder_worker_utils>=0.9.0,<1",
   "pydantic==1.10.13",
   "pyrabbit2==1.0.7",