GOBBLIN-2265: Reliable terminal job-completion status for temporal-on-YARN (AM final status + launcher hooks + exit codes + prompt un-register)

[pratapaditya04]

Summary

Makes a temporal-on-YARN job's true outcome reliably observable end-to-end without a fragile JVM shutdown hook, exposes clean extension points so a single launcher subclass can be the sole terminal-GobblinTrackingEvent (GTE) source, and fixes a shutdown race that delayed the AM's YARN un-register by minutes. (Reworks the earlier shutdown-hook approach after finding it could silently drop the GTE.)

Tracks GOBBLIN-2265.

Problem

1. The YARN ApplicationMaster un-registered with a hardcoded FinalApplicationStatus.SUCCEEDED, so a launcher polling the ApplicationReport saw SUCCEEDED even when the workflow FAILED.
2. An earlier attempt emitted the terminal GTE from a JVM shutdown hook — but RootMetricContext registers its own shutdown hook that closes the Kafka event reporters, and hook ordering is non-deterministic, so the GTE could be dropped.
3. The AM and launcher returned success exit codes in some failure scenarios, masking failures on dashboards.
4. AM un-register was delayed by minutes on shutdown. YarnService.shutDown() waits on allContainersStopped, but the last container's removal did not reliably notify that monitor (the AMRM onContainersCompleted path never notified, and DynamicScalingYarnService additionally early-returned on the shutdownInProgress branch without notifying). The waiter then slept the full timeout, so unregisterApplicationMaster fired minutes late and YARN failed the attempt even for a successful workflow.

Changes

• AM reports the real outcome to YARN. The temporal YarnService un-registers with a FinalApplicationStatus derived from the captured Temporal workflow status (COMPLETED→SUCCEEDED, CANCELED→KILLED, else FAILED), kept in lockstep with the AM JVM exit code (mapWorkflowStatusToFinalAppStatus / computeExitCode).
• Removed the unreliable shutdown-hook GTE emitter; kept status capture + exit-code propagation.
• Feature flag (default true) — gobblin.temporal.job.completion.gte.emission.enabled gates the AM in-workflow JOB_SUCCEEDED/JOB_FAILED, so OSS stays self-complete; deployments that designate a launcher subclass as the single GTE source set it false.
• GobblinYarnAppLauncher — exit code + hooks, no emission. Surfaces FAILED/KILLED/UNDEFINED, lost-AM-visibility, and never-launched applications as a non-zero exit code, and exposes protected no-op hooks (onTerminalApplicationStatus / onLostAmVisibility / onApplicationLaunchFailure) plus getApplicationId/getApplicationName/getConfig accessors. The OSS launcher itself emits no GTE — a subclass overrides the hooks to emit the single terminal GTE.
• Prompt AM un-register on shutdown. Added YarnService.notifyIfAllContainersStopped() and call it from handleContainerCompletion and onContainerStopped, and from both early-return paths of DynamicScalingYarnService.handleContainerCompletion — so whichever callback removes the last container wakes shutDown() immediately. Reduced the fallback wait from 5m to 2m (now only a missed-notify backstop). In testing this collapsed the time-to-un-register from ~8 min to seconds after the last container stops.

Downstream idempotency

KafkaJobStatusMonitor only fires the observability event / REEVALUATE dag action on the first transition into a finished status (isNewStateTransitionToFinal), so a duplicate terminal GTE (or a Kafka redelivery) for the same flow execution is a no-op — no source-side dedup is required for correctness.

Testing

• gobblin-temporal and gobblin-yarn compile clean; unit tests pass, incl. new mapWorkflowStatusToFinalAppStatus cases and launcher exit-code/hook-dispatch tests.
• End-to-end on a prod-ltx1 GaaS distcp flow: workflow COMPLETED → un-register SUCCEEDED, AM exit 0; a permission-denied source produced workflow FAILED → un-register FAILED, AM exit 1. With the shutdown fix, the un-register fired within seconds of the last container stopping instead of waiting out the timeout.

Note

The LinkedIn-internal RobinGobblinYarnAppLauncher (separate repo) overrides the new hooks to be the single terminal-GTE source; that change lands in a follow-up PR after this publishes.

🤖 Generated with Claude Code