stackablectl release install panics

[NickLarsenNZ]

Affected version

1.0.0

Current and expected behavior

When running druid tests, with stackablectl 1.0.0 installed, something deep down panics.

❯ ./scripts/run-tests --test oidc_druid-30.0.1_zookeeper-latest-3.9.3_s3-use-tls-false_openshift-false_oidc-use-tls-false --parallel 1 --skip-delete
INFO:root:Expanding test case id [smoke_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-3.9.3_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [authorizer_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_opa-1.4.2_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [ingestion-no-s3-ext_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [ingestion-s3-ext_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [s3-deep-storage_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_s3-use-tls-true_openshift-false]
INFO:root:Expanding test case id [s3-deep-storage_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_s3-use-tls-false_openshift-false]
INFO:root:Expanding test case id [hdfs-deep-storage_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [resources_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_openshift-false]
INFO:root:Expanding test case id [orphaned-resources_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [tls_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_tls-mode-no-tls_openshift-false]
INFO:root:Expanding test case id [tls_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_tls-mode-internal-and-server-tls_openshift-false]
INFO:root:Expanding test case id [tls_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_tls-mode-internal-and-server-tls-and-tls-client-auth_openshift-false]
INFO:root:Expanding test case id [ldap_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_opa-1.4.2_hadoop-latest-3.4.1_ldap-use-tls-true_ldap-no-bind-credentials-false_openshift-false]
INFO:root:Expanding test case id [ldap_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_opa-1.4.2_hadoop-latest-3.4.1_ldap-use-tls-false_ldap-no-bind-credentials-false_openshift-false]
INFO:root:Expanding test case id [logging_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-3.4.1_openshift-false]
INFO:root:Expanding test case id [cluster-operation_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-latest-3.4.1_openshift-false]
INFO:root:Expanding test case id [oidc_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_s3-use-tls-true_openshift-false_oidc-use-tls-true]
INFO:root:Expanding test case id [oidc_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_s3-use-tls-true_openshift-false_oidc-use-tls-false]
INFO:root:Expanding test case id [oidc_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_s3-use-tls-false_openshift-false_oidc-use-tls-true]
INFO:root:Expanding test case id [oidc_druid-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_s3-use-tls-false_openshift-false_oidc-use-tls-false]
INFO:root:Expanding test case id [overrides_druid-latest-30.0.1,oci.stackable.tech_sdp_druid_30.0.1-stackable25.3.0_zookeeper-latest-3.9.3_hadoop-latest-3.4.1_openshift-false]
ERROR:root:
thread 'tokio-runtime-worker' panicked at src/pb_manager.rs:160:17:
footer progress bar was hidden despite there being pending progress bars
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'tokio-runtime-worker' panicked at /tracing-indicatif-0.3.9/src/lib.rs:680:58:
called `Result::unwrap()` on an `Err` value: PoisonError { .. }

thread 'tokio-runtime-worker' panicked at /tracing-indicatif-0.3.9/src/lib.rs:680:58:
called `Result::unwrap()` on an `Err` value: PoisonError { .. }

  18:     0x55f6b332d959 - core::ptr::drop_in_place<tracing::span::Span>::hb64438d77f1747f3
                               at /rustc-1.86.0-src/library/core/src/ptr/mod.rs:523:1
  19:     0x55f6b0a13bf9 - stackable_cockpit::platform::operator::OperatorSpec::install::hdecf19b39a616bd9
                               at /source/src/platform/operator/mod.rs:189:7
  20:     0x55f6b04cd972 - stackable_cockpit::platform::release::spec::ReleaseSpec::install::{{closure}}::{{closure}}::{{closure}}::{{closure}}::h9a4b30713685e976
                               at /source/src/platform/release/spec.rs:120:25
  21:     0x55f6b07ad100 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h49b075d97703494e
                               at /tracing-0.1.40/src/instrument.rs:321:9
  22:     0x55f6b06b4736 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h788b02bbb66b34bc

The command it appears to run is:

stackablectl release install --release-file tests/release.yaml tests

However, running that myself, it appears to succeed

So I think it might have something to do with getting a lock on stdout (maybe something to do with being executed from inside python)

It does work if I disable indicatif by manually setting the log-level.

diff --git i/scripts/run-tests w/scripts/run-tests
index 31b8e7a..6a64c65 100755
--- i/scripts/run-tests
+++ w/scripts/run-tests
@@ -286,4 +286,5 @@ def maybe_install_release(skip_release: bool, release_file: str) -> None:
         stackablectl_cmd = [
             "stackablectl",
+            "--log-level", "INFO",
             "release",
             "install",

Possible solution

No response

Additional context

No response

Environment

Nix environment (so technically, I'm using the version ahead of 1.0.0 in #384)

Would you like to work on fixing this bug?

None

[NickLarsenNZ]

It panics here tracing-indicatif/src/pb_manager.rs:160.

footer progress bar was hidden despite there being pending progress bars

And because I'm building via nix, I end up with a debug build.

It is still an issue in reality (the thread can panic), but at least I can probably just switch to an optimised build.

[NickLarsenNZ]

This problem goes away with #388, but a panic is still possible (just not at the debug_assertion site).

Here's how I tested:

#!/usr/bin/env python
# vim: filetype=python syntax=python tabstop=4 expandtab

import logging
import subprocess

stackablectl_err = ""
try:
    stackablectl_cmd = [
        "/nix/store/qmsggw88mf11pbj7z8pj1fcim0al9z5p-rust_stackablectl-1.0.0/bin/stackablectl", # release build
        # "/nix/store/fl4xk2bczg8znbjlpvdpp9501krbql33-rust_stackablectl-1.0.0/bin/stackablectl", # debug build
        "release",
        "install",
        "dev",
    ]
    logging.debug(f"Running : {stackablectl_cmd}")

    completed_proc = subprocess.run(
        stackablectl_cmd,
        capture_output=True,
        check=True,
    )
    # stackablectl doesn't return a non-zero exit code on failure
    # so we need to check stderr for errors
    stackablectl_err = completed_proc.stderr.decode("utf-8")
    if "error" in stackablectl_err.lower():
        logging.error(stackablectl_err)
        logging.error("stackablectl failed")

except subprocess.CalledProcessError as e:
    # in case stackablectl starts returning non-zero exit codes
    logging.error(e.stderr.decode("utf-8"))
    logging.error("stackablectl failed")

[NickLarsenNZ]

I'll leave this closed, and if it occurs again in the release build, I'll open it back up.

[nightkr]

Debug assertion or not, sounds like there is still a bug going on somewhere.

[NickLarsenNZ]

There definitely is (it's just caught early in debug mode).

Unfortunately I don't have the gumption to investigate further, but will if I hit the problem again.