Please enhance testing workflow + docs

Thank you so much for making click. I love it and am using it for a massive project. Yesterday I dove into testing the CLI component, after having tested all the source code which underlies it. I felt pretty frustrated because testing click seemed to me to be a lot harder than other aspects of testing.

TLDR: I wish click.testing.CliRunner.invoke played nice with pytest logging, raised exceptions automatically, and returned a tuple of clean strings for multiline outputs.

Click doesn't work with pytest -o log_cli=True test_click.py -- sometimes that's necessary. I turned it off which means my logs don't print during execution. I guess that's OK, but it's often useful and fun to watch logs during tests.
E ValueError: I/O operation on closed file.
It's hard to debug problems without logs if tracebacks aren't printed by default. I would often get nothing from result.output; I figured that would have a traceback, but no. I didn't find out about if result.exc_info: raise result.exc_info[1] until later. I wish CliRunner.invoke logged exception tracebacks by default. Devs ought to have to explicitly silence exceptions, otherwise problems can be hidden. We test code to avoid hidden problems.
It's hard to test commands with multiline outputs. It took me a while to realize I had to do output_tuple = tuple(result.output.split("\n")) to get a tuple of outputs from multiple echo statements. Then there's a "" empty string at the end so it's output_tuple = tuple([r for r in result.output.split("\n") if r != ""]) I figure this would be a nice default, so we could just loop over the outputs and assert on each one, rather than fiddle with str.split and list comprehensions just to loop over CLI outputs.

Example:
pytest path/to/test_click.py

# path/to/test_click.py
import logging

from click.testing import CliRunner
import click


@click.group()
def cli():
    pass


# 1 - logging "just works"
@cli.command()
def log():
    logger = logging.getLogger(__name__)
    logger.info("habits are critical to success")
    logger.warning("don't criticize, condemn, or complain")
    logger.error("bad stuff happens to everyone")
    click.echo("logged some stuff!")  # <--- no "\n" here


def test_cli_log():
    result = CliRunner().invoke(cli, ["log"])
    # result.output isn't the same as the argument to click.echo
    # it's "logged some stuff\n" -- not "logged some stuff"
    assert (
        result.output == "logged some stuff!"
    ), "What if result.output were clean and didn't include \n?"
    # E       AssertionError: assert 'logged some stuff!\n' == 'logged some stuff!'
    assert (
        result.log
        == (  # <--- nothing... how are logs tested? (maybe pytest caplog?)
            "habits are critical to success",
            "don't criticize, condemn, or complain",
            "bad stuff happens to everyone",
        )
    ), (
        "What if Result.log is a tuple of clean log message strings?"
    )
    # E       AttributeError: 'Result' object has no attribute 'log'


# 2 - exceptions show up in output or are raised by default
@cli.command()
def crash():
    raise Exception("bad stuff happened!")


def test_cli_crash():
    # try:
    result = CliRunner().invoke(  # <--- ideally this raises exceptions automatically
        cli, ["crash"]
    )
    print("crash output:", result.output)
    print("crash exc_info:", result.exc_info)
    # ---------------------------- Captured stdout call -----------------------------
    # crash output:
    # crash exc_info: (<class 'Exception'>, Exception('bad stuff happened!'), <traceback object at 0x7ff18d161980>)
    # hmm...problems pass silently by default
    # we have to explicitly raise exceptions in every test?
    if result.exc_info:
        raise result.exc_info[1]
    assert 0, "What if CliRunner.invoke raises exceptions automatically?"
    # from "The Zen of Python:"
    #   Errors should never pass silently.
    #   Unless explicitly silenced.


# 3: multiline outputs show up as a tuple of strings:
@cli.command()
def multiline():
    click.echo("one")
    click.echo("two")


def test_cli_multiline_output():
    result = CliRunner().invoke(cli, ["multiline"])
    msg = f"multiline: {str(type(result.output))} = {result.output}"
    print(msg)
    # ---------------------------- Captured stdout call -----------------------------
    # multiline: str = one
    # two
    # ideally it would be: 
    # multiline: tuple = ("one", "two")
    assert result.output == (
        "one",
        "two",
    ), "What if result.output were a tuple of clean strings?"
    # E       AssertionError: assert 'one\ntwo\n' == ('one', 'two')
    # because then we can loop over outputs without
    # outputs = tuple([r for r in result.output.split("\n") if r != ""])
    # to make it simple, what if result.output was always a tuple, even for commands with 1 output?


if __name__ == "__main__":
    cli()

Anyway, it's a great library. I just feel like these few changes would have made my first experience with Click testing feel less frustrating and they would have saved me a few hours. Testing isn't for "fun" but this would make it more fun, or at least, less frustrating.

Thanks for your feedback. While I understand testing can be confusing, especially with CLI programs, I don't think there's much here we can change.

Click doesn't work with pytest -o log_cli=True

This is the only thing that sounds like it could potentially be addressed by Click, but it also seems like it's probably due to the way Pytest has implemented log_cli and might not be solvable. This option in Pytest seems to cause the output streams that Click is creating to be closed early. Happy to consider a PR, but I'm not going to hold this very broad issue open only for this. Might want to report it to Pytest as well, they might have feedback.

tracebacks aren't printed by default

This won't happen, because the whole point of the runner is to capture any output the command generates. However, if you don't want Click to handle exceptions in tests, you're probably looking for invoke(..., standalone_mode=False) to have exceptions bubbled up to the Result instead of handled by the CLI with an exit code. Setting catch_exceptions=False as well will cause the exceptions to bubble up to the test itself instead of recorded in the Result.

It's hard to test commands with multiline outputs.

I don't know what to tell you for this one, it mostly seems like a matter of preference. assert r.output == "a\nb\n" works fine. Your solution works too. You could use r.output.splitlines() to omit the empty last line, but an empty last line is part of how output works, it's the same with Python's print. If you find yourself testing individual lines a lot (rather than expecting an entire value, or parsing it as some other output format), you can write a helper in your tests pretty easily.

Ok, thanks for the note, ill make a little helper function to invoke with those kwargs and see how it goes

@davidism

@davidism

# https://github.com/pallets/click/issues
import logging

from click.testing import CliRunner
import click


@click.group()
def cli():
    pass


# helper:
def run_cli(*args, **kwargs):
    result = CliRunner().invoke(cli, list(args), catch_exceptions=False, **kwargs)
    output_tuple = tuple(result.output.splitlines())
    return (result.exit_code, output_tuple)


# 1 - logging "just works"
@cli.command()
def log():
    logger = logging.getLogger(__name__)
    logger.info("habits are critical to success")
    logger.warning("don't criticize, condemn, or complain")
    logger.error("bad stuff happens to everyone")
    click.echo("logged some stuff!")


def test_cli_log(caplog):
    with caplog.at_level(logging.INFO):
        exit_code, outputs = run_cli("log")
        assert exit_code == 0
        assert "logged some stuff!" in outputs
        for line in (
            "habits are critical to success",
            "don't criticize, condemn, or complain",
            "bad stuff happens to everyone",
        ):
            assert line in caplog.text


# 2 - exceptions show up in output or are raised by default
@cli.command()
def crash():
    raise Exception("bad stuff happened!")


def test_cli_crash():
    try:
        exit_code, result = run_cli("crash")
        assert 0
    except Exception as e:
        assert str(e) == "bad stuff happened!"


# 3: multiline outputs show up as a tuple of strings:
@cli.command()
def multiline():
    click.echo("one")
    click.echo("two")


def test_cli_multiline_output():
    exit_code, outputs = run_cli("multiline")
    msg = f"multiline: {type(outputs)} = {outputs}"
    print(msg)
    assert exit_code == 0
    assert outputs == ("one", "two")


if __name__ == "__main__":
    cli()

============================= test session starts =============================
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/bion/hax/greedbot
plugins: forked-1.3.0, xdist-2.2.0, order-0.9.3
collected 3 items                                                             

src/test_click.py ...                                                   [100%]

============================== 3 passed in 0.04s ==============================

Navigation Menu

Repository navigation

Please enhance testing workflow + docs #1763

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Please enhance testing workflow + docs #1763

Description

Activity

davidism commented on Jan 26, 2021

bionicles commented on Jan 26, 2021

bionicles commented on Jan 26, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions