Confused about the diff output of assert failures

[Feuermurmel]

I'm a daily user of pytest in work and private projects for the past 9.5 years. I'm looking at pytest's output for test failures tens to hundreds of times a day, from locally running tests, to CI to colleagues' computers. Over that time, I have not built any intuition for reading the diffs that pytest is showing me for some data types (strings, lists, etc.).

I know there has been previous discussion about this (#3333, #6673) and I think something has been changed. I've read both threads and I'm still not sure if what was proposed and/or implemented is what pytest currently (9.x) does, or if it has been reverted in the meantime.

Most of the time, I just ignore the diff, but from time to time, when I think it could help me to understand what went wrong, I'll give it a try to understand the output. But my assumption on which parts in the output relate to the UUT's actual output and which parts relate to the expected output (which both might be complex expressions) turn out to be wrong more often than not.

Imagine that the lists above are some large, deeply nested data structures that just look like noise. This is what I often have to deal with.

For the life of me, I would not be able to guess on which side the 40 above is coming from. Is it from some_function() or from generate_expected_output()? If I don't have any other clues, I'll try to guess. It feels like I'm guessing wrong about 75% of the time, no matter how hard I try to say to myself "your first intuition is going to be wrong, flip it around". No matter whether I follow that self-given advice, I'm wrong more often than not.

---

Now normally I don't give it too much thought and just give up, finding a different approach (e.g. printing the UUT's output or using a debugger).

But writing this down has forced me to think about the mechanisms in detail and this is what I have come up with:

If I had to guess what's happening in my brain is that there's a lot of intuition about looking at Git diffs (and other places where source code diffs are displayed, e.g. GitHub, GitLab, my IDE, …), where there's a very consistent correlation in presentation and meaning:

	A	B
git diff $left $right	$left	$right
Diff viewers with 2 panes	left side	right side
Colors in diff	red	green
Associated symbol	-	+
Order in unified diff	comes first	comes second
Meaning	what is given, what is already there	what I want, what change I indent/intended to make
Brokenness	Existing code which might be broken	New code with the intention of not being broken

So when I look at the 40 above, because it is green and it has a + in front of it, all the neurons associated with the table's right column in my brain light up, producing the following associations:

• There's a + and it's green. The 40 must come from the right side of the ==.
• The 40 is on the line after the 4, so it makes sense to relate to the right side. The right side is after the left side.
• Because it's on the right, it's what I declared as "what I want", so it's probably coming from the expected output.
• Green means "good", another indication that this is the correct output. The output from the UUT must thus be missing this.
• There's a +, this is what I need to add to the UUT to make it not broken anymore.
• etc.

But at the same time I've learned to not trust this intuition, my brain has probably learned to invert these signals in this context, but not all of them and not all of the time, leading to a feeling of being confused, leading to me having no grasp on when and when not to trust my intuition (i.e. inverting what my intuition is telling me).

This is a real downer for me. I guess that some people have an easier time building a strong intuition for a context, even when it contradicts the existing intuition from a different more common context, but it seems I can't even after many years. The negative correlation with diff output in other contexts is just too strong.

Other's in my work team have a similar experience with pytest. I'm neurodivergent and so are most in my team, so maybe there's an accessibility issue here as well.

---

How do you all deal with this? Is it always intuitive for you? Do you struggle with it from time to time? If so, have you found workarounds?

[VectorPeak]

The short answer: in your second screenshot, 40 is from some_function(); 4 and the extra 7 are from generate_expected_output().

pytest evaluates and displays the comparison left-to-right as left == right. In your example:

assert some_function() == generate_expected_output()
# left/actual right/expected

The text above the full diff already gives that mapping: At index 3 diff: 40 != 4 means left 40 vs right 4, and Right contains one more item: 7 means the RHS has the trailing 7.

The confusing part is that the Full diff uses the right-hand side as the baseline. With the common pytest style assert actual == expected, read it as:

- expected/right: present in expected, missing from actual
+ actual/left: present in actual, not expected

That matches the first screenshot too: - expected is the RHS string and + actual is the LHS string.

This is consistent with the direction discussed in #3333 / #6673: the signs are not “left vs right”; they are “expected baseline vs observed value”, assuming the test is written as assert result == expected. pytest cannot infer the human meaning of arbitrary expressions, so if a project writes assert expected == actual, that semantic mapping becomes inverted even though the mechanical left/right behavior is still consistent.

Your Git-diff intuition is a real issue here. In a source diff, - usually means “old” and + means “new”; in pytest, the same symbols are being used to show how the expected value differs from the actual assertion result. That is easy to misread, especially once the values are nested or the expressions are not simple variable names.

A small output change would probably make this much easier to learn. Something like this before the full diff would remove most of the guesswork:

Full diff (- expected/right, + actual/left):

or, when expression text is available:

Full diff (- generate_expected_output(), + some_function()):

For string assertions, assertion_text_diff_style=block may also be worth trying because it avoids the +/- convention and prints separate Left: / Right: blocks. That does not solve deeply nested sequences, but it is a useful workaround for the string case:

pytest -o assertion_text_diff_style=block

From a usability perspective, a docs clarification and/or a compact legend in the failure output seems reasonable. The current behavior is explainable, but your examples show that it is not self-evident, even to someone who reads this output constantly.

---

If my answer solved your problem, you can click answered the question. I'm really here to help, and along the way I'm also collecting Galaxy Brain badges haha 😆

[Feuermurmel]

Thank you for your reply!

Thanks for confirming the the behavior change discussed in [#3333](https://github.com/pytest-dev/pytest/issues/3333 )/[[#6673](https://github.com/pytest-dev/pytest/pull/6673)](https://github.com/pytest-dev/pytest/pull/6673) was in fact implemented and still matches the current behavior. So I think that that change decreased the usability for me by quite a bit. I would much prefer the inverted behavior.

But looking at the discussion in those issues, I see that a lot of people prefer and advocated the change that lead to the current behavior.

The change has been made 5 years ago and I still have not found a way to get accustomed to the new direction of the diff, the association with the order in Git diffs is just too strong, I guess. :/

---

An idea I just had: Replace the + and - markers with R and L:

---

I'll wait a bit before I accept an answer. I'd like to gather more input.

[The-Compiler]

I think the main takeaway here is that, with + and -, no matter what we do, some people will find it confusing. 😅

Personally, the mental model from #6673 works well for me:

"-" means that sth. expected is missing in the result and "+" means that there are unexpected extras in the result.

Using L and R instead would kind of break that mental model for me, and make it harder to understand which one of L / R is "actual", and which is "expected".

Inspired from #3721 (comment), maybe a nice middle ground would be a small "legend" in the output? Perhaps something like:

    def test_hello_2():
>       assert some_function() == generate_expected_output()
E       assert [1, 2, 3, 40, 5, 6] == [1, 2, 3, 4, 5, 6, 7]
E         
E         At index 3 diff: 40 != 4
E         Right contains one more item: 7
E         
E         Full diff: (-: missing in left side, +: extra in left side)
E           [
E               1,
E               2,
E               3,
E         -     4,
E         +     40,
E         ?      +
E               5,
E               6,
E         -     7,
E           ]

or if we can make it work, even:

    def test_hello_2():
>       assert some_function() == generate_expected_output()
E       assert [1, 2, 3, 40, 5, 6] == [1, 2, 3, 4, 5, 6, 7]
E         
E         At index 3 diff: 40 != 4
E         Right contains one more item: 7
E         
E         Full diff:
E           -: missing in `some_function()`
E           +: extra in `some_function()`
E
E           [
E               1,
E               2,
E               3,
E         -     4,
E         +     40,
E         ?      +
E               5,
E               6,
E         -     7,
E           ]