nested looping of a test inside of a scenario produces unexpected execution order

[spleshakov]

I need to get the following order of tests

test_1.py::TestAB::test_1[a1] PASSED
test_1.py::TestAB::test_2[a1] PASSED
test_1.py::TestAB::test_3[b1-a1] PASSED
test_1.py::TestAB::test_3[b2-a1] PASSED
test_1.py::TestAB::test_3[b3-a1] PASSED

test_1.py::TestAB::test_1[a2] PASSED
test_1.py::TestAB::test_2[a2] PASSED
test_1.py::TestAB::test_3[b1-a2] PASSED
test_1.py::TestAB::test_3[b2-a2] PASSED
test_1.py::TestAB::test_3[b3-a2] PASSED

test_1.py::TestAB::test_1[a3] PASSED
test_1.py::TestAB::test_2[a3] PASSED
test_1.py::TestAB::test_3[b1-a3] PASSED
test_1.py::TestAB::test_3[b2-a3] PASSED
test_1.py::TestAB::test_3[b3-a3] PASSED

Given that

@pytest.mark.parametrize("a", ['a1', 'a2', 'a3'], scope="class")
class TestAB:

    def test_1(self, a):
        pass

    def test_2(self, a):
        pass

    def test_3(self, a):
        pass

Produces

test_1.py::TestAB::test_1[a1] PASSED
test_1.py::TestAB::test_2[a1] PASSED
test_1.py::TestAB::test_3[a1] PASSED
test_1.py::TestAB::test_1[a2] PASSED
test_1.py::TestAB::test_2[a2] PASSED
test_1.py::TestAB::test_3[a2] PASSED
test_1.py::TestAB::test_1[a3] PASSED
test_1.py::TestAB::test_2[a3] PASSED
test_1.py::TestAB::test_3[a3] PASSED

and

@pytest.mark.parametrize("b", ["b1", "b2", "b3"])
def test_3(b):
    pass

produces

test_1.py::test_3[b1] PASSED
test_1.py::test_3[b2] PASSED
test_1.py::test_3[b3] PASSED

I would expect that

@pytest.mark.parametrize("a", ['a1', 'a2', 'a3'], scope="class")
class TestAB:

    def test_1(self, a):
        pass

    def test_2(self, a):
        pass

    @pytest.mark.parametrize("b", ["b1", "b2", "b3"])
    def test_3(self, a, b):
        pass

should do the work... But instead I get

test_1.py::TestAB::test_1[a1] PASSED
test_1.py::TestAB::test_2[a1] PASSED
test_1.py::TestAB::test_3[b1-a1] PASSED
test_1.py::TestAB::test_1[a2] PASSED
test_1.py::TestAB::test_2[a2] PASSED
test_1.py::TestAB::test_3[b1-a2] PASSED
test_1.py::TestAB::test_1[a3] PASSED
test_1.py::TestAB::test_2[a3] PASSED
test_1.py::TestAB::test_3[b1-a3] PASSED
test_1.py::TestAB::test_3[b2-a1] PASSED
test_1.py::TestAB::test_3[b2-a2] PASSED
test_1.py::TestAB::test_3[b2-a3] PASSED
test_1.py::TestAB::test_3[b3-a1] PASSED
test_1.py::TestAB::test_3[b3-a2] PASSED
test_1.py::TestAB::test_3[b3-a3] PASSED

Why is this happening and how to solve it?

Thanks!

[portyu9]

Hi @spleshakov

The two parametrize declarations do not promise nested-loop execution order such as:

a1/b1
a1/b2
a1/b3
a2/b1
a2/b2
a2/b3

pytest's collection/scheduling machinery groups parametrized items according to scopes and fixture/resource setup needs. The resulting order should therefore not be used as application semantics.

If your tests depend on this sequence, I would remove that dependency.

If the specific requirement is that test_3 processes all b values while each a instance is active, current pytest's subtests fixture lets you express the actual runtime loop directly:

def test_3(self, a, subtests):
    for b in ("b1", "b2", "b3"):
        with subtests.test(b=b):
            check(a, b)

This preserves the b1 -> b2 -> b3 execution within that invocation while still reporting the individual checks as subtests.

If every (a, b) combination must instead be an independently selectable normal pytest item, then keep collection-time parametrization but do not rely on its execution order; explicit collection reordering would require custom scheduling logic.

Please mark this answer as accepted if it helped, thank you.