GH-49234: [CI][Python] Nightly sdist job fails due to missing update_stub_docstrings.py file

[rok]

Rationale for this change

Nightly sdist job fails due to missing update_stub_docstrings.py file

What changes are included in this PR?

Add update_stub_docstrings.py to MANIFEST.in

Are these changes tested?

We should test with crossbow

Are there any user-facing changes?

No.

• GitHub Issue: [CI][Python] Nightly sdist job fails due to missing update_stub_docstrings.py file #49234

[github-actions]

⚠️ GitHub issue #49234 has been automatically assigned in GitHub to PR creator.

[rok]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: 99a02bf

Submitted crossbow builds: ursacomputing/crossbow @ actions-21c987b612

Task	Status
python-sdist

[raulcd]

I am wondering whether this should be part of the sdist (and wheel) or not. From what I understand this is currently executed as part of our sdist generation to update the stub docstrings so it is indeed required. A different question is whether this can be done before creating the sdist as a separate step.
I am in the mind that we should merge this, as it is indeed required now, but we should open an issue to explore whether this could be decoupled from the wheels/sdist as seems unnecessary to be delivered to end users. @rok @pitrou what do you think?

[pitrou]

I am wondering whether this should be part of the sdist (and wheel) or not. From what I understand this is currently executed as part of our sdist generation to update the stub docstrings so it is indeed required.

Well, the script is needed to generate the sdist, but why is it necessary inside the sdist?

raul note: Sorry I edited your message, I thought I clicked quote message. I've copied the old message again

[raulcd]

Well, the script is needed to generate the sdist, but why is it necessary inside the sdist?

If you try to build from the generated sdist it fails because it executes when building PyArrow, not on sdist generation:

      -- Finished cmake --build --target install for PyArrow
      -- Copying stubs: /tmp/pip-req-build-4czzjcvi/pyarrow-stubs/pyarrow -> /tmp/pip-req-build-4czzjcvi/build/lib.linux-x86_64-cpython-310/pyarrow
      error: [Errno 2] No such file or directory: '/tmp/pip-req-build-4czzjcvi/scripts/update_stub_docstrings.py'
      [end of output]

[pitrou]

If you try to build from the generated sdist it fails because it executes when building PyArrow, not on sdist generation:

Ah, ok. Then the question boils down to: would some people like to get the docstrings from the sdist, or not?
This is probably a matter how the sdist is being used at all: if it's only an intermediate step for building a wheel, then the docstrings can be generated when building a wheel. If the sdist is sometimes used as an end product of itself, then it might be better to ship the docstrings in the sdist.

[rok]

We caught the issue here, which I suppose we have a reason to test?
Other than that it seems like there are few actual users of sdist (e.g. conda-forge uses our tarball, not sure what happens on BSD, NixOS, s390x, etc). We can just not fail if the docstring population script is not present or distribute the minimal script and have consistent annotations everywhere.
I'm slightly on the distribute the script side of things.

[rok]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: 0a716fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-0607f841f4

Task	Status
python-sdist

[rok]

ping @raulcd

[raulcd]

I do agree, this seems reasonable to be added given how we are currently doing it. We can discuss later on whether we want to do this differently.

[rok]

Shall I open a followup issue for the discussion?

[raulcd]

Shall I open a followup issue for the discussion?

Sure, probably a possible improvement pointing out that we are currently copying it and whether this is something that we want to do differently so the script doesn't have to be added to the wheels / sdist for future reference.

Thanks @rok for all the work you've been doing around type hints!!

[rok]

Followup issue #49273