[WIP] [PROPOSAL] use scikit-build for packaging

[opadron]

Hi, everyone.

I have a very preliminary fork where I was able to get wheels built for symengine using scikit-build. I wanted to share it with you all early, both to gauge interest in the idea of using scikit-build (and related projects) to assist with packaging and to invite feedback on our approach.

So far, I've only managed to get wheels built locally on my macbook, but I plan on fleshing out the proposed changes and making sure the result builds on windows, and linux, too.

[opadron]

cc @thewtex @jcfr @scikit-build/developers

[thewtex]

Very cool!

CC: @isuruf @certik

It should be possible to build wheels for Windows / macOS / Linux through AppVeyor / TravisCI / CircleCI.

[jcfr]

Ditto. Well done.

It should be possible to build wheels for Windows / macOS / Linux through AppVeyor / TravisCI / CircleCI.

👍

[jcfr]

try:
    from skbuild import setup
except ImportError:
    print('scikit-build is required to build from source.', file=sys.stderr)
    print('Please run:', file=sys.stderr)
    print('', file=sys.stderr)
    print('  python -m pip install scikit-build')
    sys.exit(1)

[isuruf]

Nice. Thanks for this work.

I've evaluated scikit-build before, but didn't go with it because we'd be losing some features we have right now. From what I remember some of them are,

1. Ability to build for multiple python versions and then install them. (Needed for creating deb packages in launchpad)
2. Support for python setup.py build_ext -i (so that I can run the tests in the repo root)
3. Support for python setup.py build --compiler= (I'm not sure whether this is possible already)

I'll have a look at scikit-build code to see how hard it is to add support for these.

[opadron]

@isuruf

1: I'm not sure if this is what you mean, but I don't think there's anything in principle stopping you from repeating the build with whatever versions of python you're willing to support. python setup.py install should continue to work just like with setuptools/distutils.

2: AFAIK, this would take a little work to add to skbuild. I'll need to check in on the latest concerning our support for in-place builds.

3: Selection and configuration of the compiler would be under the purview of CMake, so you could pass an option to CMake to select your compiler. We may have added --compiler as a convenience option to skbuild. @jcfr, do you recall either way?

[isuruf]

From 1., I meant something like below.

python3.4 setup.py build
python3.5 setup.py build
python3.4 setup.py install
python3.5 setup.py install

Notice the difference between sk-build and setuptools. _skbuild/cmake-build vs build/lib.linux-x86_64-3.5

[opadron]

Hey, @isuruf sorry for the long silence, but I wanted to let you that we haven't forgotten about this investigation. I'm still very interested in exploring the possibility of improving scikit-build to meet the needs of your project, if you're willing to keep humoring us! :)

I now understand better what you were referring to on point 1. Separate build directories based on python version should be a relatively straight-forward add to skb. Do you think we should anticipate the desire to further distinguish environments based on other factors, like toolchain or target architecture?

On point 2, I had since been reminded by @jcfr that we do have some support for editable installs, through which I believe the binaries and other assets in a project may be built and assembled in-place (he would be the best person to talk to regarding the details). Would something like this address your needs?

[isuruf]

Hey, @isuruf sorry for the long silence, but I wanted to let you that we haven't forgotten about this investigation. I'm still very interested in exploring the possibility of improving scikit-build to meet the needs of your project, if you're willing to keep humoring us! :)

I'm glad that you are still interested.

I now understand better what you were referring to on point 1. Separate build directories based on python version should be a relatively straight-forward add to skb. Do you think we should anticipate the desire to further distinguish environments based on other factors, like toolchain or target architecture?

We use distutils/setuptools info to generate the build directory.

symengine.py/setup.py

Line 48 in 18b94bc

def get_build_dir(dist):

On point 2, I had since been reminded by @jcfr that we do have some support for editable installs, through which I believe the binaries and other assets in a project may be built and assembled in-place (he would be the best person to talk to regarding the details). Would something like this address your needs?

I looked at it a while ago and if skbuild would set a cmake argument like -DBUILD_INPLACE=yes then I can copy the built extensions to the src dir as a post-install script so that it doesn't mess with the manifest that skbuild is using to figure out what to move to the site-packages directory.

[isuruf]

FYI, we are building wheels separately here, https://github.com/symengine/symengine-wheels

[isuruf]

Moved to scikit-build-core in #523