Generating the wheels for a package that needs Boost

[MonkeyBreaker]

Hello,

I am currently trying to generate the wheels for a library that requires boost.
I spend part of my evening trying to figure out what I must do.

To provide more details, in the current CI, I use the github action install-boost to install boost dependency on each runner. I am then able to build and test the packages without any issues.

A different workflow then takes care of generating the wheels, the wheels are generated using [linux, macos, windows]-latest:

      - name: Install boost
        uses: MarkusJx/install-boost@v2.1.0
        id: install-boost
        with:
          boost_version: 1.76.0
          boost_install_dir: ${{ matrix.boost_install_dir }}

      - name: Build wheels
        uses: pypa/cibuildwheel@v2.3.1
        env:
          CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
          # Skip 32 bit architectures, musllinux, and i686
          CIBW_SKIP: "*-win32 *-musllinux_x86_64 *_i686"
          CIBW_BEFORE_BUILD: python -m pip install cmake
          CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
          CIBW_TEST_REQUIRES: pytest
          CIBW_ENVIRONMENT: BOOST_ROOT=${{ steps.install-boost.outputs.BOOST_ROOT }}
          CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=${{ steps.install-boost.outputs.BOOST_ROOT }}
          CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

The current behavior I am observing is as follow:

• macos the wheels are correctly generated
• linux : BOOST_ROOT is empty (I print the value in my cmake scripts) and I am not even sure that the boost installed on the host will be found in the guest.
• windows: BOOST_ROOT is wrong, the backslashes are missing (e.g. D:boost instead of D:\boost), for this one I need to check if maybe my cmake is at fault.

Questions:

1. Is cibuildwheel the correct approach when the Python package depends on external C++ libraries
2. If first one is "yes", do you have any example that tries to generate the wheels with external dependencies for windows, linux and mac ?

As additional information, I paste here the printed information in github action when loading cibuildwheels:

For mac:

Run pypa/cibuildwheel@v2.3.1
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    CIBW_BUILD: cp36-* cp37-* cp38-* cp39-*
    CIBW_SKIP: *-win32 *-musllinux_x86_64 *_i686
    CIBW_BEFORE_BUILD: python -m pip install cmake
    CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
    CIBW_TEST_REQUIRES: pytest
    CIBW_ENVIRONMENT: BOOST_ROOT=/Users/runner/work/boost/boost
    CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=/Users/runner/work/boost/boost
    CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

For linux:

Run pypa/cibuildwheel@v2.3.1
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    CIBW_BUILD: cp36-* cp37-* cp38-* cp39-*
    CIBW_SKIP: *-win32 *-musllinux_x86_64 *_i686
    CIBW_BEFORE_BUILD: python -m pip install cmake
    CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
    CIBW_TEST_REQUIRES: pytest
    CIBW_ENVIRONMENT: BOOST_ROOT=/home/runner/work/boost/boost
    CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=/home/runner/work/boost/boost
    CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

For windows:

Run pypa/cibuildwheel@v2.3.1
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    CIBW_BUILD: cp36-* cp37-* cp38-* cp39-*
    CIBW_SKIP: *-win32 *-musllinux_x86_64 *_i686
    CIBW_BEFORE_BUILD: python -m pip install cmake
    CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
    CIBW_TEST_REQUIRES: pytest
    CIBW_ENVIRONMENT: BOOST_ROOT=D:\boost\boost
    CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT_PIPELINE=D:\boost\boost
    CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

Thank you for any help you can provide me with.

Best,
Julián

[henryiii]

On macOS, you are using the system libraries, so this works. Windows should be too, so this might possibly be a bug with \ being treated as a escape (not sure if it's on our side, or YAML). You might be better off with CIBW_ENVIRONMENT_PASS (and/or using static TOML configuration instead).

On linux, you are inside a docker container (manylinux, specifically), so you need to set it up there (in CIBW_BEFORE_ALL). You will need to use the system library one (very old), or build it yourself, or possibly use a separate build for it somewhere (someone else might know if there is one).

Just to verify, are you sure you need a built boost? A lot of boost is header only, and can easily be used without building anything at all - see boost-histogram, for example, which wraps a header only boost library with 6 dependent boost libraries and pybind11; no binaries required, and everything is even packed into the SDist.

[henryiii]

I think a single backslash is not required to be escaped in a normal literal in YAML, but just in case, maybe you could try adding single quotes around the expression?

[henryiii]

Something like this:

- name: Build wheels
  uses: pypa/cibuildwheel@v2.3.1
  env:
    CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
    BOOST_ROOT: '${{ steps.install-boost.outputs.BOOST_ROOT }}'
    BOOST_ROOT_PIPELINE: '${{ steps.install-boost.outputs.BOOST_ROOT }}'

[tool.cibuidwheel]
# Skip 32 bit architectures, musllinux, and i686
skip = ["*-win32", "*-musllinux_x86_64", "*_i686"]
before-build = "python -m pip install cmake"
test-command = "python -m pytest --rootdir={package} {package}"
test-requires = ["pytest"]

Actually, there's no need for environment-pass, since the path is meaningless on linux and environment-pass doesn't do anything on Windows or macOS, since you are always in the host environment.

[MonkeyBreaker]

Hi @henryiii,

Thank you for all information !

Just to verify, are you sure you need a built boost? A lot of boost is header only, and can easily be used without building anything at all - see boost-histogram, for example, which wraps a header only boost library with 6 dependent boost libraries and pybind11; no binaries required, and everything is even packed into the SDist.

You are right, I just needs the header files from some boost packages. I will have a good look on how they do this in boost-histogram, thank you.

Maybe it is a good idea that I move everything into a TOML configuration file.

I will keep you update if I make it wok !

[MonkeyBreaker]

I looked at boost-histogram, they have the dependencies as sub modules, the issue on my repository is that we have submodules that they use multiples part of boost. I think that for making my live easier, I prefer to download the entire header library of boost.

The tips you provide me with helped a lot !

I came with the following solution:

      - name: Install boost
        if: ${{ runner.os == 'Windows' || runner.os == 'macOS' }}
        uses: MarkusJx/install-boost@v2.1.0
        id: install-boost
        with:
          boost_version: 1.76.0
          boost_install_dir: ${{ matrix.boost_install_dir }}

      - name: Build wheels
        uses: pypa/cibuildwheel@v2.3.1
        env:
          # Specify which Python versions to build wheels
          # https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip
          CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
          # Skip 32 bit architectures, musllinux, and i686
          CIBW_SKIP: "*-win32 *-musllinux_x86_64 *_i686"
          CIBW_BEFORE_BUILD: python -m pip install cmake
          CIBW_BEFORE_BUILD_LINUX: echo $BOOST_ROOT
          CIBW_BEFORE_ALL_LINUX: >
            yum install -y wget &&
            cd / &&  mkdir boost && cd boost &&
            wget --no-check-certificate 'https://sourceforge.net/projects/boost/files/boost/1.76.0/boost_1_76_0.tar.bz2' && 
            tar xf boost_1_76_0.tar.bz2 &&
            cd {package}
          CIBW_TEST_COMMAND: python -m pytest --rootdir={package} {package}
          CIBW_TEST_REQUIRES: pytest
          CIBW_ENVIRONMENT_WINDOWS: BOOST_ROOT='${{ steps.install-boost.outputs.BOOST_ROOT }}'
          CIBW_ENVIRONMENT_MACOS: BOOST_ROOT='${{ steps.install-boost.outputs.BOOST_ROOT }}'
          CIBW_ENVIRONMENT_LINUX: BOOST_ROOT=/boost
          CIBW_ENVIRONMENT_PASS_LINUX: BOOST_ROOT

I don't know if I did a mistake, but I was not able to set BOOST_ROOT without using CIBW_ENVIRONMENT_*`, because it would overwrite the CIBW_ENVIRONMENT_LINUX.

I am still not sure if this solution is the best on the long term, but at least it work for the moment. It also download a 110MB file, I will look if I can find a way to cache it.

I will not migrate to use a configuration file as you suggested.

Thank you @henryiii for your suggestions and help, I don't know if I came with the best solution, but at least it works for now, I can start improving it.

[marcmgame]

I am having the same problem as described above, but in my case I do need to install the Boost library. Is there a solution to use Boost with cibuildwheel? Is there a repo example where this is done?

[mqnc]

So I had some success with this setup here:

https://github.com/AlmondBread/gestalt-motion-planner-test/blob/main/.github/workflows/build.yml

It builds for linux-intel and mac-intel (14+).

I also got Windows to start compiling cpp files (so cmake found boost) but there were so many errors (since the code was only developed on G++ before) that I gave up. It might work if the code is CL-compatible.

The semi success with Windows is here:
https://github.com/mqnc/gestalt-motion-planner/blob/76455547355eb90b9d056adbc9f5a4234c704fd3/.github/workflows/build.yml

Note that I needed to set cmake's CMP0167 policy to OLD.