Apple silicon m2 ERROR: Could not build wheels for lightgbm, which is required to install pyproject.toml-based projects

I am experiencing an issue with installing lighgbm on Apple Silicon, the full installation process is as follows:

python3 -m pip install lightgbm
Defaulting to user installation because normal site-packages is not writeable
Collecting lightgbm
  Using cached lightgbm-4.1.0.tar.gz (1.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in ./Library/Python/3.11/lib/python/site-packages (from lightgbm) (1.23.5)
Requirement already satisfied: scipy in ./Library/Python/3.11/lib/python/site-packages (from lightgbm) (1.10.1)
Building wheels for collected packages: lightgbm
  Building wheel for lightgbm (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for lightgbm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [45 lines of output]
      2023-11-02 14:58:09,352 - scikit_build_core - INFO - CMake version: 3.27.7
      *** scikit-build-core 0.6.0 using CMake 3.27.7 (wheel)
      2023-11-02 14:58:09,356 - scikit_build_core - INFO - Build directory: /private/var/folders/7t/ys9t4mvn1fx74wckbywxd2sw0000gp/T/tmptyogi8su/build
      *** Configuring CMake...
      2023-11-02 14:58:09,372 - scikit_build_core - INFO - Ninja version: 1.11.1
      2023-11-02 14:58:09,372 - scikit_build_core - WARNING - libdir/ldlibrary: /Library/Frameworks/Python.framework/Versions/3.11/lib/Python.framework/Versions/3.11/Python is not a real file!
      loading initial cache file /var/folders/7t/ys9t4mvn1fx74wckbywxd2sw0000gp/T/tmptyogi8su/build/CMakeInit.txt
      CMake Deprecation Warning at CMakeLists.txt:35 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.
      
        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.
      
      
      -- The C compiler identification is AppleClang 15.0.0.15000040
      -- The CXX compiler identification is AppleClang 15.0.0.15000040
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
      -- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
      -- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
      -- Found OpenMP_C: -Xpreprocessor -fopenmp -I/include
      -- Found OpenMP_CXX: -Xpreprocessor -fopenmp -I/include
      -- Found OpenMP: TRUE
      -- Performing Test MM_PREFETCH
      -- Performing Test MM_PREFETCH - Failed
      -- Performing Test MM_MALLOC
      -- Performing Test MM_MALLOC - Success
      -- Using _mm_malloc
      -- Configuring done (9.3s)
      -- Generating done (0.0s)
      -- Build files have been written to: /var/folders/7t/ys9t4mvn1fx74wckbywxd2sw0000gp/T/tmptyogi8su/build
      *** Building project with Ninja...
      ninja: error: '/lib/libomp.dylib', needed by '/private/var/folders/7t/ys9t4mvn1fx74wckbywxd2sw0000gp/T/pip-install-zb1bf93d/lightgbm_d399d75f04b14379ba9d10c8bffd1542/lib_lightgbm.so', missing and no known rule to make it
      
      *** CMake build failed
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lightgbm
Failed to build lightgbm
ERROR: Could not build wheels for lightgbm, which is required to install pyproject.toml-based projects

as I understand I need to have libomp before installing lightgbm. However, I already have it installed and I still continue to experience the same issue. Is it related to the directories being different than where it is supposed to be? How can I check what the issue is?

brew install libomp      
==> Downloading https://formulae.brew.sh/api/formula.jws.json
#=#=-  #       #                                                                                                                                                                                                                                               #=O#-     #        #                                                                                                                                                                                                                                           -#O=- #      #          #                                                                                                                                                                                                                                      -=O#-   #        #           #                                                                                                                                                                                                                                 -=O=-#      #        #            #                                                                                                                                                                                                                            -=O=-  #       #          #             #                                                                                                                                                                                                                      -=O=-     #        #           #              #                                                       ######################################################################################################################################################################################################################################################### 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
######################################################################################################################################################################################################################################################### 100.0%
Warning: libomp 17.0.4 is already installed and up-to-date.
To reinstall 17.0.4, run:
  brew reinstall libomp 

Thank you for your time.

I’m on intel silicon, but here’s some things to try at least:

First start from a relatively clean slate:

pip uninstall lightgbm

Then, install miniconda (search for miniconda and go to their website to install it)

Once you have done that, then do

conda install -c conda-forge lightgbm

Conda installation will sometimes work when pip fails.

The real reason it’s failing, though, is that it is attempting to link some C binary code but that is missing. This is ugly since it’s not python anymore. I’m not too familiar with cmake so I’m not sure why brew install libomp does not compile that binary - it may be a version incompatibility, or it may be looking in the wrong place. It may be good to update CMake or to use an older version of lightgbm.

I found the solution: python - Apple silicon m2 ERROR: Could not build wheels for lightgbm, which is required to install pyproject.toml-based projects - Stack Overflow