CI/CD: Enhance the build system #22
Conversation
There was a problem hiding this comment.
Summary of Changes
Hello @haochengxia, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the project's build system by improving how dependencies are managed and installed, particularly for user-specific environments. My goal is to streamline the setup process for developers by providing an automated script to install common tools and libraries into a local directory. Concurrently, I've made the CMake configuration more flexible and resilient in finding these dependencies, ensuring a smoother build experience across different setups. These changes aim to reduce friction in getting the project up and running, especially for new contributors or in environments where system-wide installations are not preferred.
Highlights
- Automated User-Level Dependency Installation: I've introduced a new shell script,
scripts/install_deps_user.sh, which automates the installation of essential build tools (CMake, Ninja) and key project dependencies (Zstd, Google Perftools, XGBoost, LightGBM) into a user's local~/localdirectory. This script also configures environment variables and generates CMake config files to ensure these locally installed components are discoverable by the build system. - Improved CMake Dependency Resolution: I've significantly enhanced the
CMakeLists.txtto better handle user-installed dependencies. This includes setting upCMAKE_PREFIX_PATHand related variables to search~/local, and implementing more robustfind_packagelogic for ZSTD, TCMalloc, XGBoost, and LightGBM. The build system will now attempt multiple methods (e.g.,pkg-config, manual path searching) to locate these libraries, making TCMalloc and XGBoost optional and providing clearer error messages if they are not found. - Python 3 Migration and Compiler Flag Adjustments: I've updated the
CMakeLists.txtto usefind_package(Python3 ...)instead offind_package(Python ...)to explicitly target Python 3, and switched frompython_add_librarytopybind11_add_modulefor creating the Python module. Additionally, I've added-Wno-type-limitsto the GNU compiler flags to suppress specific warnings.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request significantly enhances the build system by improving dependency detection in CMakeLists.txt and adding a new script to install dependencies in a user-local environment. The changes make the project easier to build, especially for users without root permissions.
My review focuses on modernizing CMake practices, improving the robustness of the new installation script, and ensuring consistency between the build configuration and the helper scripts. I've identified a few areas for improvement, such as removing legacy CMake commands, cleaning up unused variables, and pinning dependencies to stable versions in the installation script for better reproducibility.
| # Install CMake | ||
| install_cmake() { | ||
| log_step "Installing CMake..." | ||
| local cmake_version="3.31.0" |
There was a problem hiding this comment.
The script is set to install CMake version 3.31.0. However, the CMakeLists.txt file specifies a maximum compatible version of 3.27 (cmake_minimum_required(VERSION 3.15...3.27)). Building with CMake 3.31.0 will result in an error. Please align the CMake version in this script with the range specified in CMakeLists.txt to ensure compatibility. A version like 3.27.9 would be suitable.
| local cmake_version="3.31.0" | |
| local cmake_version="3.27.9" |
| include_directories("${USER_PREFIX}/include") | ||
| link_directories("${USER_PREFIX}/lib") |
There was a problem hiding this comment.
Using include_directories and link_directories at the top-level is a legacy practice in CMake and is generally discouraged as it affects all subsequent targets globally. This can lead to unexpected behavior, such as picking up the wrong headers. This advice applies to other uses of these commands in this file as well (e.g., for ZSTD).
The modern approach is to associate include directories and libraries with specific targets using target_include_directories and target_link_libraries. Since the target libcachesim_python is defined much later, you could collect all required include paths in a list variable and then apply it to the target once it's defined.
| set(USER_PREFIX_PATHS | ||
| "$ENV{HOME}/local" | ||
| "$ENV{HOME}/.local" | ||
| "/usr/local" | ||
| ) |
There was a problem hiding this comment.
The variable USER_PREFIX_PATHS is defined here but it is not used anywhere else in the file. This appears to be dead code. Please either use it (e.g., by appending it to CMAKE_PREFIX_PATH) or remove it to avoid confusion.
| endif() | ||
|
|
||
| if(xgboost_FOUND) | ||
| include_directories(${XGBOOST_INCLUDE_DIR}) |
There was a problem hiding this comment.
The include_directories(${XGBOOST_INCLUDE_DIR}) call is redundant. The xgboost::xgboost target is created as an imported target with INTERFACE_INCLUDE_DIRECTORIES. When you link against xgboost::xgboost, CMake automatically adds the necessary include directories to the target. You can safely remove this line.
| git pull origin master | ||
| git submodule update --init --recursive |
There was a problem hiding this comment.
Building XGBoost directly from the master branch can lead to instability and non-reproducible builds, as master may contain breaking changes. It is recommended to check out and build a specific, stable release tag.
| git pull origin master | |
| git submodule update --init --recursive | |
| git fetch --all --tags | |
| git checkout tags/v2.1.0 # Or another stable release tag | |
| git submodule update --init --recursive |
| git pull origin master | ||
| git submodule update --init --recursive |
There was a problem hiding this comment.
Building LightGBM directly from the master branch can lead to instability and non-reproducible builds, as master may contain breaking changes. It is recommended to check out and build a specific, stable release tag.
| git pull origin master | |
| git submodule update --init --recursive | |
| git fetch --all --tags | |
| git checkout tags/v4.4.0 # Or another stable release tag | |
| git submodule update --init --recursive |
No description provided.