Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion of cmake-conan evolving towards Dependency Provider. #475

Open
hwhsu1231 opened this issue Mar 9, 2023 · 10 comments
Open

Discussion of cmake-conan evolving towards Dependency Provider. #475

hwhsu1231 opened this issue Mar 9, 2023 · 10 comments

Comments

@hwhsu1231
Copy link
Contributor

hwhsu1231 commented Mar 9, 2023

Recently, I found that a new experimental branch (experiment/cmake_project_include) was created, and the Conan team has submitted a few PRs related to it:

Does cmake-conan decide to evolve towards Dependency Providers?

If that is the case, then I would be very happy to see it evolves in this direction. I think this solution may be better than the cmake-conan-setup (#449) that I previously proposed.

@memsharded
Copy link
Member

It is not a decision yet, but we are experimenting with the different possibilities, trying to achieve the best possible integration.
Certainly the dependency providers looks good, but we want to learn first if there could be some blocker or problems before announcing anything, so it is expected that we will be trying things for a couple of weeks.

@hwhsu1231
Copy link
Contributor Author

@memsharded - Have you considered the CAE problem of CMakeToolchain generator in this scheme?

@memsharded
Copy link
Member

@memsharded - Have you considered the CAE problem of CMakeToolchain generator in this scheme?

Yes, hopefully it will not be a problem

@hwhsu1231
Copy link
Contributor Author

hwhsu1231 commented Mar 12, 2023

Yes, hopefully it will not be a problem

@memsharded

Based on what CMake Docs said about Dependency Provider:

When a call is made to find_package() or FetchContent_MakeAvailable(), the call may be forwarded to a dependency provider which then has the opportunity to fulfill the request.

If I haven't misunderstood the mechanism, it seems that the xxx_provide_dependency() function is called only when we call find_package() command. For example:

cmake_minimum_required(VERSION 3.24)
project(cmake-conan-test LANGUAGES CXX)
#
# Call the xxx_provide_dependency() when the find_package(fmt) is invoked.
#
find_package(fmt)

That means the conan_toolchain.cmake file of CMakeToolchain generator will be generated after the project() command. It's because the conan install command is executed inside the conan_provide_dependency() function currently.

function(conan_provide_dependency package_name)
if(NOT CONAN_INSTALL_SUCCESS)
message(STATUS "CMake-conan: first find_package() found, running 'conan install' to install deps")
detect_host_profile(${CMAKE_BINARY_DIR}/conan_host_profile)
if(CMAKE_BUILD_TYPE)
message(STATUS "CMake-conan: Intalling single configuration ${CMAKE_BUILD_TYPE}")
conan_install(-pr ${CMAKE_BINARY_DIR}/conan_host_profile --build=missing -g CMakeDeps)
else()
message(STATUS "CMake-conan: Intalling both Debug and Release")
conan_install(-pr ${CMAKE_BINARY_DIR}/conan_host_profile -s build_type=Release --build=missing -g CMakeDeps)
conan_install(-pr ${CMAKE_BINARY_DIR}/conan_host_profile -s build_type=Debug --build=missing -g CMakeDeps)
endif()
if (CONAN_INSTALL_SUCCESS)
set(CONAN_GENERATORS_FOLDER "${CONAN_GENERATORS_FOLDER}" CACHE PATH "Conan generators folder")
endif()
else()
message(STATUS "CMake-conan: find_package(${package_name}) found, 'conan install' aready ran")
endif()
if (CONAN_GENERATORS_FOLDER)
list(PREPEND CMAKE_PREFIX_PATH "${CONAN_GENERATORS_FOLDER}")
endif()
find_package(${ARGN} BYPASS_PROVIDER)
endfunction()

However, the key of solving the CAE problem is that we have to generate the conan_toolchain.cmake file and then include(conan_toolchain.cmake) before the project() command.

Right?

@hwhsu1231 hwhsu1231 changed the title [question] Does cmake-conan decide to evolve towards Dependency Providers? Discussion of cmake-conan evolving towards Dependency Provider. Mar 12, 2023
@hwhsu1231
Copy link
Contributor Author

hwhsu1231 commented Mar 12, 2023

I conduct a simple experiment of Dependency Provider. The followings are the demo files:

  • CMakeLists.txt:

    Click to expand details
    cmake_minimum_required(VERSION 3.24)
    
    message("[DEBUG] Before calling \"project()\" command...")
    project(cmake-conan-test LANGUAGES CXX)
    message("[DEBUG] After calling \"project()\" command...")
    
    message("[DEBUG] Before calling \"find_package(fmt)\" command...")
    find_package(fmt)
    message("[DEBUG] After calling \"find_package(fmt)\" command...")
  • CMakePresets.json:

    Click to expand details
    {
      "version": 4,
      "cmakeMinimumRequired": {
        "major": 3,
        "minor": 23,
        "patch": 0
      },
      "configurePresets": [
        {
          "name": "win32",
          "displayName": "Windows",
          "description": "Windows",
          "generator": "Visual Studio 16 2019",
          "toolset": "v142,host=x64",
          "architecture": "x64",
          "binaryDir": "${sourceDir}/build/${presetName}",
          "cacheVariables": {
            "CMAKE_PROJECT_TOP_LEVEL_INCLUDES": "${sourceDir}/conan_provider.cmake"
          }
        }
      ]
    }
  • conan_provider.cmake:

    Click to expand details
    message("[DEBUG] Start to include \"conan_provider.cmake\" file...")
    
    function(conan_provide_dependency package_name)
        message("[DEBUG] Start to call \"conan_provide_dependency()\" function...")
    endfunction()
    
    cmake_language(
        SET_DEPENDENCY_PROVIDER conan_provide_dependency
        SUPPORTED_METHODS FIND_PACKAGE
    )

The following is the log of running cmake --preset win32 command:

Click to expand details
D:\Test\cmake-conan-provider>cmake --preset win32
Preset CMake variables:

  CMAKE_PROJECT_TOP_LEVEL_INCLUDES="D:/Test/cmake-conan-provider/conan_provider.cmake"

[DEBUG] Before calling "project()" command...
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.22621.
[DEBUG] Start to include "conan_provider.cmake" file...
-- The CXX compiler identification is MSVC 19.29.30147.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
[DEBUG] After calling "project()" command...
[DEBUG] Before calling "find_package(fmt)" command...
[DEBUG] Start to call "conan_provide_dependency()" function...
CMake Warning at CMakeLists.txt:8 (find_package):
  By not providing "Findfmt.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "fmt", but
  CMake did not find one.

  Could not find a package configuration file provided by "fmt" with any of
  the following names:

    fmtConfig.cmake
    fmt-config.cmake

  Add the installation prefix of "fmt" to CMAKE_PREFIX_PATH or set "fmt_DIR"
  to a directory containing one of the above files.  If "fmt" provides a
  separate development package or SDK, be sure it has been installed.


[DEBUG] After calling "find_package(fmt)" command...
-- Configuring done
-- Generating done
-- Build files have been written to: D:/Test/cmake-conan-provider/build/win32

After extracting the [DEBUG] message, we can see that conan_provide_dependency() function is indeed called after project() command:

[DEBUG] Before calling "project()" command...
[DEBUG] Start to include "conan_provider.cmake" file...
[DEBUG] After calling "project()" command...
[DEBUG] Before calling "find_package(fmt)" command...
[DEBUG] Start to call "conan_provide_dependency()" function...
[DEBUG] After calling "find_package(fmt)" command...

If that is the case, I'm afraid that this scheme might not be able to solve the CAE problem...

@jcar87
Copy link
Contributor

jcar87 commented Mar 13, 2023

we are not currently considering having Conan generate conan_toolchain.cmake (CMakeToolchain). That needs to provided before the first invocation of cmake configuration, and as such, Conan would need to be invoked first in order to generate it.

In the current experiment we simply ensure that CMakeDeps has been called to satisfy dependencies by the time we enter the first invocation of find_package inside the consumer's CMake scripts. We are also assuming the existence of a conanfile.txt or conanfile.py listing the requirements.

I have tested my current version of conan_provider.cmake, alongside your CMakeLists.txt and CMakePresets.json (with a conanfile.txt expressing the dependency on fmt), and I can see the find_package call correctly finding the Conan-provided fmt.

Note that all of this is subject to change, we are simply experimenting with Conan 2.0 and CMake's new dependency provided feature.

cmake --preset=win32
Preset CMake variables:

  CMAKE_PROJECT_TOP_LEVEL_INCLUDES="C:/Users/julius/dev/conan-io/cmake-conan/conan_provider.cmake"

-- The CXX compiler identification is MSVC 19.34.31937.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMake-conan: first find_package() found, running 'conan install' to install deps
-- Conan-cmake: cmake_system_name=Windows
-- Conan-cmake: CMake compiler=MSVC
-- Conan-cmake: CMake cmpiler version=19.34.31937.0
-- Conan-cmake: [settings] compiler=msvc
-- Conan-cmake: [settings] compiler.version=193
-- Conan-cmake: Creating profile C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan_host_profile
-- Conan-cmake: Profile:
include(default)
[settings]
os=Windows
compiler=msvc
compiler.version=193
[conf]
tools.cmake.cmaketoolchain:generator=Visual Studio 17 2022

-- CMake-conan: Intalling both Debug and Release
-- CMake-conan: conan install C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer -of=C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan -pr;C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan_host_profile;-s;build_type=Release;--build=missing;-g;CMakeDeps

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows
[conf]
tools.cmake.cmaketoolchain:generator=Visual Studio 17 2022

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows


======== Computing dependency graph ========
Graph root
    conanfile.txt: C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer\conanfile.txt
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba - Cache

======== Computing necessary packages ========
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba:bed280a9c51d41820ca64294cf373083ad852aa8#37ef2fe37424dcc4928575821a1b7f3c - Cache

======== Installing packages ========
fmt/9.1.0: Already installed! (1 of 1)
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: fmt/9.1.0
tools.cmake.cmaketoolchain:generator=Visual Studio 17 2022

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamiccompiler.runtime_type=Release
compiler.version=193
os=Windows


======== Computing dependency graph ========
Graph root
    conanfile.txt: C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer\conanfile.txt
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba - Cache

======== Computing necessary packages ========
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba:d7e170053c010c715fb01e2b851bc035c96184c1#38106a0610ea359cbf4676e33c0c07d0 - Cache

======== Installing packages ========
fmt/9.1.0: Already installed! (1 of 1)
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: fmt/9.1.0

======== Finalizing install (deploy, generators) ========
conanfile.txt: Writing generators to C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan
conanfile.txt: Generator 'CMakeDeps' calling 'generate()'
conanfile.txt: Generating aggregated env files
conanfile.txt: Generated aggregated env files: ['conanbuild.bat', 'conanrun.bat']
Install finished succesfully
-- CMake-conan: CONAN_GENERATORS_FOLDER=C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan
-- Conan: Component target declared 'fmt::fmt'
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32

@hwhsu1231
Copy link
Contributor Author

Since Conan Team insisted that we should include the conan_toolchain.cmake file in the the following previous issues:

I wondered what will happen if we don't include the conan_toolchain.cmake file?

@jcar87
Copy link
Contributor

jcar87 commented Mar 13, 2023

Since Conan Team insisted that we should include the conan_toolchain.cmake file in the the following previous issues:

I wondered what will happen if we don't include the conan_toolchain.cmake file?

It will depend on some specific use cases and workflows - the conan_toolchain does indeed set up some things that need to be defined ahead of the first invocation to project(). The dependency providers feature is limited to resolving calls to find_package - I believe there's enough information available about the os/compiler/compiler version at the point of the first invocation, that we could rely on this to invoke conan install, and resolve the call to find_package - but there will be limitations: this only works for find_package, but wouldn't work for find_program, find_library, or to provide things like the compiler itself via a conan package (can't have it both ways: either you call Conan from CMake with a compiler you've already identified, or you call Conan with the tool requirements you want CMake to use in a subsequent call). Some of these we may be able to mitigate over time, but there will indeed be limitations.

@memsharded
Copy link
Member

A proposal for 2.0 based on dependency providers and providing transparent (no need to modify CMakeLists.txt) is in https://github.com/conan-io/cmake-conan/tree/develop2, testing and feedback welcome.

@hwhsu1231
Copy link
Contributor Author

Excuse me, Conan Team.

I'm glad to see that the plan of cmake-conan evolving towards Dependecny Provider succeeds. However, as @jcar87 mentioned in his comment #475 (comment), there exist some limitations in the Dependency Provider. I'm not sure whether these limitations can be mitigated in the future. Therefore, I want to see cmake-conan-setup as a fallback plan.

Besides, although the Conan 2.0 is already released, I still use Conan 1.59.0 currently. It's because I already applied the cmake-conan-setup in some of my personal projects, but the currently latest conan.cmake is not compatible with Conan 2.0. That enforces me to stay in the Conan 1.59.0 version.

Therefore, I hope that the develop branch can be kept maintained thereafter.

First of all, could you review/merge this PR (#454) that I submitted before into develop branch? (I've been waiting it for a long time...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants