[suggest][dev2] Run conan install in a subprocess when injection. (to solve CAE)

[hwhsu1231]

Suggestion

There still exists the CAE problem of CMakeToolchain in the Dependency Provider, which causes some limitations:

• [bug] dev2 branch: Could NOT find OpenSSL #478
• [suggest][dev2] Run conan install and include conan_toolchain.cmake directly when injection. #480
• [Question][devlop2] Using Conan2 CMake integration with multiple toolchains #487

However, I wondered why not adopt the techinuqe I mentioned in the cmake-conan-setup before:

Use a subproccess to run conan install command to generate conan_toolchain.cmake when injection.

The outline of the workflow may look like this:

1. When injection with CMAKE_PROJECT_TOP_LEVEL_INCLUDES:

   1. Generate a temperary CMakeLists.txt through file(WRITE) command.
   2. Call conan_install() inside the generated CMakeLists.txt.
   3. Configure the generated CMakeLists.txt through execute_process() command.
   4. Include conan_toolchain.cmake after conan install successes.

2. When find_package() invokes conan_provide_dependency():

   1. Append ${CONAN_OUTPUT_FOLDER} into CMAKE_PREFIX_PATH.
   2. Append ${CONAN_OUTPUT_FOLDER} into CMAKE_MODULE_PATH.
   3. Call find_package(${ARGN} BYPASS_PROVIDER).

Demonstration

The following project is a simple prototype, in which I only considered the situation using MSVC compiler with Ninja generator:

1. Extract the following archived file.

   cmake-conan-provider.zip

2. Call vcvarsall.bat x64 to initialize the MSVC env:

   vcvarsall.bat x64

3. Run the following cmake command to configure the porject:

   cmake --preset win32-msvc16-x64-ninja-debug

Logs

Click to expand logs

D:\Test\cmake-conan-provider>vcvarsall.bat x64
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.23
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

D:\Test\cmake-conan-provider>cmake --preset win32-msvc16-x64-ninja-debug
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  CMAKE_CXX_COMPILER="cl.exe"
  CMAKE_C_COMPILER="cl.exe"
  CMAKE_PROJECT_TOP_LEVEL_INCLUDES="D:/Test/cmake-conan-provider/conan_provider.cmake"


======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Debug
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Debug
compiler.version=192
os=Windows
[conf]
tools.cmake.cmaketoolchain:generator=Ninja

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows


======== Computing dependency graph ========
Graph root
    conanfile.py: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan\conanfile.py
Requirements
    fmt/8.1.1#e63eae266b23cf3d7f3e0dfe33d1f7e9 - Cache

======== Computing necessary packages ========
Requirements
    fmt/8.1.1#e63eae266b23cf3d7f3e0dfe33d1f7e9:a6b1a9b5962f3f69ca4f26a80d8d82e9ba1fabe7#f1762e8333d32db4225378e23e3b4070 - Cache

======== Installing packages ========
fmt/8.1.1: Already installed! (1 of 1)
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: fmt/8.1.1

======== Finalizing install (deploy, generators) ========
conanfile.py: Writing generators to D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan
conanfile.py: Generator 'CMakeToolchain' calling 'generate()'
conanfile.py: CMakeToolchain generated: conan_toolchain.cmake
conanfile.py: Preset 'conan-debug' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-debug'
conanfile.py: If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G Ninja -DCMAKE_TOOLCHAIN_FILE=D:\Test\cmake-conan-provider\build\win32-msvc16-x64-ninja-debug\conan\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Debug'
conanfile.py: CMakeToolchain generated: CMakePresets.json
conanfile.py: Generator 'CMakeDeps' calling 'generate()'
conanfile.py: Generating aggregated env files
conanfile.py: Generated aggregated env files: ['conanbuild.bat', 'conanrun.bat']
Install finished succesfully
-- Using Conan toolchain: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 14 with extensions OFF
-- The C compiler identification is MSVC 19.29.30147.0
-- The CXX compiler identification is MSVC 19.29.30147.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
========== cmake-conan-provider ==========
-- Conan: Component target declared 'fmt::fmt'
-- Configuring done
-- Generating done
-- Build files have been written to: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug

As we can see in the above logs, the conan_toolchain.cmake file is included and fmt-config.cmake file is find_packaged successfully. Although I only considered using MSVC compiler with Ninja generator currently, it proves that this scheme is feasible at least. What do you think, Conan Team?

-- Using Conan toolchain: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 14 with extensions OFF
...
...
...
========== cmake-conan-provider ==========
-- Conan: Component target declared 'fmt::fmt'

[hwhsu1231]

@memsharded

This scheme should be able to solve the situation you mentioned in: #480 (comment), right?

[hwhsu1231]

The generated directory structure will look like this:

[hwhsu1231]

And I noticed that it won't generate a CMakeUserPresets.json in the root source directory. Therefore, it might be able to solve #484 as well.

[memsharded]

The approach described cannot detect correctly CMake current compiler and configuration.

I) Generate a temperary CMakeLists.txt through file(WRITE) command.
II) Call conan_install() inside the generated CMakeLists.txt.

When conan_install() is executed at that point, it cannot detect the current CMake configuration, because it has not been configured yet. One of the main reasons of existence of cmake-conan is being able to get the current CMake configuration and install dependencies for that configuration, and this is not possible if you execute conan install before the toolchain is processed. Thus, this is not a viable approach.

[hwhsu1231]

The approach described cannot detect correctly CMake current compiler and configuration.

I) Generate a temperary CMakeLists.txt through file(WRITE) command.
II) Call conan_install() inside the generated CMakeLists.txt.

When conan_install() is executed at that point, it cannot detect the current CMake configuration, because it has not been configured yet. One of the main reasons of existence of cmake-conan is being able to get the current CMake configuration and install dependencies for that configuration, and this is not possible if you execute conan install before the toolchain is processed. Thus, this is not a viable approach.

@memsharded - Are you sure? Well...

Recently, QtCreator updated its auto-setup.cmake on 10.0 branch:

• Gerrit Review Request: auto-setup: Add support for conan 2.0
• GitHub's Link: auto-setup.cmake

As we can see, Qt Team uses nearly the same method as mine to integrate conan_support.cmake with its auto-setup.cmake. It seems that they treat conan_support.cmake as an upgrade of conan.cmake.

1. Generate a temperary CMakeLists.txt by file(WRITE) command:

   https://github.com/qt-creator/qt-creator/blob/5633533e0d485de15aa9ab03bf8a1d6229364558/src/share/3rdparty/package-manager/auto-setup.cmake#L85-L117

2. Configure the generated CMakeLists.txt by execute_process() command:

   https://github.com/qt-creator/qt-creator/blob/5633533e0d485de15aa9ab03bf8a1d6229364558/src/share/3rdparty/package-manager/auto-setup.cmake#L119-L127

Since my cmake-conan-setup project is inspired by the design of auto-setup.cmake of QtCreator, and I haven't met any fatal error so far, I'm curious about what you said. Are you saying that Qt Team's idea is simply a mistake?

[memsharded]

Their approach is to provide a profile: -pr \"${CMAKE_BINARY_DIR}/conan-dependencies/conan_host_profile, but this is not the main use case that users expect from cmake-conan, and that requires discipline and user alignment, which might be ok for qt beause they control the configuration, but not for the majority of Conan cmake-conan users.

cc/ @cristianadam

[hwhsu1231]

But their conan_host_profile is generated by the detect_host_profile() function provided by conan_support.cmake?

https://github.com/qt-creator/qt-creator/blob/5633533e0d485de15aa9ab03bf8a1d6229364558/src/share/3rdparty/package-manager/auto-setup.cmake#L94

[cristianadam]

I did the same things as the conan_provide_dependency function https://github.com/conan-io/cmake-conan/blob/develop2/conan_support.cmake#L164-L173

@hwhsu1231 thank you for looking in to Qt Creator's source code. I do hope that it works for you, if not feel free to open bug reports at http://bugreports.qt.io/

[memsharded]

The key of running function(conan_provide_dependency package_name) that contains inside the detect_host_profile(${CMAKE_BINARY_DIR}/conan_host_profile) is that it runs when a find_package() is found, which happens much later than the project() has been called and also much later that the user CMake input configuration (command line arguments like -DCMAKE_CXX_COMPILER or user provided toolchains) has been processed.

Trying to detect the profile before that, as @hwhsu1231 suggests, doesn't work, and results in the wrong profile being detected and passed to conan install, so it is not a valid alternative.

[hwhsu1231]

@memsharded

I still don't get it. Why will the wrong profile be detected? Can you provide an example?

In my opinion, if we can pass exactly the same CMake CLI arguments as the main-process is using to the sub-process, then it should be able to generate the correct profile in the sub-process. What do I miss?

[hwhsu1231]

Excuse me, @cristianadam.

Do you understand what @memsharded was talking about? I think the integration with cmake-conan that QtCreator proposed is great and feasible. However, @memsharded said that this only works with Qt, but doesn't work with the majority of Conan cmake-conan users.

Why?

[hwhsu1231]

If I understand correctly, what @memsharded insisted is that:

• detect_host_profile() must be called after the project(). Otherwise, it will detect the wrong profile, which will be passed to conan_install() command. Right?

But what I'm saying is that:

• In QtCreator's method, it is indeed after the project(). But it's not after the project() of the main-process CMakeLists.txt. Instead, it's after that of the sub-process CMakeLists.txt, which is generated by the file(WRITE) command and configured by the execute_process() command.

• And in my opinion, if we can pass nearly the same CLI arguments to the sub-process CMakeLists.txt as we pass to the main-process CMakeLists.txt, then detect_host_profile() should be able to detect the correct profile passed to conan_install() command.

[hwhsu1231]

@memsharded @jcar87 @czoido @prince-chrismc

Hello Conan Team. Can you give some responses, please?

[memsharded]

Sorry, we are busy, things take time and there are many ongoing parallel priorities, no need to rush the team, we already have this in our agendas, please be patience.