Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[question] Howto avoid "Requirement 'x' not in lockfile 'requires'" with input from conan graph build-order #17363

Open
1 task done
marlamb opened this issue Nov 21, 2024 · 13 comments · May be fixed by #17656
Open
1 task done
Assignees

Comments

@marlamb
Copy link
Contributor

marlamb commented Nov 21, 2024

Hi,
I am using conan graph build-order to determine the build order for the complete dependency tree of our central project given a central lockfile. Then I use the result to build the dependencies one by one, again using the central lockfile, to ensure consistency. Now it turns out that in some situations the central lockfile does not have a requirement in the correct section (requires vs build-requires). E.g. I have a requirement to libcurl, which in turn has a build requirement to libtool, which in turn has a requirement to automake. In my lockfile libtool and automake are part of the build-requires, which makes sense. But when I do the conan create <path_to_libtool_recipe> --lockfile=<central_lockfile> <settings_and_options_parsed_from_conan_build_order_json> it fails with

ERROR: Requirement automake/... not in lockfile 'requires'

I also saw the problem the exact other way around (often also in test packages).
I read about --lockfile-partial, which (if I understand it correctly) is also not what I want, as it would loosen almost everything, but in general I of course want that the exact libs/recipe revisions from the lockfile are used.

My question is: is there a way to avoid the error while still ensuring that only defined versions/recipe revs are used? Note that using conan install would not be ideal for me, as I would like to execute also the test packages (recipes might change along and I want to ensure that they still work as intended).

Have you read the CONTRIBUTING guide?

  • I've read the CONTRIBUTING guide
@memsharded memsharded self-assigned this Nov 22, 2024
@memsharded
Copy link
Member

Hi @marlamb

Thanks for your question.

I'd say it is very likely that the lockfile or conan graph build-order capture might be missing something.
These commands must be executed with the same arguments as the expected posterior create/install commands. Same profiles, settings, etc. But most importantly the same --build=missing or similar policy.

If you don't use --build=missing when capturing the lockfile or the build-order, then they might be missing some of the tool-requires or their transitive dependencies.

Could you please try to check this?

If this is not then:

E.g. I have a requirement to libcurl, which in turn has a build requirement to libtool, which in turn has a requirement to automake. In my lockfile libtool and automake are part of the build-requires, which makes sense. But when I do the conan create <path_to_libtool_recipe> --lockfile=<central_lockfile> <settings_and_options_parsed_from_conan_build_order_json> it fails with

We would like to have a bit more of details to reproduce this. If you can share the exact commands you are using to capture this lockfile, that would help to reproduce and understand. Thanks!

@marlamb
Copy link
Contributor Author

marlamb commented Nov 24, 2024

Hi @memsharded , thanks for the quick reply. I created a minimal example that reproduces the error. It is greatly simplified and does steps manually instead of reading some of the files, but I hope it helps nevertheless to detect the underlying issue.

I use the following file structure

conan_17363
├── experiment.bash
└── recipes
    ├── build_tool
    │   └── conanfile.py
    ├── consumer
    │   └── conanfile.py
    └── transitive_dep
        └── conanfile.py

The content of the files is:

# recipes/build_tool/conanfile.py

from conan import ConanFile

class BuildToolConan(ConanFile):
    name = "build_tool"
    version = "1.0.0"

    def requirements(self) -> None:
        self.requires("transitive_dep/2.0.0@marlamb/testing")
# recipes/consumer/conanfile.py

from conan import ConanFile

class ConsumerConan(ConanFile):
    name = "consumer"
    version = "0.0.1"

    def build_requirements(self) -> None:
        self.tool_requires("build_tool/1.0.0@marlamb/testing")
# recipes/transitive_dep/conanfile.py

from conan import ConanFile

class TransitiveDepConan(ConanFile):
    name = "transitive_dep"
    version = "2.0.0"

and finally the experiments.bash file

#!/usr/bin/env bash

set -e

ROOT_DIR="$(dirname -- "$(readlink -f -- "$0";)";)"

VENV_DIR="${ROOT_DIR}/venv"
rm -rf ${VENV_DIR}
virtualenv ${VENV_DIR}
source ${VENV_DIR}/bin/activate
pip install conan

CONAN_DIR="${ROOT_DIR}/conan_home"
rm -rf ${CONAN_DIR}

export CONAN_HOME="${CONAN_DIR}"
conan remote disable "*"
conan profile detect


RECIPES_DIR="${ROOT_DIR}/recipes"
BUILD_TOOL="${RECIPES_DIR}/build_tool/conanfile.py"
TRANSITIVE_DEP="${RECIPES_DIR}/transitive_dep/conanfile.py"
CONSUMER="${RECIPES_DIR}/consumer/conanfile.py"

USER_CHANNEL="--user marlamb --channel testing"

conan export "${BUILD_TOOL}" $(echo "${USER_CHANNEL}")
conan export "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}")
conan export "${CONSUMER}" $(echo "${USER_CHANNEL}")

LOCKFILE="${ROOT_DIR}/conan.lock"

conan lock create --lockfile-out="${LOCKFILE}" --no-remote $(echo "${USER_CHANNEL}") "${CONSUMER}"

conan graph build-order --build=missing $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}" --format=json --order-by=configuration --reduce "${CONSUMER}" > "${ROOT_DIR}/build_order.json"

# In reality now the `build_order.json` gets parsed to determine all settings/options for the subsequent "conan create" commands.
# In this minimal example no settings/options exist, so the next commands would be

conan create "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}"
# The following fails with
# "ERROR: Requirement 'transitive_dep/2.0.0@marlamb/testing' not in lockfile 'requires'"
# which is obviously true, looking at the content of ${LOCKFILE}.
conan create "${BUILD_TOOL}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}"

I saw the requires field in the info section of build_tool within the build_order.json, but I don't see a way passing it to conan create.

@memsharded
Copy link
Member

Thanks very much for the detailed reproducible case, it really helped! This helps a lot to quickly identify issues.

The missing thing is that:

conan create "${BUILD_TOOL}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}"

Is missing the --build-require specifier

conan create "${BUILD_TOOL}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}" --build-require

Note that you want to build the build_tool that will work as a tool_require in the "build" context, to be used later as such by the consumer. But that conan create is creating by default things in the "host" context, unless you add the --build-require. The --build-require does the effective (and necessary in many cases) of the build-host profiles, so it will not build the package for the host context, as such application would be useless for any cross-building scenario, but it will build it for the "build" context, so the binary will exist when it is used as tool-requires.

In the conan graph build-order you should see the --tool-requires=build_tool/version --build=build_tool/version, because that build order is intended to be used with conan install, not conan create, because that is easier than having to find the url and commit and do the correct checkout for the conan create.

Please let me know if this helps.

@marlamb
Copy link
Contributor Author

marlamb commented Nov 25, 2024

Yes, indeed this seems to solve the issue. I will try that out on a bigger problem in the next days.

I was hoping to be able to use conan graph build-order also in combination with conan create in an intended way. In my system I have full control over the recipes, but I have potential changes that might break the test packages. Therefore conan create seems to be the best choice.

I extended my example a little in order to check that I got everything right also with respect to cross-build situations, but still have some questions (will follow after the example).

It now looks like this:

conan_17363
├── config
│   └── profiles
│       ├── debug
│       └── release
├── experiment.bash
└── recipes
    ├── build_tool
    │   └── conanfile.py
    ├── consumer
    │   └── conanfile.py
    ├── runtime_dep
    │   └── conanfile.py
    └── transitive_dep
        └── conanfile.py

The files are:

# config/profiles/debug
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
# config/profiles/release
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
# recipes/build_tool/conanfile.py
from conan import ConanFile

class BuildToolConan(ConanFile):
    name = "build_tool"
    version = "1.0.0"
    settings = "build_type"

    def requirements(self) -> None:
        self.requires("transitive_dep/2.0.0@marlamb/testing")
# recipes/consumer/conanfile.py
from conan import ConanFile

class ConsumerConan(ConanFile):
    name = "consumer"
    version = "0.0.1"
    settings = "build_type"
    
    def requirements(self) -> None:
        self.requires("runtime_dep/1.2.3@marlamb/testing")
# recipes/runtime_dep/conanfile.py
from conan import ConanFile

class RuntimeDepConan(ConanFile):
    name = "runtime_dep"
    version = "1.2.3"
    settings = "build_type"

    def build_requirements(self) -> None:
        self.tool_requires("build_tool/1.0.0@marlamb/testing")
# recipes/transitive_dep/conanfile.py
from conan import ConanFile

class TransitiveDepConan(ConanFile):
    name = "transitive_dep"
    version = "2.0.0"
    settings = "build_type"
#!/usr/bin/env bash

set -e

ROOT_DIR="$(dirname -- "$(readlink -f -- "$0";)";)"

VENV_DIR="${ROOT_DIR}/venv"
rm -rf ${VENV_DIR}
virtualenv ${VENV_DIR}
source ${VENV_DIR}/bin/activate
pip install conan

CONAN_DIR="${ROOT_DIR}/conan_home"
rm -rf ${CONAN_DIR}

export CONAN_HOME="${CONAN_DIR}"
conan config install "${ROOT_DIR}"/config
conan remote disable "*"

BUILD_ORDER_FILE="${ROOT_DIR}/build_order.json"
rm -f "${BUILD_ORDER_FILE}"
LOCKFILE="${ROOT_DIR}/conan.lock"
rm -f "${LOCKFILE}"

RECIPES_DIR="${ROOT_DIR}/recipes"
TRANSITIVE_DEP="${RECIPES_DIR}/transitive_dep/conanfile.py"
BUILD_TOOL="${RECIPES_DIR}/build_tool/conanfile.py"
RUNTIME_DEP="${RECIPES_DIR}/runtime_dep/conanfile.py"
CONSUMER="${RECIPES_DIR}/consumer/conanfile.py"

USER_CHANNEL="--user marlamb --channel testing"

conan export "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}")
conan export "${BUILD_TOOL}" $(echo "${USER_CHANNEL}")
conan export "${RUNTIME_DEP}" $(echo "${USER_CHANNEL}")
conan export "${CONSUMER}" $(echo "${USER_CHANNEL}")

conan lock create \
  --lockfile-out="${LOCKFILE}" \
  --no-remote \
  $(echo "${USER_CHANNEL}") \
  --profile:build=release \
  --profile:host=release \
  --profile:host=debug \
  "${CONSUMER}"

# Determine profile:host and profile:build from desired consumer profile.
conan graph build-order \
  --build=missing \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --format=json \
  --order-by=configuration \
  --reduce \
  "${CONSUMER}" > "${BUILD_ORDER_FILE}"

# Extract settings (and options) from "info" section in build.order.json file.
# Decide for "--build-require" based on "--tool-requires" in "build_args" section in build.order.json file.
conan create \
  "${TRANSITIVE_DEP}" \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --settings:host=build_type=Release \
  --build-require

conan create \
  "${BUILD_TOOL}" \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --settings:host=build_type=Release \
  --build-require

conan create \
  "${RUNTIME_DEP}" \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --settings:host=build_type=Debug

The key point is obviously where I assemble the arguments for the conan create calls. To me it looks like the settings put into the "info" section are correct and also take into account the cross-building scenario. But it needs the --build-require in order to work also with the lockfile. In this small example it would also work without parsing the settings at all, as the profiles already contain everything and the --build-require puts the correct settings/options into place. Do you have a recommendation what information should be used? Using both looks somehow overdone, but I still like the idea to just give the settings/options to the conan create, as it is very explicit.

@memsharded
Copy link
Member

I was hoping to be able to use conan graph build-order also in combination with conan create in an intended way. In my system I have full control over the recipes, but I have potential changes that might break the test packages. Therefore conan create seems to be the best choice.

Not necessarily, see my comments in the PR for the CI Tutorial: conan-io/docs#3799 (comment)

For running conan create it would be necessary to:

  • Use the scm approach to store the SCM coordinates in conandata.yml
  • For a given package reference, retrieve the scm coordinates from its conandata.yml
  • Do the git clone and checkout for those coordinates
  • conan create in the cloned folder

Steps 2 and 3 will be automated by the new conan workspace open command.

In any case, given the above link, I'd probably recommend conan install --requires=<ref> --build=<ref> as the conan graph build-order returns, focus on the overall CI process, and leave building from conan create for later.

If you haven't yet, please check conan-io/docs#3799, hopefully it will be published soon.

@marlamb
Copy link
Contributor Author

marlamb commented Dec 1, 2024

Thanks for pointing out to the PR, it contains really a lot valuable information! Now (I think) I understand, why you are building the graph-order mainly for conan install. If I am not mistaken, it is because of the asynchronous and highly decoupled nature within your system (people providing recipes are others than consuming the packages) and hence the design to separate the package and the product pipelines is natural.

But I don't think that this is the only possible design. Within the system I use no scm problem exists by design, all recipes and consumers have a consistent state at any time. Therefore no separation into two pipelines it required, one is sufficient. For that pipeline it has turned out that test_packages have quite some value to identify errors in packaging early, preventing unnecessary uploads to the Artifactory. That is why I prefer conan create. If I would go with conan install, the test packages would never be built.

Do you think it is possible to treat conan install and conan create as equal consumers of the graph-order, enabling a higher variety of (intended) CI designs? Although the --build-require helps, it feels a little hacky to me.

In any case, given the above link, I'd probably recommend conan install --requires=<ref> --build=<ref> as the conan graph build-order returns, focus on the overall CI process, and leave building from conan create for later.

I don't get the "leave building from conan create for later": if I already built everything with conan install, why should I go for a conan create later? That would result in building everything twice, wouldn't it?

@memsharded
Copy link
Member

Steps 2 and 3 will be automated by the new conan workspace open command.

Note: the conan workspace open command is being released as dev-testing in Conan 2.10 today

Do you think it is possible to treat conan install and conan create as equal consumers of the graph-order, enabling a higher variety of (intended) CI designs? Although the --build-require helps, it feels a little hacky to me.

To clarify, a conan create is a higher layer that does:

  • conan export .
  • conan install --requires=pkg/version --build=pkg/version
  • conan test test_package pkg/version

Which is slightly different to the conan install . over the current recipe, because that install the dependencies of the current pkg recipe, while the conan install --requires=pkg/version install that pkg/version and its dependencies.

I don't get the "leave building from conan create for later": if I already built everything with conan install, why should I go for a conan create later? That would result in building everything twice, wouldn't it?

When building all packages in a dependency graph that didn't get changes to their source code, but they need to be rebuilt because some of their dependencies did change, then the conan install --requires=pkg/version --build=pkg/version and conan create flows are absolutely identical, they will build exactly the same thing in the Conan cache. The only difference is that the conan create will also run after the test_package, but on the other hand it requires to do the clone+checkout of the specific repo and commit, which is not straightforward if your packages haven't captured the scm info inside the conandata.yml.

So what I mean didn't rebuild twice the same thing, I am just suggesting that implementing the CI pipeline with conan install ---requires=pkg/version --build=pkg/version is simpler to do, and it is always possible to replace that with a conan create or add a conan test step after to make sure to execute the test_package. But still the overall pipeline architecture and functionality is the same, it is just this very local detail about the running of the test_package.

Please let me know if this clarifies the questions.

@marlamb
Copy link
Contributor Author

marlamb commented Jan 16, 2025

Hi @memsharded , thanks again for the detailed explanation and sorry for the long delay.

Indeed your answer clarifies most of my questions, just one is still on my mind (currently 😉 ): if I use conan install with the central consumer conanfile.py, wouldn't this install all system dependencies within the full dependency tree regardless of the given --requires=pkg/version --build=pkg/version? That would mean that the package I built would see a system which is potentially influenced by a totally independent part of the dependency tree, wouldn't it? But perhaps I misread my logs, would be great if you could (dis)confirm it.

@memsharded
Copy link
Member

wouldn't this install all system dependencies

Do you mean system dependencies installed with the system_requirements() method and some of the SystemPackageManager helpers such as Apt, etc?

Those are indeed global, as system installed dependencies are in the system not in Conan. But they won't be seen by consumers of the graph unless they have something specifically looking for it, like some find_package() or the like that expects to be found in the system.

@marlamb
Copy link
Contributor Author

marlamb commented Jan 16, 2025

Do you mean system dependencies installed with the system_requirements() method and some of the SystemPackageManager helpers such as Apt, etc?

Indeed, this is exactly what I meant.

As a matter of fact I am not able to model everything I use via conan (although I would love to, technically it is for sure possible, its more a matter of missing time), so the pragmatic way it to have the most important parts modeled via conan and go for system_requirements at some point within the dependency tree.

Unfortunately quite some projects (especially bigger ones) have complex build systems, reacting on the system state, e.g. will build a certain variant if libsomething is present, but a fallback solution or even drop parts of their functionality if it isn't present. Hence the result of the conan build will be different depending on the systems state at the point in time of building.

This means if I build with conan install <central_consumer_conanfile> <options>, I will build packages, which are probably not correctly modeled according to the conanfile.py given its settings and options.. I.e. reusing it on a system, which does not have the system dependencies installed will happily use the package, but it might not work due to the missing system dependencies, which will not be installed alongside with the conan package.
Note that this is even true if I have only a single consumer project, because over time I might remove the system dependency in another part of the dependency tree and suddenly also independent packages might stop working, although they are not rebuilt (version, recipe revision and package id stay the same).

My preferred workaround is to use an absolute minimal machine in order build the conan packages, ensuring that I only get what is modeled within the conanfile.py. But this again requires using conan create in order to not install the system dependencies from the entire dependency tree.

As you said that conan create is only a higher level, which basically does a conan install: is there a way to use the conan install without pointing to a using conanfile.py? Finally this whole discussion is about having a native, supported way to use conan graph build-order with conan <some_command_to_build>, which does not require too many workarounds (basically stay as close as possible to the intended way of using it).

I now saw that at least for test packages the issue can also be solved, by passing --test-folder= (nothing after the equal sign) to conan create and then doing the conan test manually omitting the lockfile. But the issue remains for combinations of packages depicted in the comments above.

@memsharded
Copy link
Member

Unfortunately quite some projects (especially bigger ones) have complex build systems, reacting on the system state, e.g. will build a certain variant if libsomething is present, but a fallback solution or even drop parts of their functionality if it isn't present. Hence the result of the conan build will be different depending on the systems state at the point in time of building.

Yes, it is important that recipes do some patching to achieve consistent builds, typically controlled by options if it is desired to make it optional, but still, the recipe has to patch the build system script so they don't do unintended things. As you said, it is unfortunate, but many build scripts have this pattern of deciding to use or not something based on existence (this is known as an anti-pattern for packaging) instead of being imperative.

My preferred workaround is to use an absolute minimal machine in order build the conan packages, ensuring that I only get what is modeled within the conanfile.py. But this again requires using conan create in order to not install the system dependencies from the entire dependency tree.

I am afraid that this is not the solution either. system_requirements() are global, and they are always executed, even for dependencies.

As you said that conan create is only a higher level, which basically does a conan install: is there a way to use the conan install without pointing to a using conanfile.py?

Yes, using conan install --requires=zlib/1.3.1 --requires=... --tool-requires="cmake/[*]" can be used without a conanfile at all. But the problem will be there, system_requirements() will be called.

What the modern system_requirements() has is the capability to check only or to install via:

tools.system.package_manager:mode: Mode for package_manager tools: 'check', 'report', 'report-installed' or 'install'

And by default is check. So you can control what to do, even per-package with mypkg/*:tools.system.package_manager:mode=<value>.

@marlamb
Copy link
Contributor Author

marlamb commented Jan 25, 2025

Yes, using conan install --requires=zlib/1.3.1 --requires=... --tool-requires="cmake/[*]" can be used without a conanfile at all. But the problem will be there, system_requirements() will be called.

This was the bit I missed before, although you already wrote it. Thanks for pointing it out again!

I am afraid that this is not the solution either. system_requirements() are global, and they are always executed, even for dependencies.

Yes, it is not a perfect solution, but at least there will be no interference of independent parts of the dependency tree, which is a lot better than pointing to the central consumer file, which would install everything.

I changed my example now to the state using conan export, conan install and conan test as you suggested and used the "build_args" and the "profiles" sections from the build_order.json file. However, I fail in using conan test without again parsing information from the "info" section. Is their a point I miss again how I can ensure that the conan test does actually test the package build? I tried by passing also the package id as part of the reference, but that did not work out.

conan_17363
├── config
│   └── profiles
│       ├── debug
│       └── release
├── experiment.bash
└── recipes
    ├── build_tool
    │   └── conanfile.py
    ├── consumer
    │   └── conanfile.py
    ├── runtime_dep
    │   └── conanfile.py
    └── transitive_dep
        └── conanfile.py
        └── test_package
            └── conanfile.py

The files are:

# config/profiles/debug
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
# config/profiles/release
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
# recipes/build_tool/conanfile.py
from conan import ConanFile

class BuildToolConan(ConanFile):
    name = "build_tool"
    version = "1.0.0"
    settings = "build_type"

    def requirements(self) -> None:
        self.requires("transitive_dep/2.0.0@marlamb/testing")
# recipes/consumer/conanfile.py
from conan import ConanFile

class ConsumerConan(ConanFile):
    name = "consumer"
    version = "0.0.1"
    settings = "build_type"
    
    def requirements(self) -> None:
        self.requires("runtime_dep/1.2.3@marlamb/testing")
# recipes/runtime_dep/conanfile.py
from conan import ConanFile

class RuntimeDepConan(ConanFile):
    name = "runtime_dep"
    version = "1.2.3"
    settings = "build_type"

    def build_requirements(self) -> None:
        self.tool_requires("build_tool/1.0.0@marlamb/testing")
# recipes/transitive_dep/conanfile.py
from conan import ConanFile

class TransitiveDepConan(ConanFile):
    name = "transitive_dep"
    version = "2.0.0"
    settings = "build_type"
# recipes/build_tool/test_package/conanfile.py
from conan import ConanFile

class TestPackageConan(ConanFile):
    settings = "build_type"
    test_type = "explicit"

    def requirements(self):
        self.requires(self.tested_reference_str)

    def test(self):
        pass
#!/usr/bin/env bash

set -e

ROOT_DIR="$(dirname -- "$(readlink -f -- "$0";)";)"

VENV_DIR="${ROOT_DIR}/venv"
rm -rf ${VENV_DIR}
virtualenv ${VENV_DIR}
source ${VENV_DIR}/bin/activate
pip install conan

CONAN_DIR="${ROOT_DIR}/conan_home"
rm -rf ${CONAN_DIR}

export CONAN_HOME="${CONAN_DIR}"
conan config install "${ROOT_DIR}"/config
conan remote disable "*"

BUILD_ORDER_FILE="${ROOT_DIR}/build_order.json"
rm -f "${BUILD_ORDER_FILE}"
LOCKFILE="${ROOT_DIR}/conan.lock"
rm -f "${LOCKFILE}"

RECIPES_DIR="${ROOT_DIR}/recipes"
TRANSITIVE_DEP="${RECIPES_DIR}/transitive_dep"
BUILD_TOOL="${RECIPES_DIR}/build_tool"
RUNTIME_DEP="${RECIPES_DIR}/runtime_dep"
CONSUMER="${RECIPES_DIR}/consumer/conanfile.py"

USER_CHANNEL="--user marlamb --channel testing"

conan export "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}")
conan export "${BUILD_TOOL}" $(echo "${USER_CHANNEL}")
conan export "${RUNTIME_DEP}" $(echo "${USER_CHANNEL}")
conan export "${CONSUMER}" $(echo "${USER_CHANNEL}")

conan lock create \
  --lockfile-out="${LOCKFILE}" \
  --no-remote \
  $(echo "${USER_CHANNEL}") \
  --profile:build=release \
  --profile:host=release \
  --profile:host=debug \
  "${CONSUMER}"

# Determine profile:host and profile:build from desired consumer profile.
conan graph build-order \
  --build=missing \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --format=json \
  --order-by=configuration \
  --reduce \
  "${CONSUMER}" > "${BUILD_ORDER_FILE}"

# Use "build_args" section in build_order.json file.
conan install \
  --lockfile="${LOCKFILE}" \
  -pr:h=debug \
  -pr:b=release \
  --tool-requires=transitive_dep/2.0.0@marlamb/testing \
  --build=transitive_dep/2.0.0@marlamb/testing \
  --envs-generation false

conan test \
  ${TRANSITIVE_DEP}/test_package \
  transitive_dep/2.0.0@marlamb/testing#f001ad160e9bb5f9438da17dbe29cef1 \
  -pr:h=debug \
  -pr:b=release
  # Need to extract the settings (and options) again from the build_order.json file (add the line below).
  # --settings:host=build_type=Release

conan install \
  --lockfile="${LOCKFILE}" \
  -pr:h=debug \
  -pr:b=release \
  --tool-requires=build_tool/1.0.0@marlamb/testing \
  --build=build_tool/1.0.0@marlamb/testing \
  --envs-generation false

conan install \
  --lockfile="${LOCKFILE}" \
  -pr:h=debug \
  -pr:b=release \
  --requires=runtime_dep/1.2.3@marlamb/testing \
  --build=runtime_dep/1.2.3@marlamb/testing \
  --envs-generation false

@memsharded memsharded linked a pull request Jan 28, 2025 that will close this issue
@memsharded
Copy link
Member

Thanks for the feedback.

I was checking the code, but it seems there were a couple of minor inconsistencies, so I am moving it to a test in the Conan test suite, to make sure this is covered, and that helps a lot in reproducing and debugging. See the test in #17656, but it is not failing so far, it seems to be working fine. Could you please check it? If you'd like help running tests, please let me know.

Minor inconsistencies:

  • the first conan lock create --profile:build=release --profile:host=release --profile:host=debug contains 2 profiles for host. As they are consecutive, the second one will prevail and it will be debug one, but it seems a small unintended thing
  • There is no test_package for transitive_dep, but for build_tool, but the conan test seems to be done for transitive_dep

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants