building in-tree from source error: Could not find FOUND_VCS

[th0ma7]

TL;DR

Two issues found when built in-tree:

1. error: Could not find FOUND_VCS, see building in-tree from source error: Could not find FOUND_VCS #130 (comment)
2. resulting lib/cmake/VCIntrinsics14/VCIntrinsics14Config*.cmake files missing (while being installed out-of-tree), see building in-tree from source error: Could not find FOUND_VCS #130 (comment)

Details

Looking into enabling opencl with SynoCommunity ffmpeg for x86_64 arch.
Related PR: SynoCommunity/spksrc#6148 (IGC, intel-clang-opencl and al not yet included in PR, pending this issue)
Note: the following below without vc-intrinsics it does configure & build properly.

Building wiht vc-intrinsics in-tree from source with the following:

CMAKE_ARGS += -DLLVM_TARGETS_TO_BUILD=X86
CMAKE_ARGS += -DLLVM_ENABLE_PROJECTS='clang'
CMAKE_ARGS += -DLLVM_EXTERNAL_PROJECTS='llvm-spirv;opencl-clang;vc-intrinsics'
CMAKE_ARGS += -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=$(WORK_DIR)/SPIRV-LLVM-Translator
CMAKE_ARGS += -DLLVM_EXTERNAL_SPIRV_HEADERS_SOURCE_DIR=$(WORK_DIR)/SPIRV-Headers
CMAKE_ARGS += -DLLVM_EXTERNAL_VC_INTRINSICS_SOURCE_DIR=$(WORK_DIR)/vc-intrinsics
CMAKE_ARGS += -DLLVM_EXTERNAL_OPENCL_CLANG_SOURCE_DIR=$(WORK_DIR)/$(PKG_NAME)

And getting the following configuration error (present with version 0.18.0, also tested much older 0.3.0, same issue):

- Using SPIR-V Headers from
          /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/SPIRV-Headers
-- Checking for one of the modules 'SPIRV-Tools'
-- [OPENCL-CLANG] Using Clang source code direcotry: /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/llvm/../clang
-- [OPENCL-CLANG] Using SPIRV-LLVM-Translator source code directory: /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/SPIRV-LLVM-Translator
-- [OPENCL-CLANG] No patches in /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/opencl-clang/patches/clang
-- [OPENCL-CLANG] No patches in /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/opencl-clang/patches/spirv
-- Found Python: /usr/bin/python3.9 (found version "3.9.2") found components: Interpreter 
CMake Error at /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/vc-intrinsics/GenXIntrinsics/include/llvm/GenXIntrinsics/CMakeLists.txt:41 (find_file):
  Could not find FOUND_VCS using the following files: VersionFromVCS.cmake


-- Configuring incomplete, errors occurred!
make[2]: *** [../../mk/spksrc.cross-cmake.mk:150: cmake_configure_target] Error 1
make[2]: Leaving directory '/home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang'
make[1]: *** [../../mk/spksrc.supported.mk:71: build-arch-x64-7.1] Error 1
make[1]: Leaving directory '/home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang'

[th0ma7]

more context, building with the following sources:
clang-18.1.8
cmake-18.1.8
llvm-18.1.8
opencl-clang-18.1.0
SPIRV-Headers-vulkan-sdk-1.3.283.0
SPIRV-LLVM-Translator-18.1.1
SPIRV-Tools-2024.1
vc-intrinsics-0.18.0

Compiler used:
Synology DSM 7.1: gcc-8.5.0, glibc-2.26
Synology DSM 7.2: gcc-12.2.0, glibc-2.36

[th0ma7]

bump - same issue still existing using v0.19

[th0ma7]

Further investigating, the following code is being called, using LLVM v14.0.5 if falls under the else:

 31 if(${LLVM_VERSION_MAJOR} LESS 9)
 32   find_file(FOUND_VCS GetSVN.cmake PATHS ${LLVM_CMAKE_DIR} REQUIRED)
 33   add_custom_command(
 34     OUTPUT "${version_inc}"
 35     COMMAND
 36     ${CMAKE_COMMAND} "-DSOURCE_DIRS=${LLVM_GENX_INTRINSICS_REPO_DIR}"
 37                      "-DNAMES=VCI"
 38                      "-DHEADER_FILE=${version_inc}"
 39                      -P "${FOUND_VCS}")
 40 else()
 41   find_file(FOUND_VCS VersionFromVCS.cmake PATHS ${LLVM_CMAKE_DIR} REQUIRED)
 42   add_custom_command(
 43     OUTPUT "${version_inc}"
 44     COMMAND
 45     ${CMAKE_COMMAND} "-DSOURCE_DIR=${LLVM_GENX_INTRINSICS_REPO_DIR}"
 46                      "-DNAME=VCI"
 47                      "-DHEADER_FILE=${version_inc}"
 48                      "-DVCS_SCRIPT=${FOUND_VCS}"
 49                      -P "${CMAKE_CURRENT_SOURCE_DIR}/ConfigureVersionFile.cmake")
 50 endif()

But I'm not using git or svn controled sources but rather downloading latest .tar.gz version from tag https://github.com/intel/vc-intrinsics/tags Therefore the sub-call to VersionFromVCS.cmake will never provide expected result.

I'll try to use a git clone of vc-intrinsics (unless it tries to find it for llvm?) or somehow fake the svn or what ever version being caught-up. But the real fix is most probably ensuring this can be built using source tar-ball and not only dependent from a git clone.

[mshelego]

I've made a few experimental builds and found out that call to VersionFromVCS.cmake works fine even with source without a .git directory (both vc-intrinsics and llvm are from .tar.gz). In such case the vc-intrinsics revision is just defaulted to an empty string and build succeeds.

The only issue I see in above example is the missing VersionFromVCS.cmake

CMake Error at /home/spksrc/ffmpeg-update/spksrc/cross/intel-opencl-clang/work-x64-7.1/vc-intrinsics/GenXIntrinsics/include/llvm/GenXIntrinsics/CMakeLists.txt:41 (find_file):
  Could not find FOUND_VCS using the following files: VersionFromVCS.cmake

Even downloaded .tar.gz llvm sources contain this file in llvm/cmake/modules and I don't understand how could it be that cmake can't find it.

@th0ma7, could you please provide the exact sequence of commands you use and the build log?

[th0ma7]

To put a bit more context, this is a cross-compilation using Synology Linux toolchain, built from a debian 12 container. This is done using the Synocommunity build framework available at https://github.com/SynoCommunity/spksrc

When adding vc-intrinsics details within my intel-opencl-clang-140 in-tree build makefile:

BUILD_DEPENDS += cross/intel-vc-intrinsics.src
CMAKE_ARGS += -DLLVM_EXTERNAL_PROJECTS='llvm-spirv;opencl-clang;vc-intrinsics'
CMAKE_ARGS += -DLLVM_EXTERNAL_VC_INTRINSICS_SOURCE_DIR=$(WORK_DIR)/vc-intrinsics

Note that the build process creates a $(WORK_DIR)/opencl-clang.build build directory, then invokes cmake pointing to $(WORK_DIR)/llvm source directory.

It ends-up failing at configure time with:

-- [OPENCL-CLANG][Warning] /home/spksrc/ffmpeg-opencl-140/spksrc/cross/intel-opencl-clang-140/work-x64-7.1 is not a git repository, therefore, local patches are not applied
-- [OPENCL-CLANG] No patches in /home/spksrc/ffmpeg-opencl-140/spksrc/cross/intel-opencl-clang-140/work-x64-7.1/opencl-clang/patches/spirv
-- Found Python: /usr/bin/python3 (found version "3.11.2") found components: Interpreter 
mCMake Error at /home/spksrc/ffmpeg-opencl-140/spksrc/cross/intel-opencl-clang-140/work-x64-7.1/vc-intrinsics/GenXIntrinsics/include/llvm/GenXIntrinsics/CMakeLists.txt:41 (find_file):
  Could not find FOUND_VCS using the following files: VersionFromVCS.cmake

-- Configuring incomplete, errors occurred!
See also "/home/spksrc/ffmpeg-opencl-140/spksrc/cross/intel-opencl-clang-140/work-x64-7.1/opencl-clang.build/CMakeFiles/CMakeOutput.log".
See also "/home/spksrc/ffmpeg-opencl-140/spksrc/cross/intel-opencl-clang-140/work-x64-7.1/opencl-clang.build/CMakeFiles/CMakeError.log".
make[2]: *** [../../mk/spksrc.cross-cmake.mk:150: cmake_configure_target] Error 1

Adding the following solves things for me:

# intel-vc-intrinsics dependent else error:
#   Could not find FOUND_VCS using the following files: VersionFromVCS.cmake
CMAKE_ARGS += -DFOUND_VCS=$(CMAKE_BUILD_DIR)/lib/cmake/llvm/VersionFromVCS.cmake

I've added a commit on my current PR that shows the additions being made for that to work: SynoCommunity/spksrc@30c51b0

[th0ma7]

Now I recall, something else is also missing when built in-tree, the lib/cmake/VCIntrinsics14/ directory & files. When built out-of-tree the following files will show-up (while missing in-tree to llvm):

$ tree cmake/VCIntrinsics14/
cmake/VCIntrinsics14/
├── VCIntrinsics14Config.cmake
└── VCIntrinsics14Config-release.cmake

I had opened-up another issue intel/intel-graphics-compiler#334 (comment) relatively to IGC where vc-intrinsics becomes one of the issue in the grand shceme of things due to the missing VCIntrinsics14Config*.cmake files not being generated (couldn't find them anywhere) nor copied over into the installation path at install time.

EDIT: Also looked at the the full build logs (which built succesfully locally, not yet trigered thru github-action) and there is absolutely nothing related to VCIntrinsics14Config in the logs.

[th0ma7]

could you please provide the exact sequence of commands you use and the build log?

@mshelego have i provided you with enough information?

[mshelego]

@th0ma7 could you also provide the full log of the build where Could not find FOUND_VCS error happens?

[th0ma7]

Here you go! full log file when built in-tree to llvm along side with intel-opencl-clang-140 + resulting CMakeCache.txt.
build-x64-7.1.log
CMakeCache.txt

EDIT: As well my auto-generated toolchain file for cross-compiling (renamed .txt for upload):
x64-toolchain.cmake.txt

Let me know if you need anything else!

EDIT2: Additional info, I tested with cmake instead of ninja and both issues still hapens, no change.

[mshelego]

@th0ma7 I've looked into your build log and downloaded all the sources

https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.5/llvm-14.0.5.src.tar.xz
https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.5/clang-14.0.5.src.tar.xz
https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.5/lld-14.0.5.src.tar.xz
https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.5/libunwind-14.0.5.src.tar.xz
https://github.com/KhronosGroup/SPIRV-LLVM-Translator/archive/refs/tags/v14.0.2.tar.gz
https://github.com/KhronosGroup/SPIRV-Headers/archive/refs/tags/vulkan-sdk-1.3.283.0.tar.gz
https://github.com/intel/vc-intrinsics/archive/refs/tags/v0.19.0.tar.gz
https://github.com/KhronosGroup/SPIRV-Tools/archive/refs/tags/v2024.1.tar.gz
https://github.com/libffi/libffi/archive/refs/tags/v3.4.6.tar.gz
https://ftp.gnu.org/pub/gnu/ncurses/ncurses-6.4.tar.gz
https://zlib.net/fossils/zlib-1.3.1.tar.gz
https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz
https://github.com/intel/opencl-clang/archive/470cf0018e1ef6fc92eda1356f5f31f7da452abc.tar.gz

Unpacked them and made symbolic links so that the directory structure looks like this

mshelego:/user/mshelego/src/test_build$ ls -l 
build
clang -> clang-14.0.5.src
clang-14.0.5.src
cmake
libffi -> libffi-3.4.6/
libffi-3.4.6
libunwind -> libunwind-14.0.5.src
libunwind-14.0.5.src
libxml2 -> libxml2-2.9.14
libxml2-2.9.14
lld -> lld-14.0.5.src
lld-14.0.5.src
llvm -> llvm-14.0.5.src
llvm-14.0.5.src
ncurses -> ncurses-6.4
ncurses-6.4
opencl-clang -> opencl-clang-470cf0018e1ef6fc92eda1356f5f31f7da452abc/
opencl-clang-470cf0018e1ef6fc92eda1356f5f31f7da452abc
SPIRV-Headers -> SPIRV-Headers-vulkan-sdk-1.3.283.0/
SPIRV-Headers-vulkan-sdk-1.3.283.0
SPIRV-LLVM-Translator -> SPIRV-LLVM-Translator-14.0.2/
SPIRV-LLVM-Translator-14.0.2
SPIRV-Tools -> SPIRV-Tools-2024.1/
SPIRV-Tools-2024.1
vc-intrinsics -> vc-intrinsics-0.19.0/
vc-intrinsics-0.19.0
zlib -> zlib-1.3.1
zlib-1.3.1

And ran this cmake command

cmake 
-S /user/mshelego/src/test_build/llvm 
-B /user/mshelego/src/test_build/build 
-Wno-dev 
-DLLVM_ENABLE_FFI=ON 
-DLLVM_ENABLE_TERMINFO=ON 
-DLLVM_ENABLE_LIBXML2=ON 
-DLLVM_ENABLE_ZLIB=ON 
-DLLVM_ENABLE_ASSERTIONS=OFF 
-DBUILD_SHARED_LIBS=OFF 
-DLLVM_BUILD_LLVM_DYLIB=ON 
-DLLVM_LINK_LLVM_DYLIB=ON 
-DLLVM_INCLUDE_BENCHMARKS=OFF 
-DLLVM_INCLUDE_TESTS=OFF 
-DLLVM_TARGETS_TO_BUILD=X86 
-DLLVM_ENABLE_PROJECTS='clang;lld' 
-DLLVM_SOURCE_DIR=/user/mshelego/src/test_build/llvm 
-DLLVM_EXTERNAL_PROJECTS='llvm-spirv;opencl-clang;vc-intrinsics' 
-DLLVM_EXTERNAL_CLANG_SOURCE_DIR=/user/mshelego/src/test_build/clang 
-DLLVM_EXTERNAL_SPIRV_HEADERS_SOURCE_DIR=/user/mshelego/src/test_build/SPIRV-Headers 
-DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=/user/mshelego/src/test_build/SPIRV-LLVM-Translator 
-DLLVM_EXTERNAL_VC_INTRINSICS_SOURCE_DIR=/user/mshelego/src/test_build/vc-intrinsics 
-DLLVM_EXTERNAL_OPENCL_CLANG_SOURCE_DIR=/user/mshelego/src/test_build/opencl-clang 
-DCMAKE_BUILD_TYPE=Release 
-G Ninja

It finished without any errors

-- LLD version: 14.0.5
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2")
-- Using SPIR-V Headers from
          /user/mshelego/src/test_build/SPIRV-Headers
-- Checking for one of the modules 'SPIRV-Tools'
-- [OPENCL-CLANG] Using Clang source code direcotry: /user/mshelego/src/test_build/clang
-- [OPENCL-CLANG] Using SPIRV-LLVM-Translator source code directory: /user/mshelego/src/test_build/SPIRV-LLVM-Translator
-- APPLY_PATCHES is enabled.
-- [OPENCL-CLANG][Warning] /user/mshelego/src/test_build is not a git repository, therefore, local patches are not applied
-- [OPENCL-CLANG] No patches in /user/mshelego/src/test_build/opencl-clang/patches/spirv
-- Found Python: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter
-- Registering Bye as a pass plugin (static build: OFF)
-- Configuring done
-- Generating done
-- Build files have been written to: /user/mshelego/src/test_build/build

I think I need to build spksrc then, so could you please share the revision and steps to reproduce the issue?

[th0ma7]

Thnx for taking the time to figure this out. The howto is pretty much here: https://github.com/SynoCommunity/spksrc
Following the docker section will pretty much get you running with our debian image + a clone of the spksrc repository. In this case you will have to use my fork from https://github.com/th0ma7/spksrc and change to the ffmpeg-opencl-140 branch. Once in a shell in spksrc docker image, change to the cross/intel-opencl-clang-140 and invoke build using make arch-x64-7.1. Current makefile has the VCS fix, so feel free to comment that out in order to reproduce.

# intel-vc-intrinsics dependent else error:
#   Could not find FOUND_VCS using the following files: VersionFromVCS.cmake
CMAKE_ARGS += -DFOUND_VCS=$(CMAKE_BUILD_DIR)/lib/cmake/llvm/VersionFromVCS.cmake

From your testing above, what is missing is the use of a toolchain file enforcing cross-compiling against the synology linux toolchain. As things worked out ok in your testing, I believe then its related to the cross-compiling portion that it may misbehave?

I'm also available thru discord if you need further assistance.

[th0ma7]

@mshelego for your awareness, I've been cleaning-up my PR over the week-end and realigned intel-opencl-compiler-140 build to use llvm-project sources like I must do with IGC instead of individual lld|clang|llvm|libunwind. The two issues still remains though.

[mshelego]

@th0ma7 I've managed to setup spksrc and build intel-opencl-clang-140. I've reproduced the issue and found the root cause.

As you suggested earlier it's caused by the toolchain file. x64-toolchain.cmake contains these lines

 # where is the target environment located
 set(CMAKE_FIND_ROOT_PATH /user/mshelego/spksrc/cross/intel-opencl-clang-140/work-x64-7.1/install/usr/local/opencl-clang)

 # adjust the default behavior of the FIND_XXX() commands:
 # search programs in the host environment
 set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)

 # search headers and libraries in the target environment
 set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
 set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

which limit find_file to searching only inside the usr/local/opencl-clang directory located in the install folder.

Commenting the set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) line makes the build pass, but I assume that it's quite important for cross compilation.

Fortunately find_file has an NO_CMAKE_FIND_ROOT_PATH option which overrides the value of CMAKE_FIND_ROOT_PATH_MODE_INCLUDE, so the issue can be fixed in vc-intrinsics like this

find_file(FOUND_VCS VersionFromVCS.cmake PATHS ${LLVM_CMAKE_DIR} REQUIRED NO_CMAKE_FIND_ROOT_PATH)

[th0ma7]

Wow, great catch @mshelego 🚀

I recall doing the work behind cmake cross-compiling for the spksrc framework a while ago, couldn't tell what was the case where I ended-up adding set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY), I'll have to read on this further. What I find strange is that this is the first project where this problem occurs, I don't recall it hapening on any other source code, weird.

Fortunately find_file has an NO_CMAKE_FIND_ROOT_PATH option which overrides the value of CMAKE_FIND_ROOT_PATH_MODE_INCLUDE, so the issue can be fixed in vc-intrinsics like this

find_file(FOUND_VCS VersionFromVCS.cmake PATHS ${LLVM_CMAKE_DIR} REQUIRED NO_CMAKE_FIND_ROOT_PATH)

So if I get this right, this is something you intend to add in vc-intrinsics cmake definition directly (i.e. for a subsequent release)? Still I'll investigate further on my end if I can confirm what this was for.

Also, where you able to reproduce the lack of lib/cmake definition installation, only when built in-tree (i.e. the second problem)?

[mshelego]

@th0ma7 the both build issues should be fixed in the latest release. Please verify

[th0ma7]

Both issues are now fixed, thnx a lot for your time!