立即构建工具，以便稍后在同一 CMake 运行中使用 [英] Building a tool immediately so it can be used later in same CMake run

问题描述

我有一个有趣的先有鸡还是先有蛋的问题，并且有一个潜在的解决方案(请参阅我发布的答案)，但该解决方案以不同寻常的方式使用了 CMake.欢迎提供更好的替代方案或评论.

I have an interesting chicken-and-egg problem and a potential solution to it (see my posted answer), but that solution uses CMake in an unusual way. Better alternatives or comments would be welcome.

问题:

问题的简单版本可以描述为具有以下特征的单个CMake项目:

The simple version of the problem can be described as a single CMake project with the following characteristics:

1. 其中一个构建目标是一个命令行可执行文件，我将其命名为 mycomp，其源代码位于 mycompdir 中，并对内容进行任何修改该目录是不可能的.
2. 该项目包含文本文件(我将它们称为 foo.my 和 bar.my)，需要在它们上运行 mycomp 才能生成一组 C++ 源和头文件，以及一些定义从这些源构建的库的 CMakeLists.txt 文件.
3. 同一项目中的其他构建目标需要链接到由那些生成的 CMakeLists.txt 文件定义的库.这些其他目标也有 #include 一些生成的标头的来源.

1. One of the build targets is a command-line executable which I'll call mycomp, the source of which is in a mycompdir and making any modifications to the contents of that directory is not possible.
2. The project contains text files (I'll call them foo.my and bar.my) which need mycomp run on them to produce a set of C++ sources and headers and some CMakeLists.txt files defining libraries built from those sources.
3. Other build targets in the same project need to link against the libraries defined by those generated CMakeLists.txt files. These other targets also have sources which #include some of the generated headers.

您可以将 mycomp 视为类似于编译器的东西，而将步骤 2 中的文本文件视为某种源文件.这会带来一个问题，因为 CMake 在配置时需要 CMakeLists.txt 文件，但 mycomp 直到构建时才可用，因此在第一次运行时不可用尽早创建 CMakeLists.txt 文件.

You can think of mycomp as being something like a compiler and the text files in step 2 as some sort of source files. This presents a problem, because CMake needs the CMakeLists.txt files at configure time, but mycomp is not available until build time and therefore isn't available on the first run to create the CMakeLists.txt files early enough.

非回答:

通常，基于 ExternalProject 的超级构建安排将是一个潜在的解决方案，但以上是我正在处理的实际项目的相当大的简化，我没有自由将构建分成不同的部分或执行其他大规模重组工作.

Normally, an ExternalProject-based superbuild arrangement would be a potential solution to this, but the above is a considerable simplification of the actual project I am dealing with and I don't have the freedom to split the build into different parts or perform other large scale restructuring work.

推荐答案

问题的关键是需要 mycomp 在 CMake 运行时可用，以便生成的 CMakeLists.txt<可以创建/code> 文件，然后使用 add_subdirectory() 拉入.实现此目的的一种可能方法是使用 execute_process() 从主构建运行嵌套的 cmake-and-build.嵌套的 cmake-and-build 将使用与顶级 CMake 运行完全相同的源代码和二进制目录(除非交叉编译).主要顶级 CMakeLists.txt 的一般结构将是这样的:

The crux of the problem is needing mycomp to be available when CMake is run so that the generated CMakeLists.txt files can be created and then pulled in with add_subdirectory(). A possible way to achieve this is to use execute_process() to run a nested cmake-and-build from the main build. That nested cmake-and-build would use the exact same source and binary directories as the top level CMake run (unless cross compiling). The general structure of the main top level CMakeLists.txt would be something like this:

# Usual CMakeLists.txt setup stuff goes here...

if(EARLY_BUILD)
    # This is the nested build and we will only be asked to
    # build the mycomp target (see (c) below)
    add_subdirectory(mycompdir)

    # End immediately, we don't want anything else in the nested build
    return()
endif()

# This is the main build, setup and execute the nested build
# to ensure the mycomp executable exists before continuing

# (a) When cross compiling, we cannot re-use the same binary dir
#     because the host and target are different architectures
if(CMAKE_CROSSCOMPILING)
    set(workdir "${CMAKE_BINARY_DIR}/host")
    execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory "${workdir}")
else()
    set(workdir "${CMAKE_BINARY_DIR}")
endif()

# (b) Nested CMake run. May need more -D... options than shown here.
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}"
                        -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
                        -DCMAKE_MAKE_PROGRAM=${CMAKE_MAKE_PROGRAM}
                        -DEARLY_BUILD=ON
                        ${CMAKE_SOURCE_DIR}
               WORKING_DIRECTORY "${workdir}")

# (c) Build just mycomp in the nested build. Don't specify a --config
#     because we cannot know what config the developer will be using
#     at this point. For non-multi-config generators, we've already
#     specified CMAKE_BUILD_TYPE above in (b).
execute_process(COMMAND ${CMAKE_COMMAND} --build . --target mycomp
                WORKING_DIRECTORY "${workdir}")

# (d) We want everything from mycompdir in our main build,
#     not just the mycomp target
add_subdirectory(mycompdir)

# (e) Run mycomp on the sources to generate a CMakeLists.txt in the
#     ${CMAKE_BINARY_DIR}/foobar directory. Note that because we want
#     to support cross compiling, working out the location of the
#     executable is a bit more tricky. We cannot know whether the user
#     wants debug or release build types for multi-config generators
#     so we have to choose one. We cannot query the target properties
#     because they are only known at generate time, which is after here.
#     Best we can do is hardcode some basic logic.
if(MSVC)
    set(mycompsuffix "Debug/mycomp.exe")
elseif(CMAKE_GENERATOR STREQUAL "Xcode")
    set(mycompsuffix "Debug/mycomp")
else()
    set(mycompsuffix "mycomp")
endif()
set(mycomp_EXECUTABLE "${workdir}/mycompdir/${mycompsuffix}")
execute_process(COMMAND "${mycomp_EXECUTABLE}" -outdir foobar ${CMAKE_SOURCE_DIR}/foo.my ${CMAKE_SOURCE_DIR}/bar.my)

# (f) Now pull that generated CMakeLists.txt into the main build.
#     It will create a CMake library target called foobar.
add_subdirectory(${CMAKE_BINARY_DIR}/foobar ${CMAKE_BINARY_DIR}/foobar-build)

# (g) Another target which links to the foobar library
#     and includes headers from there
add_executable(gumby gumby.cpp)
target_link_libraries(gumby PUBLIC foobar)
target_include_directories(gumby PUBLIC foobar)

如果我们在 (b) 和 (c) 处不重复使用与主构建相同的二进制目录，我们最终会构建 mycomp 两次，这显然是我们想要的避免.对于交叉编译，我们无法避免这种情况，因此在这种情况下，我们将 mycomp 工具放在一个单独的二进制目录中.

If we don't re-use the same binary directory at (b) and (c) as we use for the main build, we end up building mycomp twice, which we obviously want to avoid. For cross compiling, we cannot avoid that, so in such cases we build the mycomp tool off to the side in a separate binary directory.

我已经尝试过上述方法，并且确实它似乎在引发原始问题的现实世界项目中工作，至少对于 Unix Makefiles、Ninja、Xcode(OS X 和 iOS)和 Visual Studio 生成器是这样.这种方法的部分吸引力在于它只需要将少量代码添加到顶级 CMakeLists.txt 文件中.尽管如此，还是有一些需要注意的地方:

I've experimented with the above approach and indeed it appears to work in the real world project that prompted the original question, at least for the Unix Makefiles, Ninja, Xcode (OS X and iOS) and Visual Studio generators. Part of the attractiveness of this approach is that it only requires a modest amount of code to be added just to the top level CMakeLists.txt file. Nevertheless, there are some observations that should be made:

• 如果 mycomp 的编译器或链接器命令及其源在嵌套构建和主构建之间有任何不同，mycomp 目标最终会重新构建第二次在 (d).如果没有差异，mycomp 只在不交叉编译时构建一次，这正是我们想要的.
• 我认为没有简单的方法可以将与传递给顶级 CMake 运行的参数完全相同的参数传递给 (b) 处的 CMake 嵌套调用(基本上是描述的问题 此处).读取 CMakeCache.txt 不是一个选项，因为它在第一次调用时不存在，并且无论如何它不会从当前运行中为您提供任何新的或更改的参数.我能做的最好的事情就是设置那些我认为可能会使用的 CMake 变量，这可能会影响 mycomp 的编译器和链接器命令.这可以通过在遇到我发现需要的变量时添加越来越多的变量来解决，但这并不理想.
• 当重新使用相同的二进制目录时，我们依赖 CMake 直到生成阶段(好吧，至少在 (c) 处的构建完成之后)才开始将其任何文件写入二进制目录.对于测试的生成器，看起来我们没问题，但我不知道 all 平台上的 all 生成器是否也遵循此行为(我无法测试每个结合找出来！).这是我最关心的部分.如果有人能够通过推理和/或证据确认这对所有生成器和平台都是安全的，那将是有价值的(如果您想将其作为单独的答案解决，则值得点赞).

• If the compiler or linker commands for mycomp and its sources are different in any way between the nested build and the main build, the mycomp target ends up getting rebuilt a second time at (d). If there are no differences, mycomp only gets built once when not cross compiling, which is exactly what we want.
• I see no easy way to pass exactly the same arguments to the nested invocation of CMake at (b) as was passed to the top level CMake run (basically the problem described here). Reading CMakeCache.txt isn't an option since it won't exist on the first invocation and it would not give you any new or changed arguments from the current run anyway. The best I can do is to set those CMake variables I think are potentially going to be used and which may influence the compiler and linker commands of mycomp. This can be worked around by adding more and more variables as I encounter ones I discover I need, but that's not ideal.
• When re-using the same binary directory, we are relying on CMake not starting to write any of its files to the binary directory until the generate stage (well, at least until after the build at (c) completes). For the generators tested, it appears we are okay, but I don't know if all generators on all platforms follow this behaviour too (and I can't test every single combination to find out!). This is the part that gives me the greatest concern. If anyone can confirm with reasoning and/or evidence that this is safe for all generators and platforms, that would be valuable (and worth an upvote if you want to address this as a separate answer).

更新:在对 CMake 具有不同程度的熟悉程度的员工的一些实际项目中使用上述策略后，可以得出一些观察结果.

UPDATE: After using the above strategy on a number of real world projects with staff of varying levels of familiarity with CMake, some observations can be made.

• 让嵌套构建重复使用与主构建相同的构建目录有时会导致问题.具体来说，如果用户在嵌套构建完成后但在主构建完成之前终止 CMake 运行，CMakeCache.txt 文件将保留 EARLY_BUILD 设置为 ON.这使得所有后续的 CMake 运行就像嵌套构建一样，因此在手动删除 CMakeCache.txt 文件之前，主构建基本上会丢失.项目的 CMakeLists.txt 文件之一中的某个错误也可能导致类似的情况(未经证实).尽管没有出现此类问题，但在其自己单独的构建目录中执行嵌套构建的效果非常好.

• Having the nested build re-use the same build directory as the main build can occasionally lead to problems. Specifically, if a user kills the CMake run after the nested build completes but before the main build does, the CMakeCache.txt file is left with EARLY_BUILD set to ON. This then makes all subsequent CMake runs act like a nested build, so the main build is essentially lost until the CMakeCache.txt file is manually removed. It is possible that an error somewhere in one of the project's CMakeLists.txt file may also lead to a similar situation (unconfirmed). Performing the nested build off to the side in its own separate build directory has worked very well though with no such problems.

嵌套构建可能应该是 Release 而不是 Debug.如果不重新使用与主构建相同的构建目录(现在我推荐的)，我们不再关心试图避免编译相同的文件两次，因此不妨将 mycomp 设为尽快.

The nested build should probably be Release rather than Debug. If not re-using the same build directory as the main build (now what I'd recommend), we no longer care about trying to avoid compiling the same file twice, so may as well make mycomp as fast as possible.

使用 ccache 以便任何费用由于使用不同的设置两次重建某些文件被最小化.实际上，我们发现使用 ccache 通常会使嵌套构建非常快，因为与主构建相比，它很少发生变化.

Use ccache so that any costs due to rebuilding some files twice with different settings are minimised. Actually, we found using ccache typically makes the nested build very quick since it rarely changed compared to the main build.

嵌套构建可能需要在某些平台上将 CMAKE_BUILD_WITH_INSTALL_RPATH 设置为 FALSE 以便可以找到 mycomp 需要的任何库无需设置环境变量等

The nested build probably needs to have CMAKE_BUILD_WITH_INSTALL_RPATH set to FALSE on some platforms so that any libraries mycomp needs can be found without having to set environment variables, etc.

这篇关于立即构建工具，以便稍后在同一 CMake 运行中使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！