在Ninja中使用ExternalProject下载步骤 [英] Using an ExternalProject download step with Ninja

问题描述

这似乎是一个常见问题，没有明确的答案。

This seems to be a common problem without a clear answer.

情况是：我们有一个第三方依赖关系，我们希望在构建依赖它的目标时在构建时安装 。大致是这样：

The situation is: we have a 3rd party dependency that we want to install at build time when building a target that depends on it. That's roughly:

ExternalProject_Add(target-ep
    DOWNLOAD_COMMAND <whatever>
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND "")

add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES /path/to/install/include
    IMPORTED_LOCATION /path/to/install/lib/libwhatever.a)

add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)

（由于 cmake问题15052 ）

使用Unix Makefile时作为发电机，这很棒。

When using Unix Makefiles as the generator, this works great. Only installs dependencies on demand, all the builds work correctly.

但是，在Ninja上，此操作立即失败，并显示以下内容：

However, on Ninja, this fails immediately with something like:

ninja: error: '/path/to/install/lib/libwhatever.a', needed by 'something', missing and no known rule to make it

这是因为Ninja扫描依赖关系与Make的方式有所不同（请参见忍者问题760 ）。因此，我们要做的实际上是告诉Ninja该外部依赖关系存在。我们可以这样做：

This is because Ninja scans dependencies differently from Make (see ninja issue 760). So what we have to do is actually tell Ninja that this external dependency exists. We can do that:

ExternalProject_Add(target-ep
    DOWNLOAD_COMMAND <whatever>
    BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND "")

不幸的是也失败了：

No build step for 'target-ep'ninja: error: mkdir(/path/to/install): Permission denied

这是因为我的下载步骤具有写入该路径的权限，但是底层 add_custom_command（）所运行的任何 mkdir 命令 ExternalProject_Add（）不会。

This is because my download step has permissions to write to that path, but whatever mkdir command is being run by the underlying add_custom_command() from with ExternalProject_Add() does not.

所以：

1. 忍者和CMake可能完全 吗？ （版本不是问题，如果可以解决问题，我可以使用最新的CMake）


2. 如果有某种方法可以明确列出 BUILD_BYPRODUCTS ，有没有一种方法可以简单地传达将要安装的整个目录是副产品？也就是说， / path / to / install / * 是副产品吗？

1. Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
2. If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?

推荐答案

ExternalProject 的隐藏的 mkdir 步骤（所有其他步骤直接或间接依赖）总是尝试创建完整的目录集，即使它们不会被使用。您可以在此处。作为参考，它执行以下操作：

The hidden mkdir step of ExternalProject (which all other steps directly or indirectly depend on) always tries to create the full set of directories, even if they won't be used. You can see this here. For reference, it does this:

ExternalProject_Add_Step(${name} mkdir
  COMMENT "Creating directories for '${name}'"
  COMMAND ${CMAKE_COMMAND} -E make_directory ${source_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${binary_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${install_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${tmp_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${stamp_dir}${cfgdir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${download_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${log_dir}   # This one only since CMake 3.13
  )

Unix系统上的默认安装位置可能是 / usr / local ，因此，如果您没有写权限到它尝试建立的所有目录，则可能与您的问题有关。我建议您检查每个位置的权限，并确保它们已经存在或可写。另外，您可以指定一个在构建树本地的安装目录，这样即使不使用它，也至少可以始终创建它（请参见下面的示例）。

The default install location on Unix systems is probably going to be /usr/local, so if you don't have write permissions to all of the directories it tries to make, then that may be related to your problem. I suggest you check the permissions of each of these locations and make sure they either already exist or are writable. Alternatively, you could specify an install directory that is local to the build tree so that even though it won't be used, it can at least always be created (see example further below).

如果使用忍者，它的依赖性检查要比make更为严格。您有 target-ep 做提供 libwhatever.a 的下载，因此您确实需要 BUILD_BYPRODUCTS 告诉忍者 target-ep 是创建该文件的原因。如您所知，如果不这样做， target-imp 会指向最初不存在的库，Ninja会正确地抱怨它丢失了并且它不知道如何创建它。如果您提供 BUILD_BYPRODUCTS ，则有意义的是构建步骤不应为空，因此，即使它只是一个 BUILD_COMMAND 实际上并没有做任何有意义的事情。

If you use Ninja, it will be more rigorous in its dependency checking than make. You have target-ep doing the download that provides libwhatever.a, so you do need BUILD_BYPRODUCTS to tell Ninja that target-ep is what creates that file. As you've found out, if you don't then target-imp will point at a library that won't initially exist and Ninja rightly complains that it is missing and it doesn't know how to create it. If you provide BUILD_BYPRODUCTS, it makes sense that the build step shouldn't be empty, so you probably need to do something as a build step, even if it is just a BUILD_COMMAND that doesn't actually do anything meaningful.

对 target- ep 希望可以为您工作：

ExternalProject_Add(target-ep
    INSTALL_DIR ${CMAKE_CURRENT_BUILD_DIR}/dummyInstall
    DOWNLOAD_COMMAND <whatever>
    BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
    BUILD_COMMAND ${CMAKE_COMMAND} -E echo_append
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND "")

您原来的问题也导致了对错误目标的依赖。 target-imp 应该取决于 target-ep ，但是您有 target 取决于 target-ep 。正确的依赖关系可以这样表示：

Your original question also creates a dependency on the wrong target. target-imp should depend on target-ep, but you had target depend on target-ep instead. The correct dependency can be expressed by this:

 add_dependencies(target-imp target-ep)

通过 BUILD_BYPRODUCTS 选项，Ninja已经知道上述依赖性，但是需要

With the BUILD_BYPRODUCTS option, Ninja already knows the above dependency, but it is needed for other generators, including make.

您还没有指定<随便什么> 下载命令的用途，但是我假设它负责确保该库执行后将存在于 /path/to/install/lib/libwhatever.a 中。您也可以尝试将 DOWNLOAD_COMMAND 设为空，然后将< whatever> 放入 BUILD_COMMAND 。

You haven't specified what your <whatever> download command does, but I'm assuming it is responsible for ensuring that the library will exist at /path/to/install/lib/libwhatever.a when it has executed. You could also try making the DOWNLOAD_COMMAND empty and putting <whatever> as the BUILD_COMMAND instead.

要解决您的特定问题：

1. 忍者和CMake完全有可能吗？ （版本不是问题，如果可以解决问题，我可以使用最新的CMake）

是的，我验证了上述方法可与Ninja 1.8.2配合使用CMake 3.11.0在macOS上进行虚拟测试项目。我希望它可以与CMake 3.2或更高版本一起使用（那是在添加了对 BUILD_BYPRODUCTS 选项的支持时）。

Yes, I verified that the approach mentioned above works with Ninja 1.8.2 for a dummy test project on macOS using CMake 3.11.0. I would expect it to work with CMake 3.2 or later (that's when support for the BUILD_BYPRODUCTS option was added).

1. 如果有某种方法可以明确列出BUILD_BYPRODUCTS，有一种方法可以简单地传达将要安装的整个目录是副产品？也就是说，/ path / to / install / *是副产品吗？

不太可能。忍者将如何知道这样的目录中应该包含什么内容？获得可靠依赖关系的唯一方法是显式列出预期存在的每个文件，在您的情况下，您可以使用 BUILD_BYPRODUCTS 来完成。

Unlikely. How would Ninja know what is expected to be in such a directory? The only way to get reliable dependencies would be to explicitly list each file that was expected to be there, which you do using BUILD_BYPRODUCTS in your case.

这篇关于在Ninja中使用ExternalProject下载步骤的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！