cmake-链接静态库pytorch在构建过程中找不到其内部功能 [英] cmake - linking static library pytorch cannot find its internal functions during build

问题描述

我正在尝试使用cmake构建程序。出于几个原因，该程序必须使用静态库而不是动态库来构建，而我需要使用PyTorch，所以这就是我要做的事情：

I'm trying to build a program using cmake. For several reasons, the program must be built using static libraries rather than dynamic libraries, and I need to use PyTorch so this is what I've done:

1. 下载并安装了PyTorch静态库（我在正确的路径中 / home / me / pytorch /中找到了 libtorch.a 火炬/ lib ）


2. 制作 CMakeLists.txt 并包含以下内容：

1. Downloaded and installed PyTorch static library (I've found libtorch.a in the proper path, in /home/me/pytorch/torch/lib)
2. Made CMakeLists.txt with the following contents:

cmake_minimum_required(VERSION 3.5.1 FATAL_ERROR)
project(example-app LANGUAGES CXX)
find_package(Torch REQUIRED)
add_executable(example-app example-app.cpp argparse/argparse.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}" -static -fopenmp)
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

FYI， example-app。 cpp 是具有主要功能的文件，而 argparse / 是一个目录，其中包含一些在 example中调用的函数的源代码-app.cpp

FYI, example-app.cpp is the file with the main function, and argparse/ is a directory with some source code for functions called in example-app.cpp

它可以工作到 cmake -DCMAKE_PREFIX_PATH = / home / me / pytorch / torch .. ，但以下 build 会引起一些错误，表示找不到对某些功能的引用，即以 fbgemm开头的功能：: 。 fbgemm （据我所知）是用于实现PyTorch的某种GEMM库。

It works until cmake -DCMAKE_PREFIX_PATH=/home/me/pytorch/torch .., but the following build incurs some errors, saying it could not find the reference to some functions, namely functions starting with fbgemm::. fbgemm is (as long as I know) some sort of GEMM library used in implementing PyTorch.

我在链接静态PyTorch库时，其内部库（例如 fbgemm 的东西）未正确链接，但是我不是 cmake <的专家/ code>，说实话也不完全确定。

It seems to me that while linking the static PyTorch library, its internal libraries like fbgemm stuff have not been linked properly, but I'm not an expert on cmake and honestly not entirely sure.

我做错了什么，还是有解决此问题的方法？

Am I doing something wrong, or is there a workaround for this problem? Any help or push in the right direction would be greatly appreciated.

PS

1. 确切的错误由于过长而未发布，但主要由对〜错误的未定义引用组成。如果查看错误消息可能对某些人有帮助，我很乐意编辑问题并将其发布。

1. The exact error has not been posted because it is way too long, but it consists of mostly undefined reference to ~ errors. If looking at the error message might be helpful for some people, I'd be happy to edit the question and post it.

如果我从代码中删除了需要库功能的部分，而没有注释掉 #include< torch / torch.h> 来自 example-app.cpp 。

推荐答案

最近通过静态链接PyTorch经历了类似的过程，说实话这还不太好。

Lately went through similar process with static linking of PyTorch and to be honest it wasn't too pretty.

我将概述我已执行的步骤（您可以在 torchlambda 中找到确切的源代码， 此处是 CMakeLists.txt （还包括AWS开发工具包和AWS Lambda静态构建），此处是一个脚本构建 pytorch 通过仅支持CPU的 /scripts/build_mobile.sh 进行克隆和构建）），
尽管仅支持CPU（尽管如果需要，类似的步骤应该没问题） CUDA，至少可以帮助您入门。）

I will outline the steps I have undertaken (you can find exact source code in torchlambda, here is CMakeLists.txt (it also includes AWS SDK and AWS Lambda static builds), here is a script building pytorch from source ( cloning and building via /scripts/build_mobile.sh with only CPU support)), though it's only with CPU support (though similar steps should be fine if you need CUDA, it will get you started at least).

首先，您需要预先构建的静态库文件（所有都需要是静态的，因此没有 .so ，只有扩展名为 .a 的用户才适用）。

First of all, you need pre-built static library files (all of them need to be static, hence no .so, only those with .a extension are suitable).

Tbh I一直在上找到 PyTorch 提供的内容。安装页面，但只有共享版本。
在一个GitHub问题中，我找到了一种下载方法，如下所示：

Tbh I've been looking for those provided by PyTorch on installation page, yet there is only shared version. In one GitHub issue I've found a way to download them as follows:

而不是下载（此处通过 wget ）共享库：

Instead of downloading (here via wget) shared libraries:

$ wget https://download.pytorch.org/libtorch/cu101/libtorch-shared-with-deps-1.4.0.zip

您重命名 shared 到静态（如在此问题中），因此它将变为：

you rename shared to static (as described in this issue), so it would become:

$ wget https://download.pytorch.org/libtorch/cu101/libtorch-static-with-deps-1.4.0.zip

但是，当您下载它时，在 lib 文件夹下没有 libtorch.a （找不到 libcaffe2.a ，如此问题），那么剩下的就是从源代码显式构建。

Yet, when you download it there is no libtorch.a under lib folder (didn't find libcaffe2.a either as indicated by this issue), so what I was left with was building explicitly from source.

如果您以某种方式拥有这些文件（如果有的话，请提供您从何处获取文件），您可以跳过下一步。

If you have those files somehow (if so, please provide where you got them from please), you can skip the next step.

对于CPU版本，我使用了 / pytorch / scripts / build_mobile.sh 文件，如果需要GPU支持，则可以基于此版本（也许您只需要通过 -DUSE_CUDA = ON 这个脚本，虽然不确定）。

For CPU version I have used /pytorch/scripts/build_mobile.sh file, you can base your version off of this if GPU support is needed (maybe you only have to pass -DUSE_CUDA=ON to this script, not sure though).

最重要的是 cmake 的 -DBUILD_SHARED_LIBS = OFF 以便将所有内容构建为 static 库。您也可以从我的工具中检查脚本 build_mobile.sh 的参数。

Most important is cmake's -DBUILD_SHARED_LIBS=OFF in order to build everything as static library. You can also check script from my tool which passes arguments to build_mobile.sh as well.

在上面运行将为您提供中的静态文件/ pytorch / build_mobile / install 默认为您需要的一切。

Running above will give you static files in /pytorch/build_mobile/install by default where there is everything you need.

现在您可以将上述构建文件复制到 / usr / local （最好不要复制，除非您使用的是 Docker 作为 torchlambda ），或从您的 CMakeLists.txt 中设置路径，如下所示：

Now you can copy above build files to /usr/local (better not to unless you are using Docker as torchlambda) or set path to it from within your CMakeLists.txt like this:

set(LIBTORCH "/path/to/pytorch/build_mobile/install")

# Below will append libtorch to path so CMake can see files
set(CMAKE_PREFIX_PATH "${CMAKE_PREFIX_PATH};${LIBTORCH}")

其余的都很好除了 target_link_libraries ，（如此问题所示，请参见此处列出的相关问题（供其他参考）与 -Wl，-完整存档链接器标志一起使用，这使我想到了：

Now the rest is fine except target_link_libraries, which should be (as indicated by this issue, see related issues listed there for additional reference) used with -Wl,--whole-archive linker flag, which brought me to this:

target_link_libraries(example-app PRIVATE -lm
        -Wl,--whole-archive "${TORCH_LIBRARIES}"
        -Wl,--no-whole-archive
        -lpthread
        ${CMAKE_DL_LIBS})

您可能不需要 -lm ， -lpthread 或 $ {CMAKE_DL_LIBS} ，尽管在 Amazon Linux AMI 上构建时需要它。

You may not need either of -lm, -lpthread or ${CMAKE_DL_LIBS}, though I needed it when building on Amazon Linux AMI.

现在您可以开始构建应用程序了。标准的 libtorch 方法应该可以，但是这是我使用的另一条命令：

Now you are off to building your application. Standard libtorch way should be fine but here is another command I used:

mkdir build && \
  cd build &&  \
  cmake .. && \
  cmake --build . --config Release

以上将创建 build 现在应该安全地放置 example-app 二进制文件的文件夹。

Above will create build folder where example-app binary should be now safely located.

最后使用 ld构建/ example-app 来验证 PyTorch 中的所有内容都是静态链接的，请参见前面提到的问题点 5。，您的输出应类似于。

Finally use ld build/example-app to verify everything from PyTorch was statically linked, see aforementioned issue point 5., your output should look similar.

这篇关于cmake-链接静态库pytorch在构建过程中找不到其内部功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！