使用 Cython 将 Python 代码编译为静态链接的可执行文件 [英] Compile Python code to statically linked executable with Cython

问题描述

我有一个纯 Python 脚本，我想将它分发到具有未知 Python 配置的系统.因此，我想将 Python 代码编译为独立的可执行文件.

我运行 cython --embed ./foo.py 没有给出 foo.c 的问题.然后，我跑

gcc $(python3-config --cflags) $(python3-config --ldflags) ./foo.c

其中 python3-config --cflags 给出

-I/usr/include/python3.5m -I/usr/include/python3.5m -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.5-MLq5fN/python3.5-3.5.3=.-fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

和 python3-config --ldflags 给出

-L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -L/usr/lib -lpython3.5m -lpthread -ldl -lutil -lm -Xlinker -export-动态 -Wl,-O1 -Wl,-B 符号函数

这样我获得了一个运行没有问题的动态链接的可执行文件.ldd a.out 产生

 linux-vdso.so.1 (0x00007ffcd57fd000)libpython3.5m.so.1.0 =>/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007fda76823000)libpthread.so.0 =>/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fda76603000)libdl.so.2 =>/lib/x86_64-linux-gnu/libdl.so.2 (0x00007fda763fb000)libutil.so.1 =>/lib/x86_64-linux-gnu/libutil.so.1 (0x00007fda761f3000)libm.so.6 =>/lib/x86_64-linux-gnu/libm.so.6 (0x00007fda75eeb000)libc.so.6 =>/lib/x86_64-linux-gnu/libc.so.6 (0x00007fda75b4b000)libexpat.so.1 =>/lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fda7591b000)libz.so.1 =>/lib/x86_64-linux-gnu/libz.so.1 (0x00007fda756fb000)/lib64/ld-linux-x86-64.so.2 (0x00007fda77103000)

现在，我尝试将选项 -static 添加到 gcc，但这会导致错误:

/usr/bin/ld: 动态 STT_GNU_IFUNC 符号 `strcmp' 在 `/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-制作可执行文件时不能使用 linux-gnu/libc.a(strcmp.o)'；使用 -fPIE 重新编译并使用 -pie 重新链接collect2:错误:ld 返回 1 个退出状态

我检查了 ldd 提供的所有共享库也安装为静态库.

那么，这是否与 python3-config 提供的选项有些不兼容?

解决方案

遇到的问题很明显是来自于链接器(gcc在后台启动了一个链接器，看看——只要用-v启动gcc - 在详细模式下).因此，让我们从一个简短的提示开始，链接过程是如何工作的:

链接器保留它需要解析的所有符号的名称.一开始它只是符号main.当链接器检查库时会发生什么?

1. 如果是静态库，链接器会查看这个库中的每个目标文件，如果这个目标文件定义了一些查找符号，则包含整个目标文件(这意味着一些符号被解析，但是可以添加更多新的未解析符号).链接器可能需要多次传递静态库.

2. 如果它是一个共享库，它被链接器视为一个由单个巨大目标文件组成的库(毕竟，我们必须在运行时加载这个库，而不必一遍又一遍地传递多次以修剪未使用的符号):如果至少有一个需要的符号，则整个库是链接"的.(实际上并不是在运行时发生链接，这是一种试运行)，如果不是 - 整个库将被丢弃并且不再查看.

例如，如果您链​​接到:

gcc -L/path -lpython3.x <其他库>foo.o

你会遇到一个问题，不管 python3.x 是共享库还是静态库:当链接器看到它时，它只查找符号 main, 但是这个符号没有在 python-lib 中定义，所以它 python-lib 被丢弃并且再也没有看过.只有当链接器看到目标文件 foo.o 时，它才意识到需要整个 Python-Symbols，但现在已经太迟了.

有一个简单的规则来处理这个问题:把目标文件放在第一位！这意味着:

gcc -L/path foo.o -lpython3.x <其他库>

现在，当链接器第一次看到 python-lib 时，它就知道它需要什么.

还有其他方法可以达到类似的效果.

A) 让链接器重复一组档案，只要每次扫描至少添加一个新符号定义:

gcc -L/path --Wl,-start-group -lpython3.x <其他库>foo.o -Wl,-end-group

Linker-options -Wl,-start-group 和 -Wl,-end-group 表示链接器对这组档案进行多次迭代，因此链接器有第二次机会(或更多)包含符号.此选项可能会导致更长的链接时间.

B) 打开选项 --no-as-needed 将导致链接到一个共享库(并且只有共享库)，无论在这个库中是否需要定义的符号或不是.

gcc -L/path -Wl,-no-as-needed -lpython3.x -Wl,-as-needed <其他库>foo.o

其实默认的 ld-behavior 是 --no-as-needed，但是 gcc-frontend 调用 ld 时带有选项 --as-needed，所以我们可以通过在 python-library 之前添加 -no-as-needed 来恢复行为，然后再次将其关闭.

---

现在你的静态链接问题.我认为不建议使用所有标准库的静态版本(都在 glibc 之上)，你应该做的可能是只静态链接 python-library.

链接的规则很简单:默认情况下，链接器会先尝试打开库的共享版本，然后再打开静态版本.IE.对于库 libmylib 和路径 A 和 B，即

 -L/A -L/B lmylib

它尝试按以下顺序打开库:

A/libmylib.soA/libmylib.aB/libmylib.soB/libmylib.a

所以如果A文件夹只有一个静态版本，那么就使用这个静态版本(不管B文件夹是否有共享版本).p>

因为真正使用哪个库是非常不透明的 - 这取决于您的系统设置，通常会通过 -Wl,-verbose 打开链接器的日志记录以进行故障排除.

通过使用选项 -Bstatic 可以强制使用库的静态版本:

gcc foo.o -L/path -Wl,-Bstatic -lpython3.x -Wl,-Bdynamic <其他库>-Wl,-verbose -o foo

值得注意的事情:

1. foo.o 链接在库之前.
2. 在 python 库之后直接关闭静态模式，以便动态链接其他库.

现在:

 gcc <cflags>L/paths foo.c -Wl,-Bstatic -lpython3.X -Wl,-Bdynamic <其他库>-o foo -Wl,-verbose...尝试打开路径/libpython3.6m.a 成功...ldd foo 显示不依赖于 python-lib./foo有用！

---

是的，如果您链​​接静态 glibc(我不推荐)，您需要从命令行中删除 -Xlinker -export-dynamic.

没有-Xlinker -export-dynamic 编译的可执行文件将无法加载依赖于使用ldopen<加载到的可执行文件的这个属性的一些c-extension/代码>.

---

隐式 -pie 选项可能导致的问题.

最新版本的 gcc 默认使用 pie-option 构建.通常/有时，较旧的 python 版本使用较旧的 gcc 版本构建，因此 python-config --cflags 会错过现在必需的 -no-pie，因为它是当时不需要.在这种情况下，链接器将产生如下错误消息:

<块引用>

重定位 R_X86_64_32S 对符号 'XXXXX' 不能使用时制作一个 PIE 对象；用 -fPIC 重新编译

在这种情况下，-no-pie 选项应该添加到 <cflags>.

I have a pure Python script that I would like to distribute to systems with unkown Python configuration. Therefore, I would like to compile the Python code to a stand-alone executable.

I run cython --embed ./foo.py without problems giving foo.c. Then, I run

gcc $(python3-config --cflags) $(python3-config --ldflags) ./foo.c

where python3-config --cflags gives

-I/usr/include/python3.5m -I/usr/include/python3.5m  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.5-MLq5fN/python3.5-3.5.3=. -fstack-protector-strong -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

and python3-config --ldflags gives

-L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -L/usr/lib -lpython3.5m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

This way I obtain a dynamically linked executable that runs without a problem. ldd a.out yields

 linux-vdso.so.1 (0x00007ffcd57fd000)
 libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007fda76823000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fda76603000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fda763fb000)
 libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fda761f3000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fda75eeb000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fda75b4b000)
 libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fda7591b000)
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fda756fb000)
 /lib64/ld-linux-x86-64.so.2 (0x00007fda77103000)

Now, I try to add the option -static to gcc, but this results in an error:

/usr/bin/ld: dynamic STT_GNU_IFUNC symbol `strcmp' with pointer equality in `/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(strcmp.o)' can not be used when making an executable; recompile with -fPIE and relink with -pie
collect2: error: ld returned 1 exit status

I checked that all shared libraries given by ldd are also installed as static libraries.

So, is this some incompatibility with the options given by python3-config?

解决方案

The experienced problems are obviously from the linker (gcc started a linker under the hood, to see it - just start gcc with -v - in verbose mode). So let's start with a short reminder how the linkage process works:

The linker keeps the names of all symbols it needs to resolve. In the beginning it is only the symbol main. What happens, when linker inspects a library?

1. If it is a static library, the linker looks at every object file in this library, and if this object files defines some looked for symbols, the whole object file is included (which means some symbols becomes resolved, but some further new unresolved symbols can be added). Linker might need to pass multiple times over a static library.

2. If it is a shared library, it is viewed by the linker as a library consisting out of a single huge object file (after all, we have to load this library at the run time and don't have to pass multiple times over and over to prune unused symbols): If there is at least one needed symbol the whole library is "linked" (not really the linkage happens at the run-time, this is a kind of a dry-run), if not - the whole library is discarded and never looked again at.

For example if you link with:

gcc -L/path -lpython3.x <other libs> foo.o 

you will get a problem, no matter whether python3.x is a shared or a static lib: when the linker sees it, it looks only for the symbol main, but this symbol is not defined in the python-lib, so it the python-lib is discarded and never looked again at. Only when the linker sees the object-file foo.o, it realizes, that the whole Python-Symbols are needed, but now it is already too late.

There is a simple rule to handle this problem: put the object files first! That means:

gcc -L/path  foo.o -lpython3.x <other libs> 

Now the linker knows what it needs from the python-lib, when it first sees it.

There are other ways to achieve a similar result.

A) Let the linker to reiterate a group of archives as long as at least one new symbol definition was added per sweep:

gcc -L/path --Wl,-start-group -lpython3.x <other libs> foo.o -Wl,-end-group

Linker-options -Wl,-start-group and -Wl,-end-group says to linker iterate more than once over this group of archives, so the linker has a second chance (or more) to include symbols. This option can lead to longer linkage time.

B) Switching on the option --no-as-needed will lead to a shared library (and only shared library) being linked in, no matter whether in this library defined symbols are needed or not.

gcc -L/path -Wl,-no-as-needed -lpython3.x -Wl,-as-needed <other libs> foo.o

Actually, the default ld-behavior is --no-as-needed, but the gcc-frontend calls ld with option --as-needed, so we can restore the behavior by adding -no-as-needed prior to the python-library and then switch it off again.

---

Now to your problem of statical linking. I don't think it is advisable to use static versions of all standard libraries (all above glibc), what you should probably do is to link only the python-library statically.

The rules of the linkage are simple: per default the linker tries to open a shared version of the library first and than the static version. I.e. for the library libmylib and paths A and B, i.e.

 -L/A -L/B lmylib

it tries to open libraries in the following order:

A/libmylib.so
A/libmylib.a
B/libmylib.so
B/libmylib.a

Thus if the folder A has only a static version, so this static version is used (no matter whether there is a shared version in folder B).

Because it is quite opaque which library is really used - it depends on the setup of your system, usually one would switch on the logging of the linker via -Wl,-verbose to trouble-shoot.

By using the option -Bstatic one can enforce the usage of the static version of a library:

gcc  foo.o -L/path -Wl,-Bstatic -lpython3.x -Wl,-Bdynamic <other libs>  -Wl,-verbose -o foo

Notable thing:

1. foo.o is linked before the libraries.
2. switch the static-mode off, directly after the python-library, so other libraries are linked dynamically.

And now:

 gcc <cflags> L/paths foo.c -Wl,-Bstatic -lpython3.X -Wl,-Bdynamic <other libs> -o foo -Wl,-verbose
...
attempt to open path/libpython3.6m.a succeeded
...
ldd foo shows no dependency on python-lib
./foo
It works!

---

And yes, if you link against static glibc (I don't recommend), you will need to delete -Xlinker -export-dynamic from the command line.

The executable compiled without -Xlinker -export-dynamic will not be able to load some of c-extension which depend on this property of the executable to which they are loaded with ldopen.

---

Possible issues due to implicit -pie option.

Recent versions of gcc build with pie-option per default. Often/sometimes, older python versions where build with an older gcc-version, thus python-config --cflags would miss the now necessary -no-pie, as it was not needed back then. In this case the linker will produce an error message like:

relocation R_X86_64_32S against symbol `XXXXX' can not be used when making a PIE object; recompile with -fPIC

In this case, -no-pie option should be added to <cflags>.

这篇关于使用 Cython 将 Python 代码编译为静态链接的可执行文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！