使用Cython将Python代码编译为静态链接的可执行文件 [英] Compile Python code to statically linked executable with Cython

问题描述

我有一个纯Python脚本，我想分发给具有未知Python配置的系统。因此，我想将Python代码编译为独立的可执行文件。

我运行 cython --embed ./foo.py 给出 foo.c 没有问题。然后，我运行

  gcc $（python3-config --cflags）$（python3-config --ldflags）./foo .c 
  

其中 python3-config --cflags 给

  -I / usr / include / python3.5m -I / usr / include / python3.5m -Wno-unused -result -Wsign-compare -g -fdebug-prefix-map = / build / python3.5-MLq5fN / python3.5-3.5.3 =。 -fstack-protector-strong -Wformat -Werror = format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes 
  

和 python3-config --ldflags 给出

  -L / usr / lib / python3.5 / config-3.5m-x86_64-linux-gnu -L / usr / lib -lpython3.5m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl， -O1 -Wl，-Bsymbolic函数
  

这样，我获得了一个动态链接的可执行文件，该可执行文件无需运行问题。 ldd a.out 收益率

  linux-vdso.so.1（ 0x00007ffcd57fd000）
 libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0（0x00007fda76823000）
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0（0x00007fda76603000）
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2（0x00007fda763fb000）
 libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1（0x00007fda761f3000）
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6（0x00007fda75eeb000）
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6（0x00007fda75b4b000）
 libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1（0x00007fda7591b000）
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1（0x00007fda756fb000）
 /lib64/ld-linux-x86-64.so.2（0x00007fda77103000）
  

现在，我尝试将选项 -static 添加到gcc，但这会导致错误：

  / usr / bin / ld：动态STT_GNU_IFUNC符号`strcmp'，在/ usr / lib / gcc / x86_64中具有指针相等性-linux-gnu / 6 /../../../ x86_64-linux-gnu / libc.a（strcmp.o）'在制作可执行文件时不能使用；用-fPIE重新编译并用-pie 
 collect2重新链接：错误：ld返回1退出状态
  

我检查了ldd给定的所有共享库是否也已安装为静态库。

那么，这与python3-config提供的选项有些不兼容吗？

解决方案

遇到的问题显然来自链接程序（gcc在后台启动了一个链接程序，要看到它-只需以 -v 启动gcc-以详细模式） 。因此，让我们首先简短地提醒一下链接过程的工作原理：

链接器保留需要解析的所有符号的名称。一开始只是符号 main 。链接器检查库时会发生什么情况？

1. 如果它是静态库，则链接器会查看此库中的每个目标文件库，并且如果此目标文件定义了一些查找的符号，则将包括整个目标文件（这意味着某些符号已解析，但可以添加一些其他新的未解析符号）。链接器可能需要在静态库中传递多次。




2. 如果它是共享库，则链接器将其视为由以下内容组成的库：一个巨大的目标文件（毕竟，我们必须在运行时加载该库，而不必一次又一次地遍历以修剪未使用的符号）：如果存在至少一个所需的符号，则整个库都已链接（实际上链接不是在运行时发生的，这是一种试运行），否则-整个库将被丢弃，再也不会看了。

例如，如果您链​​接：

  gcc -L /路径-lpython3.x< other libs> foo.o 
  

无论是否 python3.x，您都会遇到问题是共享库或静态库：当链接器看到它时，它仅查找符号 main ，但该符号未在python-lib，因此将python-lib丢弃并且不再关注。仅当链接器看到目标文件 foo.o 时，它才意识到需要整个Python符号，但现在已经来不及了。

有一个简单的规则可以处理此问题：将目标文件放在第一位！这意味着：

  gcc -L / path foo.o -lpython3.x< other libs> 
  

现在，链接器在第一次看到它时就从python-lib知道了它的需要。

还有其他方法可以达到类似的结果。

A）让链接器将一组档案重申为只要每次扫描至少添加了一个新的符号定义：

  gcc -L / path --Wl，-start-group -lpython3.x< other libs> foo.o -Wl，-end-group 
  

链接器选项 -Wl ，-start-group 和 -Wl，-end-group 表示链接器在这组存档中重复访问多次，因此链接器具有第二次机会（或更多次）包含符号。此选项可能会导致更长的链接时间。

B）开启选项-无需 c不管是否需要在此库中定义符号，都会导致链接共享库（并且只有共享库）。

  gcc -L / path -Wl-不需要-lpython3.x -Wl-需要< other libs> foo.o 
  

实际上，默认的ld行为为-否-根据需要，但是gcc-frontend调用带有选项的ld-根据需要，因此我们可以通过添加<$ c来恢复行为$ c>-无需按需在python库之前，然后再次将其关闭。

---

现在遇到静态链接问题。我认为不建议使用所有标准库的静态版本（在glibc之上），您应该做的只是静态地链接python库。

<链接的规则很简单：默认情况下，链接器会首先尝试打开库的共享版本，而不是打开静态版本。即对于库 libmylib 和路径 A 和 B ，即

  -L / A -L / B lmylib 
  

它尝试按以下顺序打开库：

  A / libmylib。因此
 A / libmylib.a 
 B / libmylib.so 
 B / libmylib.a 
  

因此，如果文件夹 A 仅具有静态版本，则使用此静态版本（无论文件夹<$中是否有共享版本c $ c> B ）。

因为真正使用哪个库是相当不透明的-通常取决于系统的设置可以通过 -Wl，-verbose 打开链接器的日志记录以进行故障排除。

通过使用选项 -Bstatic 可以强制使用库的静态版本：

  gcc foo.o -L / path -Wl，-Bstatic -lpython3.x -Wl，-Bdynamic< other libs> -Wl，-verbose -o foo 
  

值得注意的事情：

1. foo.o 在库之前已链接。


2. 切换静态模式关闭，直接在python库之后，因此动态链接了其他库。

现在：

  gcc< cflags> L /路径foo.c -Wl，-静态-lpython3.X -Wl，-Bdynamic< other libs> -o foo -Wl，-verbose 
 ... 
尝试打开路径/libpython3.6m.a成功
 ... 
 ldd foo显示不依赖python-lib 
 ./foo 
有效！ 
  

---

是的，如果您链​​接静态 glibc （我不建议），您需要从命令行中删除 -Xlinker -export-dynamic 。

在没有 -Xlinker -export-dynamic 的情况下编译的可执行文件将无法加载某些c-extension，具体取决于 ldopen 加载到的可执行文件的此属性上。

I have a pure Python script that I would like to distribute to systems with unkown Python configuration. Therefore, I would like to compile the Python code to a stand-alone executable.

I run cython --embed ./foo.py without problems giving foo.c. Then, I run

gcc $(python3-config --cflags) $(python3-config --ldflags) ./foo.c

where python3-config --cflags gives

-I/usr/include/python3.5m -I/usr/include/python3.5m  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.5-MLq5fN/python3.5-3.5.3=. -fstack-protector-strong -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

and python3-config --ldflags gives

-L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -L/usr/lib -lpython3.5m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

This way I obtain a dynamically linked executable that runs without a problem. ldd a.out yields

 linux-vdso.so.1 (0x00007ffcd57fd000)
 libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007fda76823000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fda76603000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fda763fb000)
 libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fda761f3000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fda75eeb000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fda75b4b000)
 libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fda7591b000)
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fda756fb000)
 /lib64/ld-linux-x86-64.so.2 (0x00007fda77103000)

Now, I try to add the option -static to gcc, but this results in an error:

/usr/bin/ld: dynamic STT_GNU_IFUNC symbol `strcmp' with pointer equality in `/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libc.a(strcmp.o)' can not be used when making an executable; recompile with -fPIE and relink with -pie
collect2: error: ld returned 1 exit status

I checked that all shared libraries given by ldd are also installed as static libraries.

So, is this some incompatibility with the options given by python3-config?

解决方案

The experienced problems are obviously from the linker (gcc started a linker under the hood, to see it - just start gcc with -v - in verbose mode). So let's start with a short reminder how the linkage process works:

The linker keeps the names of all symbols it needs to resolve. In the beginning it is only the symbol main. What happens, when linker inspects a library?

1. If it is a static library, the linker looks at every object file in this library, and if this object files defines some looked for symbols, the whole object file is included (which means some symbols becomes resolved, but some further new unresolved symbols can be added). Linker might need to pass multiple times over a static library.

2. If it is a shared library, it is viewed by the linker as a library consisting out of a single huge object file (after all, we have to load this library at the run time and don't have to pass multiple times over and over to prune unused symbols): If there is at least one needed symbol the whole library is "linked" (not really the linkage happens at the run-time, this is a kind of a dry-run), if not - the whole library is discarded and never looked again at.

For example if you link with:

gcc -L/path -lpython3.x <other libs> foo.o 

you will get a problem, no matter whether python3.x is a shared or a static lib: when the linker sees it, it looks only for the symbol main, but this symbol is not defined in the python-lib, so it the python-lib is discarded and never looked again at. Only when the linker sees the object-file foo.o, it realizes, that the whole Python-Symbols are needed, but now it is already too late.

There is a simple rule to handle this problem: put the object files first! That means:

gcc -L/path  foo.o -lpython3.x <other libs> 

Now the linker knows what it needs from the python-lib, when it first sees it.

There are other ways to achieve a similar result.

A) Let the linker to reiterate a group of archives as long as at least one new symbol definition was added per sweep:

gcc -L/path --Wl,-start-group -lpython3.x <other libs> foo.o -Wl,-end-group

Linker-options -Wl,-start-group and -Wl,-end-group says to linker iterate more than once over this group of archives, so the linker has a second chance (or more) to include symbols. This option can lead to longer linkage time.

B) Switching on the option --no-as-needed will lead to a shared library (and only shared library) being linked in, no matter whether in this library defined symbols are needed or not.

gcc -L/path -Wl,-no-as-needed -lpython3.x -Wl,-as-needed <other libs> foo.o

Actually, the default ld-behavior is --no-as-needed, but the gcc-frontend calls ld with option --as-needed, so we can restore the behavior by adding -no-as-needed prior to the python-library and then switch it off again.

---

Now to your problem of statical linking. I don't think it is advisable to use static versions of all standard libraries (all above glibc), what you should probably do is to link only the python-library statically.

The rules of the linkage are simple: per default the linker tries to open a shared version of the library first and than the static version. I.e. for the library libmylib and paths A and B, i.e.

 -L/A -L/B lmylib

it tries to open libraries in the following order:

A/libmylib.so
A/libmylib.a
B/libmylib.so
B/libmylib.a

Thus if the folder A has only a static version, so this static version is used (no matter whether there is a shared version in folder B).

Because it is quite opaque which library is really used - it depends on the setup of your system, usually one would switch on the logging of the linker via -Wl,-verbose to trouble-shoot.

By using the option -Bstatic one can enforce the usage of the static version of a library:

gcc  foo.o -L/path -Wl,-Bstatic -lpython3.x -Wl,-Bdynamic <other libs>  -Wl,-verbose -o foo

Notable thing:

1. foo.o is linked before the libraries.
2. switch the static-mode off, directly after the python-library, so other libraries are linked dynamically.

And now:

 gcc <cflags> L/paths foo.c -Wl,-Bstatic -lpython3.X -Wl,-Bdynamic <other libs> -o foo -Wl,-verbose
...
attempt to open path/libpython3.6m.a succeeded
...
ldd foo shows no dependency on python-lib
./foo
It works!

---

And yes, if you link against static glibc (I don't recommend), you will need to delete -Xlinker -export-dynamic from the command line.

The executable compiled without -Xlinker -export-dynamic will not be able to load some of c-extension which depend on this property of the executable to which they are loaded with ldopen.

这篇关于使用Cython将Python代码编译为静态链接的可执行文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！