Distutils：构建共享一个方法的多个Python扩展模块（用Swig编写） [英] Distutils: build multiple Python extension modules (written in Swig) that share a method
本文介绍了Distutils：构建共享一个方法的多个Python扩展模块（用Swig编写）的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我有四个C ++文件：Ah，A.cpp，Bh，B.cpp，Ah包含在B.cpp中。
Ah：
#pragma一旦
无效A（）;
A.cpp：
#include< iostream>
void A（）{
std :: cout<< A<< std :: endl;
}
Bh：
#pragma一旦
无效B（）;
B.cpp：
#include Ah
#include< iostream>
无效B（）{
A（）;
std :: cout<< B<< std :: endl;
}
现在我写了两个SWIG接口文件Ai和Bi
Ai：
％module A
％{
#include Ah
％}
％include Ah
Bi：
％module B
％{
#include Bh
％}
％include Bh
setup.py文件为：
来自distutils.core导入设置，扩展名
A_ext =扩展名（ _A，[ Ai， A.cpp，]，swig_opts = ['-c ++']，extra_compile_args = ['-g']）
B_ext =扩展名（ _B，[ Bi， B.cpp，]，swig_opts = ['-c ++'] ，extra_compile_args = ['-g']）
setup（
name = test，
version = 1.0，
ext_modules = [A_ext，B_ext]，
py_modules = [ A， B]
）
如果我输入
python -c'import A;代码库中复制注释），以减少代码，但这并没有真正算作更改）python -c'import A; AA（）'
如果我在下面键入命令，则会出现分段错误：
python -c'导入B; B.B（）’
如何使此命令正确运行？由于我不想多次编译B.cpp，除了下面的方法之外，还有什么办法吗？
B_ext =扩展名（ _B，[ Bi， A.cpp， B.cpp，]，swig_opts = ['-c ++']，extra_compile_args = ['-g']）
解决方案为清晰起见，我对文件进行了一些更改。
ah ：
#pragma一旦
void funcA（）;
a.cpp ：
#include< iostream>
void funcA（）{
std :: cout<< __FILE__<< << __LINE__<< << __FUNCTION__<< std :: endl;
}
ai ：
％module a
％{
#include ah
％}
％include ah
bh ：
#pragma一旦
void funcB（）;
b.cpp ：
#include ah
#include< iostream>
void funcB（）{
std :: cout<< __FILE__<< << __LINE__<< << __FUNCTION__<< std :: endl;
funcA（）;
}
bi ：
％module b
％{
#include bh
％}
％include bh
setup.py ：
从distutils.core导入设置
从distutils.extension导入扩展
a = a
b = b
ext_a = Extension（ _ + a，[a + .i，a + .cpp ]，swig_opts =（-c ++，），extra_compile_args = [-g]）
ext_b = Extension（ _ + b，[b + .i，b + .cpp] ，swig_opts =（-c ++，），extra_compile_args = [-g]）
setup（
name = test，
version = 1.0，
ext_modules = [ext_a，ext_b]，
py_modules = [a，b]
）
调用 b.funcB 时会发生什么事情（简化）（仅 stacktrace 会保留导入）。每个步骤都调用下一个步骤：
- funcB 来自模块 b （ b.py ）
来自模块 _b （ _b.so ）的- funcB _b.cpython-35m-x86_64-linux-gnu.so ）
- 这里所有的事情都在 C中发生（或 C ++ ）
- 当前 funcB 与 b中的一个不同 .cpp ：由 swig 生成，名称为 _wrap_funcB
- 以前的项目符号也适用于 funcA 和 a.cpp
- funcB b.cpp
- funcA 来自 a.cpp
问题在于步骤 ＃4。 中的代码不在模块 _b 中，并且它会在运行时失败。但是事情有点奇怪：调用 funcB 时不会出现故障（核心转储），而是在模块（ b -> _b ）导入时间（我认为这是由于 swig 在幕后魔术所致），如下所示。
输出：
[cfati @ cfati-ubtu16x64-0：〜/ Work / Dev / StackOverflow / q050938128]> 〜/ sopr.sh
***设置较短的提示以使其更好地适合于StackOverflow（或其他）页面时的*** ***
[064bit提示]> ls
a.cpp a.h a.i b.cpp b.h b.i setup.py
[064bit提示]> python3 setup.py build> / dev / null 2>& 1
[064位提示]>回声$？
0
[064位提示] ls
a.cpp a.h a.i a.py a_wrap.cpp b.cpp b.h b.i b.py build b_wrap.cpp setup.py
[064bit提示]> ls ./build/lib.linux-x86_64-3.5
_a.cpython-35m-x86_64-linux-gnu.so _b.cpython-35m-x86_64-linux-gnu.so
[064bit提示] > PYTHONPATH = $ {PYTHONPATH}：./ build / lib.linux-x86_64-3.5 python3 -c import _a; _a.funcA（）
a.cpp 6 funcA
[064bit提示] PYTHONPATH = $ {PYTHONPATH}：./ build / lib.linux-x86_64-3.5 python3 -c import _b
分段错误（核心已转储）
要解决该问题，请：
- 正如您所指出的，在模块 _b 中包含 funcA （通过在 ext_b 的源代码中添加 a.cpp 文件列表）。这样，两个模块都是自包含的（来自 funcA 的 PoV ），每个模块都可以独立工作，而且 funcA 将在他们两个
- 使 _b 依赖于 _a （毕竟，它们是共享对象） 。但这不是 Python 扩展模块的使用方式，它不能在 Win （以及某些 Nix 口味）上使用。因此，这更像是（lame）解决方法（ gainarie ）
- 将 a.cpp 构建到另一个共享库中（ .so ，但不是一个 Python 扩展模块）要被两个模块使用。不用说，它们在运行时每个都需要 .so 出现
选项 ＃3。 是最佳选择。但是 distutils （（ [Python.Docs]： API参考）没有提供所需的功能 OOTB （显然，构建扩展模块和它依赖的外部共享库不是 distutils 旨在），或者至少我找不到任何东西。
有一个 build_clib 模块，该模块提供了构建 static 库（供扩展模块使用）的功能，但与选项 ＃1。 。
setup.py ：
导入系统
从distutils.core导入操作系统
从distutils.extension导入
从distutils.command.build_clib导入扩展
从distutils.command.build_ext导入build_clib
从distutils.ccompiler导入build_ext
从distutils.ccompiler导入CCompiler
__win = sys.platform [：3] .lower（）== win
export_symbols_option = export_symbols
class build_clib_dyn（build_clib）：
def finalize_options（ self）：
self.set_undefined_options（'build'，
（'build_lib'，'build_clib'），
（'build_temp'，'build_temp'），
（'compiler'，'compiler'），
（'debug'，'debug'），
（'force'，'force'））
self.libraries = self.distribution。库
如果self.libraries：
self.check_library_list（self.libraries）
如果self.include_dirs为None：
self.include_dirs = self.distribution.include_dirs或[]
if isinstance（self.include_dirs，str）：
self.include_dirs = self.include_dirs.split（os.pathsep）
def build_libraries（self，库）：
对于库中的（lib_name，build_info）：
sources = build_info.get（'sources'）
如果source为None或not isinstance（sources，（list，tuple））：
提高DistutilsSetupError（
在库选项中（库'％s'），必须存在
sources，并且必须为
a源文件名列表％lib_name）
源=列表（源）
宏= build_info.get（'macros'）
include_dirs = build_info.get（'include_dirs'）
objects = self.compiler.compile（来源，
output_dir = self.build_temp，
宏=宏，
include_dirs = include_dirs，
debug = self.debug）
self.compiler.link（CCompiler.SHARED_OBJECT，objects，self.compiler.library_filename（lib_name，lib_type = shared），
output_dir = self.build_clib，
export_symbols = build_info.get（export_symbols_option），
debug = self.debug）
如果__win：
class build_ext_w_dyn_dep（build_ext）：
def finalize_options（self）：
super（build_ext_w_dyn_dep，self）.finalize_options（）
self.library_dirs.append（os.path.dirname（self.build_temp））
else：
类build_ext_w_dyn_dep（build_ext） ：
pass
a_name = a
b_name = b
common_name = a_name + b_name + common
swig_opts = [ -c ++]
库= [common_name]
lib_common_build_info = { sources：[a_name + .cpp]}
如果__win：
extra_compile_args = None
extra_link_args =无
lib_common_build_info [export_symbols_option] = [ funcA]
其他：
extra_compile_args = [ -g]
extra_link_args = [ -Wl， -rpath，$ {ORIGIN}]
lib_common_info =（公共名称，lib_common_build_info）
ext_a = Extension（ _ + a_name，[a_name + .i]，库=库，extra_compile_args = extra_compile_args，extra_link_args = extra_link_args，swig_opts = swig_opts）
ext_b = Extension（ _ + b_name，[b_name + .i，b_name + .cpp]，库=库，extra_compile_args = extra_compile_args，extra_link_args = extra_link_args，swig_opts = swig_opts）
setup（
name = test，
version = 1.0，
库= lib_common_info]，
cmdclass = { build_clib：build_clib_dyn， build_ext：build_ext_w_dyn_dep}，
ext_modules = [ext_a，ext_b]，
py_modules = [a_name，b_name]
）
注释：
- build_clib_dyn 扩展了 build_clib ，因为其功能必须修改。覆盖了2个方法，但实际上只有一小部分发生了更改（未从基类方法（ Python 3.5.4
输出（再次运行上述命令）：
[064位提示]> ls
a.cpp a.h a.i b.cpp b.h b.i setup.py
[064bit提示]> python3 setup.py build> / dev / null 2>& 1
[064位提示]>回声$？
0
[064位提示] ls
a.cpp a.h a.i a.py a_wrap.cpp b.cpp b.h b.i b.py build b_wrap.cpp setup.py
[064bit提示]> ls build / lib.linux-x86_64-3.5 /
_a.cpython-35m-x86_64-linux-gnu.so _b.cpython-35m-x86_64-linux-gnu.so libabcommon.so
[064bit提示]> ldd build / lib.linux-x86_64-3.5 / _a.cpython-35m-x86_64-linux-gnu.so
linux-vdso.so.1 => （0x00007fffadb49000）
libabcommon.so => /home/cfati/Work/Dev/StackOverflow/q050938128/build/lib.linux-x86_64-3.5/libabcommon.so（0x00007f91cd50f000）
libstdc ++。so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6（0x00007f91cd18d000）
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1（0x00007f91ccf77000）
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6（0x00007f91ccbad000）
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0（0x00007f91cc990000）
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6（0x00007f91cc687000）
/lib64/ld-linux-x86-64.so.2（0x00007f91cd916000）
[064位提示] PYTHONPATH = $ {PYTHONPATH}：./ build / lib.linux-x86_64-3.5 python3 -c import _a; _a.funcA（）
a.cpp 6 funcA
[064bit提示] PYTHONPATH = $ {PYTHONPATH}：./ build / lib.linux-x86_64-3.5 python3 -c import _b; _b.funcB（）
b.cpp 7 funcB
a.cpp 6 funcA
- 添加了对 Win的支持（我想这个问题并不是很重要）
- 由于模块结构很简单，因此可以在 setup.py 的级别（不必修改源文件）
- 唯一的额外要求是 swig.exe ' s dir 应该位于％PATH％
I have four C++ files: A.h, A.cpp, B.h, B.cpp, and A.h is included in B.cpp
A.h:
#pragma once
void A();
A.cpp:
#include <iostream>
void A() {
std::cout << "A" << std::endl;
}
B.h:
#pragma once
void B();
B.cpp:
#include "A.h"
#include <iostream>
void B() {
A();
std::cout << "B" << std::endl;
}
Now I wrote two SWIG inerface files A.i and B.i
A.i:
%module A
%{
#include "A.h"
%}
%include "A.h"
B.i:
%module B
%{
#include "B.h"
%}
%include "B.h"
The setup.py file is:
from distutils.core import setup, Extension
A_ext = Extension( "_A", [ "A.i", "A.cpp", ], swig_opts = ['-c++'], extra_compile_args = ['-g'])
B_ext = Extension( "_B", [ "B.i", "B.cpp", ], swig_opts = ['-c++'], extra_compile_args = ['-g'])
setup(
name = "test",
version = "1.0",
ext_modules = [ A_ext, B_ext ],
py_modules = [ "A", "B" ]
)
If I type the command below, it will show 'A'.
python -c 'import A; A.A()'
If I type the command below, the segmentation fault appears:
python -c 'import B; B.B()'
How could I do to get this command run correctly? Since I don't want to compile B.cpp many times, is there any way except the one below?
B_ext = Extension( "_B", [ "B.i", "A.cpp", "B.cpp", ], swig_opts = ['-c++'], extra_compile_args = ['-g'])
解决方案
I changed your files a bit, for clarity.
a.h:
#pragma once
void funcA();
a.cpp:
#include <iostream>
void funcA() {
std::cout << __FILE__ << " " << __LINE__ << " " << __FUNCTION__ << std::endl;
}
a.i:
%module a
%{
#include "a.h"
%}
%include "a.h"
b.h:
#pragma once
void funcB();
b.cpp:
#include "a.h"
#include <iostream>
void funcB() {
std::cout << __FILE__ << " " << __LINE__ << " " << __FUNCTION__ << std::endl;
funcA();
}
b.i:
%module b
%{
#include "b.h"
%}
%include "b.h"
setup.py:
from distutils.core import setup
from distutils.extension import Extension
a = "a"
b = "b"
ext_a = Extension("_" + a, [a + ".i", a + ".cpp"], swig_opts=("-c++",), extra_compile_args=["-g"])
ext_b = Extension("_" + b, [b + ".i", b + ".cpp"], swig_opts=("-c++",), extra_compile_args=["-g"])
setup(
name="test",
version="1.0",
ext_modules=[ext_a, ext_b],
py_modules=[a, b]
)
What happens (simplified) when calling b.funcB (only the stacktrace, imports left aside). Each step invokes the next:
- funcB from module b (b.py)
- funcB from module _b (_b.so, or _b.cpython-35m-x86_64-linux-gnu.so)
- Everything from here happens in C (or C++)
- Current funcB is different than the one from b.cpp: it's generated by swig and its name is _wrap_funcB
- Previous bullet also applies to funcA and a.cpp
- funcB from b.cpp
- funcA from a.cpp
The problem is that the code from step #4. is not in the module _b, and it will fail at runtime. But things are a little bit stranger: the failure (core dump) doesn't appear when funcB is called but at module (b -> _b) import time (I think this happens because of swig's behind the scene magic), as seen below.
Output:
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ~/sopr.sh *** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages *** [064bit prompt]> ls a.cpp a.h a.i b.cpp b.h b.i setup.py [064bit prompt]> python3 setup.py build > /dev/null 2>&1 [064bit prompt]> echo $? 0 [064bit prompt]> ls a.cpp a.h a.i a.py a_wrap.cpp b.cpp b.h b.i b.py build b_wrap.cpp setup.py [064bit prompt]> ls ./build/lib.linux-x86_64-3.5 _a.cpython-35m-x86_64-linux-gnu.so _b.cpython-35m-x86_64-linux-gnu.so [064bit prompt]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _a;_a.funcA()" a.cpp 6 funcA [064bit prompt]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _b" Segmentation fault (core dumped)
In order to solve it, either:
- As you pointed out include funcA in module _b (by adding a.cpp in ext_b's source files list). This way, both modules will be self contained (from funcA's PoV), each will work independently, but also funcA will be duplicated in both of them
- Make _b depend on _a (after all, they're shared objects). But that's not how Python extension modules are to be used, and it won't work on Win (and also on some Nix flavors). So, this is more like a (lame) workaround (gainarie)
- Build a.cpp into a different shared library (an .so, but not a Python extension module) to be used by both modules. Needless to say that at runtime each of them will require the .so to be present
Obviously, to option #3. is the perfect candidate. But distutils ([Python.Docs]: API Reference) doesn't provide the needed functionality OOTB (apparently building an extension module and an external shared library that it depends on, is not a scenario that distutils aims for), or at least, I couldn't find any.
There is a build_clib module, which offers the functionality of building a static lib (to be used by the extension modules), but that would be same as option #1..
setup.py:
import sys
import os
from distutils.core import setup
from distutils.extension import Extension
from distutils.command.build_clib import build_clib
from distutils.command.build_ext import build_ext
from distutils.ccompiler import CCompiler
__win = sys.platform[:3].lower() == "win"
export_symbols_option = "export_symbols"
class build_clib_dyn(build_clib):
def finalize_options(self):
self.set_undefined_options('build',
('build_lib', 'build_clib'),
('build_temp', 'build_temp'),
('compiler', 'compiler'),
('debug', 'debug'),
('force', 'force'))
self.libraries = self.distribution.libraries
if self.libraries:
self.check_library_list(self.libraries)
if self.include_dirs is None:
self.include_dirs = self.distribution.include_dirs or []
if isinstance(self.include_dirs, str):
self.include_dirs = self.include_dirs.split(os.pathsep)
def build_libraries(self, libraries):
for (lib_name, build_info) in libraries:
sources = build_info.get('sources')
if sources is None or not isinstance(sources, (list, tuple)):
raise DistutilsSetupError(
"in 'libraries' option (library '%s'), "
"'sources' must be present and must be "
"a list of source filenames" % lib_name)
sources = list(sources)
macros = build_info.get('macros')
include_dirs = build_info.get('include_dirs')
objects = self.compiler.compile(sources,
output_dir=self.build_temp,
macros=macros,
include_dirs=include_dirs,
debug=self.debug)
self.compiler.link(CCompiler.SHARED_OBJECT, objects, self.compiler.library_filename(lib_name, lib_type="shared"),
output_dir=self.build_clib,
export_symbols=build_info.get(export_symbols_option),
debug=self.debug)
if __win:
class build_ext_w_dyn_dep(build_ext):
def finalize_options(self):
super(build_ext_w_dyn_dep, self).finalize_options()
self.library_dirs.append(os.path.dirname(self.build_temp))
else:
class build_ext_w_dyn_dep(build_ext):
pass
a_name = "a"
b_name = "b"
common_name = a_name + b_name + "common"
swig_opts = ["-c++"]
libraries = [common_name]
lib_common_build_info = {"sources": [a_name + ".cpp"]}
if __win:
extra_compile_args = None
extra_link_args = None
lib_common_build_info[export_symbols_option] = ["funcA"]
else:
extra_compile_args = ["-g"]
extra_link_args = ["-Wl,-rpath,${ORIGIN}"]
lib_common_info = (common_name, lib_common_build_info)
ext_a = Extension("_" + a_name, [a_name + ".i"], libraries=libraries, extra_compile_args=extra_compile_args, extra_link_args=extra_link_args, swig_opts=swig_opts)
ext_b = Extension("_" + b_name, [b_name + ".i", b_name + ".cpp"], libraries=libraries, extra_compile_args=extra_compile_args, extra_link_args=extra_link_args, swig_opts=swig_opts)
setup(
name="test",
version="1.0",
libraries=[lib_common_info],
cmdclass={"build_clib": build_clib_dyn, "build_ext": build_ext_w_dyn_dep},
ext_modules=[ext_a, ext_b],
py_modules=[a_name, b_name]
)
Notes:
- build_clib_dyn extends build_clib as its functionality had to be modified. 2 methods overridden but only small parts of them actually changed (comments were not copied from the base class methods (Python3.5.4 codebase), in order to reduce the amount of code, but that doesn't really count as a change)
- The task took quite some distutils code browsing, as some options are not documented (and I'm not very familiar with it)
- Some Nix knowledge was required as well, as things get tricky when loading shared libraries that are not in the system paths, and also keeping things smooth (e.g. not altering ${LD_LIBRARY_PATH})
- build_ext_w_dyn_dep is similar to build_clib_dyn (Win only). Since when building a Win dynamic link library (.dll), 2 files are generated, and in this case they are not in the same dir, library search paths needed some adjustments
- Python 3 and Python 2 compatible
Output (running the above commands again):
[064bit prompt]> ls a.cpp a.h a.i b.cpp b.h b.i setup.py [064bit prompt]> python3 setup.py build > /dev/null 2>&1 [064bit prompt]> echo $? 0 [064bit prompt]> ls a.cpp a.h a.i a.py a_wrap.cpp b.cpp b.h b.i b.py build b_wrap.cpp setup.py [064bit prompt]> ls build/lib.linux-x86_64-3.5/ _a.cpython-35m-x86_64-linux-gnu.so _b.cpython-35m-x86_64-linux-gnu.so libabcommon.so [064bit prompt]> ldd build/lib.linux-x86_64-3.5/_a.cpython-35m-x86_64-linux-gnu.so linux-vdso.so.1 => (0x00007fffadb49000) libabcommon.so => /home/cfati/Work/Dev/StackOverflow/q050938128/build/lib.linux-x86_64-3.5/libabcommon.so (0x00007f91cd50f000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f91cd18d000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f91ccf77000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f91ccbad000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f91cc990000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f91cc687000) /lib64/ld-linux-x86-64.so.2 (0x00007f91cd916000) [064bit prompt]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _a;_a.funcA()" a.cpp 6 funcA [064bit prompt]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _b;_b.funcB()" b.cpp 7 funcB a.cpp 6 funcA
- Added support for Win (I guess, it's not very important for the question)
- Since the module structure is simple, it was possible to do everything at setup.py's level (didn't have to modify the source files)
- The only extra requirement is that swig.exe's dir should be in %PATH%
这篇关于Distutils：构建共享一个方法的多个Python扩展模块（用Swig编写）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
