如何在Windows上使用cython编译__init__.py文件 [英] How to compile __init__.py file using cython on Windows
问题描述
当我使用 setup.py build_ext --inplace
命令在Windows上编译任意__init__.py文件时,它具有无法解决的外部符号错误(即LINK:错误LNK2001:无法解析的外部符号PyInit ___ init __)。
When I compile an arbitrary __init__.py file on Windows with setup.py build_ext --inplace
command, it has an unresolvable external symbol error (i.e. "LINK : error LNK2001: An unresolvable external symbol PyInit___init__").
本地环境:
python3.7,
Cython 0.29.14,
window10 x64,
Microsoft Visual Studio 2017,
ctest / __ init __。py
ctest/__init__.py
# cython: language_level=3
print('__init__')
setup.py
from distutils.core import setup
from Cython.Build import cythonize
def compile_code(name, filename):
setup(
name=name,
ext_modules=cythonize(filename),
)
if __name__ == '__main__':
compile_code('a', 'ctest/__init__.py')
终端打印的信息:
Compiling ctest/__init__.py because it changed.
[1/1] Cythonizing ctest/__init__.py
running build_ext
building 'ctest.__init__' extension
creating build
creating build\temp.win-amd64-3.7
creating build\temp.win-amd64-3.7\Release
creating build\temp.win-amd64-3.7\Release\ctest
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Id:\py37\include -Id:\py37\incl
ude "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Pro
gram Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\includ
e\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcctest/__init__
.c /Fobuild\temp.win-amd64-3.7\Release\ctest/__init__.obj
__init__.c
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTU
AC:NO /LIBPATH:d:\py37\Libs /LIBPATH:D:\ENVS\cpytrantest\libs /LIBPATH:D:\ENVS\cpytrantest\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC
\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x6
4" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit___init__ build\temp.win-amd64-3.7\Release\ctest/__init__.obj /OUT:C:\Users\76923\Deskto
p\cpythonrecord\ctest\__init__.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\ctest\__init__.cp37-win_amd64.lib
LINK : error LNK2001: An unresolvable external symbol PyInit___init__
build\temp.win-amd64-3.7\Release\ctest\__init__.cp37-win_amd64.lib : fatal error LNK1120: An external command that cannot be parsed
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120
推荐答案
也许此行为可能被视为 distutils
-package中的一个小错误(如@DavidW所指出的那样)是一个未解决的问题: https://bugs.python.org/issue35893 )。但是,它也表明,对 __ init __。py
进行cythonize / compiling的方法不是很流行,它使用了一些未记录的实现细节,这些细节将来可能会改变,因此这样做可能更明智。避免干预 __ init __。py
。
Maybe this behaviour might be viewed as a small bug in distutils
-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py
isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py
.
但是如果您必须...
But if you must...
显式导入软件包时,例如
When a package is imported explicitly, e.g.
import ctest
或隐式,例如
import ctest.something
FileFinder
将看到已导入一个包而不是模块,并且将尝试加载 ctest / __ init __。py
代替 ctest.py
的值(很可能不存在):
The FileFinder
will see that a package, and not a module, is imported and will try to load ctest/__init__.py
instead of ctest.py
(which most likely doesn't exists):
# Check if the module is the name of a directory (and thus a package).
if cache_module in cache:
base_path = _path_join(self.path, tail_module)
for suffix, loader_class in self._loaders:
init_filename = '__init__' + suffix
full_path = _path_join(base_path, init_filename)
if _path_isfile(full_path):
return self._get_spec(loader_class, fullname, full_path, [base_path], target)
使用的后缀loader_class
用于加载 __ init __。so
, __ init __。py
和 __ init __。pyc
的顺序(另请参见SO-post )。这意味着,如果我们设法创建 __ init __。so
而不是 __ init __。py
,则会加载它。
Used suffix, loader_class
are for loading __init__.so
, __init__.py
and __init__.pyc
in this order (see also this SO-post). This means, __init__.so
will be loaded instead of __init__.py
if we manage to create one.
执行 __ init __。py
时,属性 __ name __
是包的名称,例如您的情况下的 ctest
,而不是人们认为的 __ init __
。因此,Python解释器在加载扩展名 __ init __。so
为 PyInit_ctest
时将调用的init函数的名称。在您的情况下(而不是人们可能认为的 PyInit ___ init __
)。
While __init__.py
is executed, The property __name__
is the name of the package, i.e. ctest
in your case, and not __init__
as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so
is PyInit_ctest
in your case (and not PyInit___init__
as one might think).
以上内容说明了为什么所有这些都不能在Linux上运行盒子。 Windows呢?
The above explains, why it all works on Linux out-of-the-box. What about Windows?
加载程序只能使用so / dll中未被隐藏的符号。默认情况下,所有使用gcc构建的符号都是可见的,但Windows上的VisualStudio则不可见-默认情况下,所有符号都隐藏(请参见例如 SO- post )。
The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).
但是,C扩展的init函数必须是可见的(并且只有init函数),因此可以在加载程序的帮助下进行调用-解决方案是在链接时导出该符号(即 PyInit_ctest
),在您的情况下,这是错误的 / EXPORT:PyInit ___ init __
-链接程序的选项。
However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest
) while linking, in your case it is the wrong /EXPORT:PyInit___init__
-option for the linker.
该问题可以在distutils中找到,或更精确的在 build_ext
级:
The problem can be found in distutils, or more precise in build_ext
-class:
def get_export_symbols(self, ext):
"""Return the list of symbols that a shared extension has to
export. This either uses 'ext.export_symbols' or, if it's not
provided, "PyInit_" + module_name. Only relevant on Windows, where
the .pyd file (DLL) must export the module "PyInit_" function.
"""
initfunc_name = "PyInit_" + ext.name.split('.')[-1]
if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
return ext.export_symbols
在这里,可悲的是 ext.name
具有 __ init __
。
Here, sadly ext.name
has __init__
in it.
从这里开始,一种可能的解决方案很简单:覆盖 get_export_symbols
,即添加跟随您的 setup.py
文件(请阅读以获取更简单的版本):
From here, one possible solution is easy : to override get_export_symbols
, i.e. to add the following to your setup.py
-file (read on for a even simpler version):
...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
names = ext.name.split('.')
if names[-1] != "__init__":
initfunc_name = "PyInit_" + names[-1]
else:
# take name of the package if it is an __init__-file
initfunc_name = "PyInit_" + names[-2]
if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
return ext.export_symbols
# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...
调用 python setup.py build_ext -i
现在应该足够了(因为 __ init __。so
将被加载,而不是 __ init__ .py
)。
Calling python setup.py build_ext -i
should be enough now (because __init__.so
will be loaded rather than __init__.py
).
但是,正如@DawidW指出的,Cython使用宏 PyMODINIT_FUNC
,已定义as
However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC
, which is defined as
#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*
带有 Py_EXPORTED_SYMBOL
被标记为在Windows上可见/已导出:
with Py_EXPORTED_SYMBOL
being marked as visible/exported on Windows:
#define Py_EXPORTED_SYMBOL __declspec(dllexport)
因此,无需将符号标记为在命令行可见。更糟糕的是,这是警告LNK4197 的原因:
Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:
__ init __。obj:警告LNK4197:多次指定导出 PyInit_ctest;
使用第一个规范
__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times; using first specification
如 PyInit_test
标记为 __declspec(dllexport)
并同时通过选项 / EXPORT:
导出。
as PyInit_test
is marked as __declspec(dllexport)
and exported via option /EXPORT:
at the same time.
/ EXPORT:
-option将被distutils跳过,如果 export_symbols
为空,我们甚至可以使用更简单的 command.build_ext
:
/EXPORT:
-option will be skipped by distutils, if export_symbols
is empty, we can use even a simpler version of command.build_ext
:
...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
pass # return [] also does the job!
# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...
这比第一个版本更好,因为它还修复了警告LNK4197!
This is even better than the first version, as it also fixes warning LNK4197!
这篇关于如何在Windows上使用cython编译__init__.py文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!