仅打包使用Cython编译的python库的二进制编译.so文件 [英] Package only binary compiled .so files of a python library compiled with Cython

问题描述

我有一个名为 mypack 的软件包，里面有一个模块 mymod.py ，而
__ init __。py 。
出于某些原因，这是没有争议的，我需要将此模块打包为
打包（也不允许使用.py或.pyc文件）。也就是说， __ init __。py 是分布式压缩文件中唯一允许的
源文件。

I have a package named mypack which inside has a module mymod.py, and the __init__.py. For some reason that is not in debate, I need to package this module compiled (nor .py or .pyc files are allowed). That is, the __init__.py is the only source file allowed in the distributed compressed file.

文件夹结构为：

. 
│  
├── mypack
│   ├── __init__.py
│   └── mymod.py
├── setup.py

我发现Cython能够做到这一点，方法是将.so库
中的每个.py文件转换为可以直接用python导入。

I find that Cython is able to do this, by converting each .py file in a .so library that can be directly imported with python.

问题是：为了便于打包和安装， setup.py 文件必须如何放置？

The question is: how the setup.py file must be in order to allow an easy packaging and installation?

目标系统有一个virtualenv，必须使用
来安装软件包，无论哪种方法都可以轻松安装和卸载（easy_install，pip等都是
欢迎）。

The target system has a virtualenv where the package must be installed with whatever method that allows easy install and uninstall (easy_install, pip, etc are all welcome).

我尽了一切努力。我阅读了 setuptools 和 distutils 文档，
所有与stackoverflow相关的问题，
并尝试了所有一种命令（sdist，bdist，bdist_egg等），在文件条目中有很多
的setup.cfg和MANIFEST。组合。

I tried all that was at my reach. I read setuptools and distutils documentation, all stackoverflow related questions, and tried with all kind of commands (sdist, bdist, bdist_egg, etc), with lots of combinations of setup.cfg and MANIFEST.in file entries.

是下面的安装文件，它将子类化bdist_egg
命令以同时删除.pyc文件，但这破坏了安装。

The closest I got was with the below setup file, that would subclass the bdist_egg command in order to remove also .pyc files, but that is breaking the installation.

A如果覆盖了正确的
安装中包括的所有辅助文件，则手动安装venv中的文件的解决方案也是不错的
（我需要运行 pip Frozen 并看到
mymod == 0.0.1 ）。

A solution that installs "manually" the files in the venv is also good, provided that all ancillary files that are included in a proper installation are covered (I need to run pip freeze in the venv and see mymod==0.0.1).

使用以下命令运行它：

python setup.py bdist_egg --exclude-source-files

并（尝试）使用

easy_install mymod-0.0.1-py2.7-linux-x86_64.egg

您可能会注意到，目标是使用python 2.7的linux 64位。

As you may notice, the target is linux 64 bits with python 2.7.

from Cython.Distutils import build_ext
from setuptools import setup, find_packages
from setuptools.extension import Extension
from setuptools.command import bdist_egg
from setuptools.command.bdist_egg import  walk_egg, log 
import os

class my_bdist_egg(bdist_egg.bdist_egg):

    def zap_pyfiles(self):
        log.info("Removing .py files from temporary directory")
        for base, dirs, files in walk_egg(self.bdist_dir):
            for name in files:
                if not name.endswith('__init__.py'):
                    if name.endswith('.py') or name.endswith('.pyc'):
                        # original 'if' only has name.endswith('.py')
                        path = os.path.join(base, name)
                        log.info("Deleting %s",path)
                        os.unlink(path)

ext_modules=[
    Extension("mypack.mymod", ["mypack/mymod.py"]),
]

setup(
  name = 'mypack',
  cmdclass = {'build_ext': build_ext, 
              'bdist_egg': my_bdist_egg },
  ext_modules = ext_modules,
  version='0.0.1',
  description='This is mypack compiled lib',
  author='Myself',
  packages=['mypack'],
)

更新。
在@Teyras答案之后，可以按照答案中的要求建造轮子。 setup.py 文件内容为：

import os
import shutil
from setuptools.extension import Extension
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext

class MyBuildExt(build_ext):
    def run(self):
        build_ext.run(self)
        build_dir = os.path.realpath(self.build_lib)
        root_dir = os.path.dirname(os.path.realpath(__file__))
        target_dir = build_dir if not self.inplace else root_dir
        self.copy_file('mypack/__init__.py', root_dir, target_dir)

    def copy_file(self, path, source_dir, destination_dir):
        if os.path.exists(os.path.join(source_dir, path)):
            shutil.copyfile(os.path.join(source_dir, path), 
                            os.path.join(destination_dir, path))


setup(
  name = 'mypack',
  cmdclass = {'build_ext': MyBuildExt},
  ext_modules = cythonize([Extension("mypack.*", ["mypack/*.py"])]),
  version='0.0.1',
  description='This is mypack compiled lib',
  author='Myself',
  packages=[],
  include_package_data=True )

关键是设置 packages = []，。为了获得 __ init__.py，需要覆盖 build_ext 类 run 方法。 code>文件。

The key point was to set packages=[],. The overwriting of the build_ext class run method was needed to get the __init__.py file inside the wheel.

推荐答案

不幸的是，接受的答案设置为 packages = [] 是错误的，可能会破坏很多内容，例如可以在此问题中看到。不要使用它。不应从dist中排除所有软件包，而应该仅排除将被cythonized并编译为共享对象的python文件。

Unfortunately, the accepted answer with setting packages=[] is wrong and may break a lot of stuff, as can e.g. be seen in this question. Don't use it. Instead of excluding all packages from the dist, you should exclude only the python files that will be cythonized and compiled to shared objects.

下面是一个有效的示例；它使用我的食谱 >从python bdist_egg或bdist_wheel 中排除单个源文件。示例项目包含带有两个模块 spam.eggs 和 spam.bacon <的软件包 spam / code>，以及带有一个模块 spam.fizz.buzz 的子包 spam.fizz ：

Below is a working example; it uses my recipe from the question Exclude single source file from python bdist_egg or bdist_wheel. The example project contains package spam with two modules, spam.eggs and spam.bacon, and a subpackage spam.fizz with one module spam.fizz.buzz:

root
├── setup.py
└── spam
    ├── __init__.py
    ├── bacon.py
    ├── eggs.py
    └── fizz
        ├── __init__.py
        └── buzz.py

模块查找是在 build_py 中完成的

如果您要编译每个 .py 文件（包括 __ init __。py s），覆盖 build_py.build_packages 方法就足够了，使其成为noop。由于 build_packages 不执行任何操作，因此将不会收集任何 .py 文件，并且dist仅包含经过cythonized处理的文件。扩展名：

If you are about to compile every .py file (including __init__.pys), it is already sufficient to override build_py.build_packages method, making it a noop. Because build_packages doesn't do anything, no .py file will be collected at all and the dist will include only cythonized extensions:

import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize


extensions = [
    # example of extensions with regex
    Extension('spam.*', ['spam/*.py']),
    # example of extension with single source file
    Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]


class build_py(build_py_orig):
    def build_packages(self):
        pass


setup(
    name='...',
    version='...',
    packages=find_packages(),
    ext_modules=cythonize(extensions),
    cmdclass={'build_py': build_py},
)

复杂的情况：混合sou的cythonized扩展rce模块

如果只想编译选定的模块，而其余的都保持不变，则需要更复杂的逻辑。在这种情况下，您需要覆盖模块查找。在下面的示例中，我仍然编译 spam.bacon ， spam.eggs 和 spam。 fizz.buzz 共享对象，但不修改 __ init __。py 文件，因此它们将作为源模块包括在内：

Complex case: mix cythonized extensions with source modules

If you want to compile only selected modules and leave the rest untouched, you will need a bit more complex logic; in this case, you need to override module lookup. In the below example, I still compile spam.bacon, spam.eggs and spam.fizz.buzz to shared objects, but leave __init__.py files untouched, so they will be included as source modules:

import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize


extensions = [
    Extension('spam.*', ['spam/*.py']),
    Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
cython_excludes = ['**/__init__.py']


def not_cythonized(tup):
    (package, module, filepath) = tup
    return any(
        fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
    ) or not any(
        fnmatch.fnmatchcase(filepath, pat=pattern)
        for ext in extensions
        for pattern in ext.sources
    )


class build_py(build_py_orig):
    def find_modules(self):
        modules = super().find_modules()
        return list(filter(not_cythonized, modules))

    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return list(filter(not_cythonized, modules))


setup(
    name='...',
    version='...',
    packages=find_packages(),
    ext_modules=cythonize(extensions, exclude=cython_excludes),
    cmdclass={'build_py': build_py},
)

这篇关于仅打包使用Cython编译的python库的二进制编译.so文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！