覆盖python setuptools中的shebang mangling [英] Override the shebang mangling in python setuptools

问题描述

背景

我为使用模块( https://luarocks.org/)的系统编写小型python软件包管理软件包.对于那些不了解它的人，可以运行module load x，然后运行一个小的脚本来修改各种环境变量以使软件"x"正常工作，然后可以使用module unload x撤消该操作.

这种软件管理方法在科学计算中几乎无处不在，并且在该领域具有很大的价值:您可以运行古老的未维护软件以及该软件可能会干扰的软件包，还可以运行多个版本的软件，从而可以来精确地复制数据(您可以返回到旧版本)，并且可以运行具有过时依赖关系的坦率地编写，未更新的软件.

这些功能很棒，但是它们在python 2/3 split中造成了问题:

如果您想编写一个与两者 python 2和3均兼容的软件包并将其与需要 python 2或3的软件一起使用，怎么办?

使旧的python2依赖软件在这些大型系统上工作的方式是，制作python/2.7.x模块和python/3.5模块.当您要运行使用python 2的脚本时，请加载该模块等.

但是，我想编写一个可以在任一环境中使用的python软件包，因为我希望该软件能够正常运行，无论使用哪个python解释器.

从根本上讲这非常容易:只需使用#!/usr/bin/env python shebang行即可.那个有效.我编写了所有软件都可以使用，所以没问题.

问题

问题是:我想使用setuptools将软件包分发给处于相同情况的其他科学家，而setup工具会破坏shebang线.

我不想就修改Shebang线是否是一个好主意展开辩论，我相信这是因为它已经在同一状态下存在多年了.老实说，我不在乎，这对我不起作用.默认的setuptools安装会导致软件无法运行，因为当未加载python解释器的模块时，该python解释器无法运行，因此PYTHONPATH完全是错误的.

如果我所有的用户都具有root访问权限，则可以使用data_files选项将脚本复制到/usr/bin，但这对兼容性不是一个好主意，而且我的用户也没有root访问权限，因此这是有争议的.

到目前为止我尝试过的事情:

我尝试在setup.py文件中将sys.executable设置为/usr/bin/env python，但这不起作用，因为那样的话，shebang是:#!"/usr/bin/env python"，显然不起作用.

我在以下问题中尝试了不要碰我的shebang类"想法:不要碰我的shebang ！(这是底数为0的答案).那也不起作用，可能是因为它是为distutils而不是setuptools编写的.再加上这个问题已有6年历史了.

我也看了这些问题:

Setuptools entry_points/console_scripts具有特定的Python shebang中的版本

更改用于包装的console_script入口点解释器

那里描述的方法不起作用，shebang行仍然被更改.

使用内容创建setup.cfg文件:

[build]
executable = /usr/bin/env python

也不会改变shebang线的扭曲行为.

setuptools github页面上有一个未解决的问题，讨论了类似的事情:

https://github.com/pypa/setuptools/issues/494

所以我认为这不可能在本地完成，但是我想知道是否有解决方法吗?

最后，我不喜欢任何涉及要求用户修改安装标志的解决方案，例如与-e.

反正有没有修改此行为的方法，或者我可以使用另一个发行系统来代替?还是这太过分了，我只需要编写某种自定义安装脚本即可?

谢谢.

---

更新

我想我对最初的问题还不够清楚，我希望用户能够做到的是:

• 在python2和python3中都安装软件包(模块将进入lib/pythonx/site-lib.
• 无论哪个Python环境处于活动状态，都能够运行脚本.

如果有一种方法可以在不阻止shebang骚扰的情况下做到这一点，那就太好了.

我所有的代码都已经可以立即与python 2.7和python 3.3+兼容，主要是使脚本无论运行的python环境如何都可以运行.

解决方案

在尝试编写自定义安装脚本时，我偶然发现了一种解决方法.

 import os
from setuptools import setup
from setuptools.command.install import install

here = os.path.abspath(os.path.dirname(__file__))

# Generate a list of python scripts
scpts = []
scpt_dir = os.listdir(os.path.join(here, 'bin'))
for scpt in scpt_dir:
    scpts.append(os.path.join(here, 'bin', scpt))

class ScriptInstaller(install):

    """Install scripts directly."""

    def run(self):
        """Wrapper for parent run."""
        super(ScriptInstaller, self).run()

setup(
    cmdclass={'install': ScriptInstaller},
    scripts=scpts,
    ...
)
 

此代码并没有完全满足我的要求(仅更改了shebang行)，实际上只是将整个脚本复制到了~/.local/bin，而不是将其包装在::

 __import__('pkg_resources').run_script()
 

此外，更重要的是，此方法使setuptools创建一个根模块目录以及一个egg-info目录，如下所示::

 .local/lib/python3.5/site-packages/cluster
.local/lib/python3.5/site-packages/python_cluster-0.6.1-py3.5.egg-info
 

通常的行为是代替单个鸡蛋:

 .local/lib/python3.5/site-packages/python_cluster-0.6.1-py3.5.egg
 

据我所知，这是旧distutils的行为，这使我担心此安装在某些系统上会失败或具有其他意外行为(尽管如果我输入错误，请更正我，我真的不是专家在此).

但是，考虑到我的代码几乎将全部在linux和OS X上使用，这还不是世界末日.我更担心这种行为会很快消失.

我在setuptools github页面上对开放功能请求发表了评论:

https://github.com/pypa/setuptools/issues/494

理想的解决方案是，如果我可以在setup.cfg中添加executable=/usr/bin/env python语句，希望可以尽快重新实现.

不过，该变通办法现在对我来说仍然有效.谢谢大家.

Background

I write small python packages for a system that uses modules (https://luarocks.org/) to manage packages. For those of you who don't know it, you can run module load x and a small script is run that modifies various environmental variables to make software 'x' work, you can then undo this with module unload x.

This method of software management is nearly ubiquitous in scientific computing and has a lot of value in that arena: you can run ancient unmaintained software alongside packages that that software would interfere with, you can run multiple versions of software, which allows you to reproduce your data exactly (you can go back to old versions), and you can run frankly poorly written non updated software with outdated dependencies.

These features are great, but they create an issue with the python 2/3 split:

What if you want to write a package that works with both python 2 and 3 and use it alongside software that requires either python 2 or 3?

The way you make old python2 dependent software work on these large systems is that you make a python/2.7.x module and a python/3.5 module. When you want to run a script that uses python 2, you load that module, etc.

However, I want to write a single python package that can work in either environment, because I want that software to be active regardless of which python interpreter is being used.

This is fundamentally extremely easy: just use a #!/usr/bin/env python shebang line, done. That works. I write all my software to work with either, so no problem.

Question

The issue is: I want to use setuptools to distribute my package to other scientists in the same situation, and setup tools mangles the shebang line.

I don't want to get into a debate about whether mangling the shebang line is a good idea or not, I am sure it is since it has existed for years now in the same state. I honestly don't care, it doesn't work for me. The default setuptools install causes the software not to run because when a python interpreter's module is not loaded, that python interpreter does not function, the PYTHONPATH is totally wrong for it.

If all of my users had root access, I could use the data_files option to just copy the scripts to /usr/bin, but this is a bad idea for compatibility, and my users don't have root access anyway so it is a moot point.

Things I tried so far:

I tried setting the sys.executable to /usr/bin/env python in the setup.py file, but that doesn't work, because then the shebang is: #!"/usr/bin/env python", which obviously doesn't work.

I tried the Don't touch my shebang class idea in this question: Don't touch my shebang! (it is the bottom answer with 0 votes). That didn't work either, probably because it is written for distutils and not setuptools. Plus that question is 6 years old.

I also looked at these questions:

Setuptools entry_points/console_scripts have specific Python version in shebang

Changing console_script entry point interpreter for packaging

The methods described there do not work, the shebang line is still altered.

Creating a setup.cfg file with the contents::

[build]
executable = /usr/bin/env python

also does not change the shebang line mangling behavior.

There is an open issue on the setuptools github page that discusses something similar:

https://github.com/pypa/setuptools/issues/494

So I assume this isn't possible to do natively, but I wonder if there is a workaround?

Finally, I don't like any solution that involves asking the user to modify their install flags, e.g. with -e.

Is there anyway to modify this behavior, or is there another distribution system I can use instead? Or is this too much of an edge case and I just need to write some kind of custom installation script?

Thanks all.

---

Update

I think I was not clear enough in my original question, what I want the user to be able to do is:

• Install the package in both python2 and python3 (the modules will go into lib/pythonx/site-lib.
• Be able to run the scripts irrespective of which python environment is active.

If there is a way to accomplish this without preventing shebang munging, that would be great.

All my code is already compatible with python 2.7 and python 3.3+ out of the box, the main thing is just making the scripts run irrespective of active python environment.

解决方案

I accidentally stumbled onto a workaround while trying to write a custom install script.

import os
from setuptools import setup
from setuptools.command.install import install

here = os.path.abspath(os.path.dirname(__file__))

# Generate a list of python scripts
scpts = []
scpt_dir = os.listdir(os.path.join(here, 'bin'))
for scpt in scpt_dir:
    scpts.append(os.path.join(here, 'bin', scpt))

class ScriptInstaller(install):

    """Install scripts directly."""

    def run(self):
        """Wrapper for parent run."""
        super(ScriptInstaller, self).run()

setup(
    cmdclass={'install': ScriptInstaller},
    scripts=scpts,
    ...
)

This code doesn't do exactly what I wanted (alter just the shebang line), it actually just copies the whole script to ~/.local/bin, instead of wrapping it in::

__import__('pkg_resources').run_script()

Additionally, and more concerningly, this method makes setuptools create a root module directory plus an egg-info directory like this::

.local/lib/python3.5/site-packages/cluster
.local/lib/python3.5/site-packages/python_cluster-0.6.1-py3.5.egg-info

Instead of a single egg, which is the usual behavior::

.local/lib/python3.5/site-packages/python_cluster-0.6.1-py3.5.egg

As far as I am aware this is the behavior of the old distutils, which makes me worry that this install would fail on some systems or have other unexpected behavior (although please correct me if I am wrong, I really am no expert on this).

However, given that my code is going to be used almost entirely on linux and OS X, this isn't the end of the world. I am more concerned that this behavior will just disappear sometime very soon.

I posted a comment on an open feature request on the setuptools github page:

https://github.com/pypa/setuptools/issues/494

The ideal solution would be if I could add an executable=/usr/bin/env python statement to setup.cfg, hopefully that is reimplemented soon.

This workaround will work for me for now though. Thanks all.

这篇关于覆盖python setuptools中的shebang mangling的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！