一个简单的 Python 部署问题——一个痛苦的世界 [英] A simple Python deployment problem - a whole world of pain

问题描述

我们有几个 Python 2.6 应用程序在 Linux 上运行.其中一些是 Pylons Web 应用程序，其他只是我们使用 nohup 从命令行运行的长时间运行的进程.我们还在开发和生产中使用 virtualenv.将这些应用程序部署到生产服务器的最佳方式是什么?

We have several Python 2.6 applications running on Linux. Some of them are Pylons web applications, others are simply long-running processes that we run from the command line using nohup. We're also using virtualenv, both in development and in production. What is the best way to deploy these applications to a production server?

在开发中，我们只需将源代码树放入任何目录，设置一个 virtualenv 并运行 - 非常简单.我们可以在生产中做同样的事情，也许这确实是最实用的解决方案，但在生产中运行 svn update 感觉有点错误.我们也尝试过 fab，但它只是第一次没有奏效.对于每个应用程序，其他地方都会出错.我觉得整个过程太难，因为我们要实现的目标基本上非常简单.这是我们希望从部署过程中得到的东西.

In development we simply get the source tree into any directory, set up a virtualenv and run - easy enough. We could do the same in production and perhaps that really is the most practical solution, but it just feels a bit wrong to run svn update in production. We've also tried fab, but it just never works first time. For every application something else goes wrong. It strikes me that the whole process is just too hard, given that what we're trying to achieve is fundamentally very simple. Here's what we want from a deployment process.

1. 我们应该能够运行一个简单的命令来部署应用程序的更新版本.(如果初始部署涉及一些额外的复杂性，那很好.)
2. 当我们运行这个命令时，它应该将某些文件从 Subversion 存储库或本地工作副本复制到服务器上的指定环境"，这可能意味着不同的 virtualenv.我们在同一台服务器上同时拥有应用程序的临时版本和生产版本，因此需要以某种方式将它们分开.如果它安装到站点包中，那也没关系，只要它可以工作.
3. 我们在服务器上有一些应该保留的配置文件(即部署过程不会覆盖或删除).
4. 其中一些应用程序从其他应用程序导入模块，因此它们需要能够以某种方式作为包相互引用.这是我们遇到的最麻烦的部分！我不在乎它是否通过相对导入、站点包或其他方式工作，只要它在开发和生产中都能可靠地工作.
5. 理想情况下，部署过程应自动安装我们的应用程序所依赖的外部包(例如 psycopg2).

1. We should be able to run one simple command to deploy an updated version of an application. (If the initial deployment involves a bit of extra complexity that's fine.)
2. When we run this command it should copy certain files, either out of a Subversion repository or out of a local working copy, to a specified "environment" on the server, which probably means a different virtualenv. We have both staging and production version of the applications on the same server, so they need to somehow be kept separate. If it installs into site-packages, that's fine too, as long as it works.
3. We have some configuration files on the server that should be preserved (ie. not overwritten or deleted by the deployment process).
4. Some of these applications import modules from other applications, so they need to be able to reference each other as packages somehow. This is the part we've had the most trouble with! I don't care whether it works via relative imports, site-packages or whatever, as long as it works reliably in both development and production.
5. Ideally the deployment process should automatically install external packages that our applications depend on (eg. psycopg2).

原来如此！有多难?

推荐答案

setuptools 结合 virtualenv 和 pip.

我发现最棘手的部分是运行一个开发环境，该环境尽可能地反映已部署的设置，同时尊重 Pythonic 工具和习惯用法.但事实证明，使用 pip 和 setuptools 很容易实现这一点，它们一起允许您将开发树安装"到 Python 环境中，而无需四处移动文件.(实际上 setuptools 自己完成了这一切，但是 pip 作为前端处理依赖关系更好.)

The trickiest part, I've found, is running a development environment that mirrors the deployed setup as closely as possible, while at the same time respecting Pythonic tools and idioms. But it turns out that this is very easy to achieve with pip and setuptools, which together allow you to "install" a development tree into a Python environment without moving the files around. (Actually setuptools does this all by itself, but pip acts as a front end handle dependencies better.)

另一个关键问题是准备一个干净的环境，其中包含跨两个环境的已知包集.Python 的 virtualenv 在这方面是神赐之物，它允许您使用自己选择的软件包配置完全定制的 Python 环境，无需 root 访问权限或操作系统软件包(rpm 或 dpkg)，也不受任何软件包和版本的限制它恰好安装在您的发行版上.

Another key issue is preparing a clean environment with a known package set across both environments. Python's virtualenv is a god-send in this respect, allowing you to configure a completely customised Python environment with your own choice of packages, without requiring root access, or OS packages (rpm or dpkg), and without being constrained by whatever packages and versions thereof that happen to be installed on your distro.

最后，一个令人讨厌的问题是难以创建与 PYTHON_PATH 配合良好的命令行脚本.这也可以通过 setuptools 优雅地处理.

Finally, one annoying bug-bear is the difficulty of creating command-line scripts that play nice with PYTHON_PATH. This is also dealt with quite elegantly by setuptools.

(为了简单起见，这是相当规范的.您可以酌情发散.)

(To keep things simple, this is fairly prescriptive. Feel free to diverge as appropriate.)

1. 准备一个工作目录，您的 Python 内容可以称之为家.
2. 从这个页面的底部获取最新的 virtualenv 并解压它(没关系在哪里).

3. 从工作目录，设置一个新的 Python 虚拟环境:

1. Prepare a working directory that your Python stuff can call home.
2. Grab the latest virtualenv from the bottom of this page and untar it (doesn't matter where).

3. From the working directory, setup a new Python virtual environment:

$ python <untarred_directory>/virtualenv.py venv

您会希望在这个虚拟环境中完成大部分工作.使用此命令执行此操作(. 是 source 的快捷方式):

You'll want to do most of your work from inside this virtual environment. Use this command to do so (. is a shortcut for source):

$ . venv/bin/activate

安装pip:

Install pip:

$ easy_install pip

为您要创建的每个可安装包创建目录.

Create directories for each installable package you want to create.

开发

一旦您的树结构准备就绪，您就几乎可以开始编码了.但是现在，相互依赖的包无法像在部署环境中那样相互看到.这个问题是通过 setuptools 提供的一个巧妙的小技巧解决的，pip 使用了哪个.对于您正在开发的每个包，运行以下命令(确保您在项目的虚拟环境中，按照上面的第 3 步):

Development

Once your tree structure is ready, you are almost ready to begin coding. But right now, packages that depend on each other can't see each other as they will in the deployed environment. This problem is resolved with a neat little trick that setuptools offers, and which pip makes use of. For each package you are developing, run the following command (make sure you are in the virtual environment for your project, as per step 3, above):

$ pip install -e pkg1

此命令会将 pkg1 安装到您的虚拟环境中，并且不会复制您的任何文件.它只是添加一个指向 site-packages 目录的链接，指向包的开发根目录，并在该根目录中创建一个 egg-info 目录.您也可以不使用 pip 执行此操作，如下所示:

This command will install pkg1 into your virtual environment, and it does so without copying any of your files. It simply adds a link to the site-packages directory pointing to the package's development root and creates an egg-info directory in that root. You can also do this without pip, as follows:

$ cd pkg1
$ python setup.py develop

它通常会起作用，但如果您有第三方依赖项(应在 setup.py 中列出，如此处 在 setuptools 文档中)，pip 更智能地找到它们.

And it will usually work, but if you have third-party dependencies (which should be listed in setup.py, as explained here in the setuptools documentation), pip is smarter about finding them.

需要注意的一点是，setuptools 和 pip 都没有在您自己的包之间查找依赖项的任何智能.如果B目录下的PkgB，依赖A目录下的PkgA，那么pip install -e B就会失败，因为pip无法知道A目录下可以找到PkgA；相反，它将尝试从其在线存储库源下载 PkgA，但失败了.解决方法是在依赖项之后安装每个包.

One caveat to note is that neither setuptools nor pip have any smarts about finding dependencies amongst your own packages. If PkgB in directory B, depends on PkgA in directory A, then pip install -e B will fail, because pip has no way of knowing that PkgA can be found in directory A; it will instead try, and fail, to download PkgA from its online repository sources. The workaround is simply to install each package after its dependencies.

此时，您可以启动 python，加载您的一个模块并开始使用它.您可以编辑代码，下次导入时将立即可用.

At this point, you can start python, load up one of your modules and start toying with it. You can edit code, and it will be immediately available the next time you import it.

最后，如果你想用你的包创建命令行工具.不要手写它们.你最终会遇到一团糟的 PYTHON_PATH 黑客，这些黑客永远无法正常工作.只需阅读 setuptools 文档中的自动脚本创建.这会让你免去很多悲伤.

Finally, if you want to create command-line tools with your packages. Don't write them by hand. You'll end up with a horrible mess of PYTHON_PATH hacks that never quite works properly. Just read up on automatic script creation in the setuptools documentation. This will spare you a lot of grief.

当您的包准备好用于操作时，您可以使用 setup.py 创建部署包.这里有太多选择，但以下内容应该可以帮助您入门:

When your packages are ready for action, you can use setup.py to create deployment packages. There are way too many options to go into here, but the following should get you started:

$ cd pkg1
$ python setup.py --help
$ python setup.py --help-commands

松散的两端

由于问题的广泛性，这个答案必然是不完整的.我没有处理过长时间运行的服务器、web 框架或实际的部署过程本身(特别是使用 pip install 的 --index-url 来管理第三方和内部的私有存储库包和 -e vcs+...，它将从 svn、git、hg 或 bzr 中提取包).但我希望我已经给了你足够的绳子把它们绑在一起(只是不要用它来吊死自己:-).

Loose ends

Due to the broad nature of the question, this answer is necessarily incomplete. I haven't dealt with long-running servers, web frameworks or the actual deployment process itself (in particular, use of pip install's --index-url to manage a private repository of third-party and internal packages and -e vcs+..., which will pull packages out of svn, git, hg or bzr). But I hope I've given you enough rope to tie it all together (just don't hang yourself with it :-).

这篇关于一个简单的 Python 部署问题——一个痛苦的世界的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！