一个简单的Python部署问题 - 一个痛苦的世界 [英] A simple Python deployment problem - a whole world of pain

问题描述

我们在Linux上运行了几个Python 2.6应用程序。其中一些是Pylons Web应用程序，其他一些是简单的长期运行的进程，我们从命令行使用 nohup 运行。我们还在开发和生产中使用 virtualenv 。 将这些应用程序部署到生产服务器的最佳方法是什么？

在开发中，我们只需将源代码树导入任何目录，设置一个virtualenv和运行 - 足够容易。我们可以在生产中做同样的事情，也许这是最实际的解决方案，但是在生产中运行 svn update 感觉有点错误。我们也尝试过 fab ，但它从来不会第一次工作。对于每个应用程序，其他的都出错了。它使我觉得，整个过程是太难了，因为我们要实现的是从根本上非常简单。这是我们从部署过程中想要的。

1. 我们应该能够运行一个简单的命令来部署更新版本的应用程序。 （如果初始部署涉及到一些额外的复杂性，那么很好）。


2. 当我们运行这个命令时，它应该拷贝某些文件，不管是在Subversion的存储库还是从本地的工作副本，到服务器上的指定环境，这可能意味着不同的virtualenv。我们在同一台服务器上都有分期和生产版本的应用程序，所以他们需要以某种方式保持分开。如果它安装到site-packages中，那就行了。


3. 我们在服务器上有一些应该保留的配置文件（即不被覆盖或被删除）部署过程）


4. 其中一些应用程序从其他应用程序导入模块，因此他们需要能够以不同的方式引用包。这是我们遇到最麻烦的部分！只要在开发和生产中可靠地运行，我不关心它是否通过相对的导入，站点包或其他方式工作。


5. 理想情况下，部署过程应自动安装外部包我们的应用程序取决于（例如psycopg2）。

这是真的！

解决方案

开发和部署Python代码通过 setuptools 与 virtualenv 和 pip 。

< h2>核心思想

我发现最棘手的部分是运行一个尽可能接近部署的设置的开发环境，同时尊重Pythonic工具和成语。但事实证明，这是非常容易实现的pip和setuptools，它们一起允许您将开发树安装到Python环境中，而不会移动文件。 （实际上，setuptools本身就是这样做的，但是pip作为一个前端句柄依赖更好。）

另一个关键问题是准备一个清晰的环境，两种环境。 Python的virtualenv在这方面是上帝发送的，允许您使用自己选择的软件包配置完全自定义的Python环境，而无需root访问或操作系统软件包（rpm或dpkg），并且不受任何软件包和版本的约束它最终安装在您的发行版上。

最后，一个令人讨厌的bug是创建使用PYTHON_PATH播放不错的命令行脚本的难点。

设置

（为了简单起见，这是简单的

1. 准备一个工作目录，你的Python东西可以回家。


2. 从此页面的底部获取最新的virtualenv，并解压缩（不含



3. 从工作目录中，设置一个新的Python虚拟环境：

     $ python< untarred_directory> /virtualenv.py venv 
     




4. 你我想从这个虚拟环境中做大部分的工作。使用此命令执行此操作（。是源代码的快捷方式）：

     $。 venv / bin / activate 
     




5. 安装pip：

     $ easy_install pip 
     




6. 创建您要创建的每个可安装软件包的目录。



7. 在每个目录中，您需要一个定义软件包的内容和结构的setup.py。设置工具文档是开始使用此功能的非常好的资源。这是值得花时间吸收大块的。

开发

一旦你的树结构准备就绪，你几乎准备好开始编码了。但是，现在，依赖于彼此的软件包将不会在部署环境中看到对方。这个问题是通过一个简单的小工具来解决的，这个工具就是一些设置工具提供的，哪个点可以使用。对于您正在开发的每个包，请运行以下命令（确保您在项目的虚拟环境中，按照上述步骤3）：

  $ pip install -e pkg1 
  

此命令将安装 pkg1 到您的虚拟环境中，它不会复制任何文件。它只是添加指向包的开发根的 site-packages 目录的链接，并在该根目录中创建一个egg-info目录。您也可以这样做，不需要点数，如下所示：

  $ cd pkg1 
 $ python setup.py develop 
  

它通常可以工作，但如果您有第三方依赖项（应该在setup.py中列出） ，如在setuptools文档中的 here 所述），pip更聪明地找到它们。

需要注意的一点是，setuptools和pip都不会有任何关于在自己的包中查找依赖关系的智慧。如果目录B中的PkgB取决于目录A中的PkgA，则 pip install -e B 将失败，因为pip无法知道在目录A中可以找到PkgA ;它将反而尝试并且失败，从其在线存储库源下载PkgA。解决方法只是在依赖之后安装每个软件包。

此时，您可以启动python，加载其中一个模块并开始使用它。您可以编辑代码，并在下次导入时立即可用。

最后，如果要使用包创建命令行工具。不要用手写。你最终会遇到一个糟糕的PYTHON_PATH黑客，这些黑客从来没有正常工作。只需在setuptools文档中阅读自动脚本创建。

部署

当您的软件包准备采取行动时，您可以使用setup.py创建部署包。有这么多的选择进入这里，但以下应该让你开始：

  $ cd pkg1 
 $ python setup.py --help 
 $ python setup.py --help-commands 
  

松散结束

由于问题的广泛性质，这个答案一定是不完整的。我没有处理长时间运行的服务器，Web框架或实际的部署过程本身（特别是使用pip install的 - index-url 来管理私有存储库的第三方和内部软件包和 -e vcs + ... ，这将从svn，git，hg或bzr中提取软件包）。但我希望我给你足够的绳子把它们捆绑在一起（只是不要挂着自己： - ）。

We have several Python 2.6 applications running on Linux. Some of them are Pylons web applications, others are simply long-running processes that we run from the command line using nohup. We're also using virtualenv, both in development and in production. What is the best way to deploy these applications to a production server?

In development we simply get the source tree into any directory, set up a virtualenv and run - easy enough. We could do the same in production and perhaps that really is the most practical solution, but it just feels a bit wrong to run svn update in production. We've also tried fab, but it just never works first time. For every application something else goes wrong. It strikes me that the whole process is just too hard, given that what we're trying to achieve is fundamentally very simple. Here's what we want from a deployment process.

1. We should be able to run one simple command to deploy an updated version of an application. (If the initial deployment involves a bit of extra complexity that's fine.)
2. When we run this command it should copy certain files, either out of a Subversion repository or out of a local working copy, to a specified "environment" on the server, which probably means a different virtualenv. We have both staging and production version of the applications on the same server, so they need to somehow be kept separate. If it installs into site-packages, that's fine too, as long as it works.
3. We have some configuration files on the server that should be preserved (ie. not overwritten or deleted by the deployment process).
4. Some of these applications import modules from other applications, so they need to be able to reference each other as packages somehow. This is the part we've had the most trouble with! I don't care whether it works via relative imports, site-packages or whatever, as long as it works reliably in both development and production.
5. Ideally the deployment process should automatically install external packages that our applications depend on (eg. psycopg2).

That's really it! How hard can it be?

解决方案

Development and deployment of Python code is made much easier by setuptools in combination with virtualenv and pip.

Core ideas

The trickiest part, I've found, is running a development environment that mirrors the deployed setup as closely as possible, while at the same time respecting Pythonic tools and idioms. But it turns out that this is very easy to achieve with pip and setuptools, which together allow you to "install" a development tree into a Python environment without moving the files around. (Actually setuptools does this all by itself, but pip acts as a front end handle dependencies better.)

Another key issue is preparing a clean environment with a known package set across both environments. Python's virtualenv is a god-send in this respect, allowing you to configure a completely customised Python environment with your own choice of packages, without requiring root access, or OS packages (rpm or dpkg), and without being constrained by whatever packages and versions thereof that happen to be installed on your distro.

Finally, one annoying bug-bear is the difficulty of creating command-line scripts that play nice with PYTHON_PATH. This is also dealt with quite elegantly by setuptools.

Setting up

(To keep things simple, this is fairly prescriptive. Feel free to diverge as appropriate.)

1. Prepare a working directory that your Python stuff can call home.
2. Grab the latest virtualenv from the bottom of this page and untar it (doesn't matter where).

3. From the working directory, setup a new Python virtual environment:

   $ python <untarred_directory>/virtualenv.py venv
   

4. You'll want to do most of your work from inside this virtual environment. Use this command to do so (. is a shortcut for source):

   $ . venv/bin/activate
   

5. Install pip:

   $ easy_install pip
   

6. Create directories for each installable package you want to create.

7. In each directory, you'll need a setup.py that defines the content and structure of the package. The setuptools documentation is a very good resource for getting started on this. It's worth taking the time absorb large chunks of it.

Development

Once your tree structure is ready, you are almost ready to begin coding. But right now, packages that depend on each other can't see each other as they will in the deployed environment. This problem is resolved with a neat little trick that setuptools offers, and which pip makes use of. For each package you are developing, run the following command (make sure you are in the virtual environment for your project, as per step 3, above):

$ pip install -e pkg1

This command will install pkg1 into your virtual environment, and it does so without copying any of your files. It simply adds a link to the site-packages directory pointing to the package's development root and creates an egg-info directory in that root. You can also do this without pip, as follows:

$ cd pkg1
$ python setup.py develop

And it will usually work, but if you have third-party dependencies (which should be listed in setup.py, as explained here in the setuptools documentation), pip is smarter about finding them.

One caveat to note is that neither setuptools nor pip have any smarts about finding dependencies amongst your own packages. If PkgB in directory B, depends on PkgA in directory A, then pip install -e B will fail, because pip has no way of knowing that PkgA can be found in directory A; it will instead try, and fail, to download PkgA from its online repository sources. The workaround is simply to install each package after its dependencies.

At this point, you can start python, load up one of your modules and start toying with it. You can edit code, and it will be immediately available the next time you import it.

Finally, if you want to create command-line tools with your packages. Don't write them by hand. You'll end up with a horrible mess of PYTHON_PATH hacks that never quite works properly. Just read up on automatic script creation in the setuptools documentation. This will spare you a lot of grief.

Deployment

When your packages are ready for action, you can use setup.py to create deployment packages. There are way too many options to go into here, but the following should get you started:

$ cd pkg1
$ python setup.py --help
$ python setup.py --help-commands

Loose ends

Due to the broad nature of the question, this answer is necessarily incomplete. I haven't dealt with long-running servers, web frameworks or the actual deployment process itself (in particular, use of pip install's --index-url to manage a private repository of third-party and internal packages and -e vcs+..., which will pull packages out of svn, git, hg or bzr). But I hope I've given you enough rope to tie it all together (just don't hang yourself with it :-).

这篇关于一个简单的Python部署问题 - 一个痛苦的世界的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！