为什么多线程 python 程序在 ec2 微实例上运行缓慢? [英] Why multi-threaded python program slow on ec2 micro-instance?

问题描述

我正在开发 Online Judge 代码检查器.我的代码在 python 2.7 中使用多线程.我本地机器上的同一个程序(我的核心 3 RAM 4GB)在 1 分 10 秒内评估了大约 1000 个提交.但是当我在 ec2 微型实例(大约 600 MB RAM)上运行它时，大约需要40 分钟(随机数秒变慢).要知道我把事情搞砸的原因.

I am working on a Online Judge code checker.My code uses multi-threading in python 2.7.The same program on my local machine (i core 3 RAM 4GB) evaluates about 1000 submisions in 1 minute 10 seconds. But when I run it on ec2 micro instance(about 600 MB RAM) it takes about 40 minutes(It gets slow for some random seconds).To know the reason I broke down things.

1. 首先这是我的评估器的工作方式:
   

1. First this is how my evaluator works:
   

• 我有一个主程序 worker.py ，它创建了多个线程
• 主线程从文件(暂时)中提取提交(一次 10 个)并将它们放入全局队列中
• 侧线程从队列中获取提交(一个提交仅由一个线程评估)
• 在一个侧线程接受一个提交之后，它将它发送到一个函数compile，它将提交的可执行文件返回给该线程
• 然后线程将此可执行文件发送到运行该可执行文件的函数 run(使用具有定义内存和时间限制的沙箱)并将可执行文件的输出写入文件，然后检查它
  针对标准输出
• 队列清空后，主线程再次拉取 10 个提交并将它们放入队列
  
  

• I have a main program worker.py , which creates multiple threads
• The main thread pulls submissions(10 at a time) from a file(for time being) and puts them in a global queue
• The side threads take submisions from queue(one submission evaluated solely by one thread)
• After a side thread takes a submission it sends it to a function compile,which returns the executable of submission back to that thread
• Then the thread sends this executable to a function run which runs the executable (using sandbox with defined memory and time limits) and writes the output of the executable to file and then checks it
  against standard output
• After the queue gets empty the main thread again pulls 10 submissions and places them in queue
  
  

compile 和 run 函数:

• 编译函数和运行函数将可执行文件和输出保存在文件中(分别)命名为像 .exe 和 .txt 这样每个线程都有自己的文件并且不存在覆盖问题.
• 只有当编译函数的状态正常(编译的文件)时，线程才会去运行函数，否则抛出编译错误该提交
  
  

• The compile function and run function save the executable and output in files(repectively) named like <thread_Name>.exe and <thread_Name>.txt so that every thread has its own files and there is no issue of overwriting.
• A thread goes to run function only if status from compile function was OK(the file compiled)otherwise throws compile error for that submission
  
  

现在我的疑惑:

• 在ec2上执行缓慢的问题是因为它拥有的资源还是由于到 python 的多线程.在我的脚本中，线程访问全局变量例如队列(我加锁)和 test.py(我不加锁) 在运行函数用标准输出逐字符检查输出(vimdiff 之类)，和 mysandbox.py(libsandbox the sandbox)和其他一些全局变量.因此，由于 python 的 GIL，工作缓慢.如果它那么为什么它在我的本地机器上运行得很快.
• 暂时我也给出了相同的文件test.cpp(添加两个数字并打印结果)1000 次.所以当我故意在这个文件中编译错误并运行我的主要ec2 上的程序运行得非常快.从那我推断编译和我的程序的运行(编译和运行函数)占用了主要时间，而不是线程创建和管理.

• Is the problem of slow execution on ec2 due to the resources it has or due to multi-threading of python.In my scripts the threads to access global variables such as the queue(i put locks) and test.py(I dont put lock on it) which in run function checks the output with standard output character by character(vimdiff like), and mysandbox.py(libsandbox the sandbox) and some other global variables.So is the slow working due to GIL of python.If it is so then why does it work fast on my local machine.
• Also for time being I give the same file test.cpp(adds two numbers and prints result) 1000 times.So when I purposely make a compile error in this file and run my main program on ec2 it runs pretty fast.From that I deduced that the compiling and and running(compile and run functions) of my program take the main time,not the thread creation and management.

我知道这是一个很大的问题，但真的很感激任何帮助(否则我将不得不继续悬赏我所有的声誉:)).

I know its a vast question but any help is really appreciated(or i will have to keep bounty on it betting all my reputation :) ).

推荐答案

对于持续计算任务(按设计)，微实例变得极其缓慢.

Micro instances become extremely slow for sustained computational tasks (by design).

您将代码编写为多线程，以利用整个机器"的 CPU 资源执行文件检索和编译等任务，这是提高性能的好做法.

You wrote your code to be multi-threaded to take advantage of the entire "machine's" CPU resources for tasks like file retrieval and compilation, which is good practice for performance.

虽然这在您保证配置硬件资源的物理机或虚拟机上有意义，但由于亚马逊分配资源的方式，它在微实例上没有意义.

And while this makes sense on a physical machine or a virtual machines where you have guaranteed provisioned hardware resources, it doesn't makes sense on a micro instance due to the way resources are allocated by Amazon.

根据亚马逊的文档，微实例仅设计用于 CPU 短时突发操作，因此如果您尝试使用占用 CPU 使用率的多个线程，则会遇到 Amazon 自身施加的巨大瓶颈:

Per Amazon's documentation, micro instances are designed for short-burst CPU operations only and will therefore experience huge bottlenecks imposed by Amazon itself if you try to use multiple threads that eat CPU usage:

如果应用程序消耗的 CPU 资源超过您的实例分配的 CPU 资源，我们会暂时限制该实例，使其在较低的 CPU 级别上运行.如果您的实例继续使用其分配的所有资源，其性能将会下降.我们将增加限制其 CPU 级别的时间，从而增加允许实例再次爆发之前的时间.

If the application consumes more than your instance's allotted CPU resources, we temporarily limit the instance so it operates at a low CPU level. If your instance continues to use all of its allotted resources, its performance will degrade. We will increase the time that we limit its CPU level, thus increasing the time before the instance is allowed to burst again.

查看我刚刚链接到的文档中的 CPU 使用率图表以获取更多详细信息.

Take a look at the CPU usage graphs in the documentation I just linked to to get more details.

为了证明这是问题所在，您可以简单地启动一个小实例并在那里运行您的判断软件 - 您应该会体验到类似于台式机的显着改进.

To prove that this is the issue, you could simply launch a small instance instead and run your judge software there - you should experience a dramatic improvement similar to your desktop machine.

TL;DR 在微型实例上尝试使用持续 CPU 时，它的功能可能不如旧的 Palm Treo.

这篇关于为什么多线程 python 程序在 ec2 微实例上运行缓慢?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！