错误处理：宝途：[错误104]被连接重置 [英] Error Handling: Boto: [Error 104] Connection Reset by Peer

问题描述

我有一个脚本，下载从Amazon S3。脚本工作99.9％的时间。偶尔我得到以下错误（socket.error：[错误104]将连接复位）。一旦我重新启动code中的错误似乎消失。因为它很难重现错误。我希望的code中的文档片断下方会修正这个错误。具体而言，我希望如果错误出现，它会尝试重新下载该文件。我想知道，如果这code将工作，如果有什么事我应该加入。我想一个错误计数器可能是很好的，所以如果错误不断来了，它会最终继续前进。 （不完全​​知道如何添加计数器）

 文件= [#list文件下载]

对于文件中的文件：
    在bucket.list键（preFIX =文件）：
        而真正的：
            尝试：
                keys.get_contents_to_filename（）
            除了socket.error：
                继续
            打破
 

解决方案

我有完全相同的问题

。如果您搜索博托在GitHub上，你会看到，我们并不孤独。

还有一个已知的问题接受： https://github.com/boto/boto/issues / 2207

AWS S3的

接触性能极限

事实是，我们得到了这样用博托和AWS S3服务，我们都忘记了，这些都是真正的分布式系统，这可能会在某些情况下打破。

我归档（下载，焦油，上传）的文件数量庞大（约3年约15张进为每天约1440版本），并使用芹菜做到这一点更快。我不得不说，我是某个时候得到这些错误次数多，可能达到AWS S3的性能极限。这些错误往往出现在块（在我的情况，我的几个小时上传约60 Mbps的）。

培训S3的性能

当我衡量性能，它被训练。有些小时后，S3存储的响应跳起来，AWS可能已经检测到更高的负载和自旋一些更多的情况下，为它服务。

请尝试最新稳定版本博托

另一件事是，博托正试图在许多情况下重试，所以很多故障都是隐藏的，以我们的呼吁。有时我有点更好地升级到最新的稳定版本。

我的结论是：

• 尝试升级到最新的稳定博托
• 当错误率长大，降低了pressure
• 接受这样的事实，即有罕见的性能问题AWS S3的分布式服务

在你的code，我肯定会推荐增加一些睡眠（至少5％，但3​​0秒似乎没什么问题），否则你只是推难当系统，这可能是在摇摇欲坠局面的时刻。

I have a script that downloads from Amazon S3. The scripts works 99.9% of the time. Occasionally I get the following error (socket.error: [Errno 104] Connection reset by peer). Once I restart the code the error seems to go away. Since its hard to recreate the error. I'm hoping the snipped of code below will fix the error. Specifically, I'm hoping if the error comes up, it'll try to re-download the file. I'm wondering if this code will work, and if there is anything else I should add in. I'm thinking an error counter might be good, so if the error does keep coming up it'll eventually move on. (Not exactly sure how to add a counter)

files = [#list of files to download]

for file in files:
    for keys in bucket.list(prefix=file):
        while True:
            try:
                keys.get_contents_to_filename()
            except socket.error:
                continue
            break

解决方案

I had exactly the same problem. If you search boto on GitHub, you will see, we are not alone.

There's also a known accepted issue: https://github.com/boto/boto/issues/2207

Reaching performance limits of AWS S3

The truth is, that we got so used to boto and AWS S3 service, we have forgotten, these are really distributed systems, which might break in some cases.

I was archiving (download, tar, upload) huge number of files (about 3 years with around 15 feeds each having about 1440 versions a day) and using Celery to do this faster. And I have to say, that I was sometime getting these errors more often, probably reaching performance limits of AWS S3. These errors were often appearing in chunks (in my case I was uploading about 60 Mbps for couple of hours).

Training S3 performance

When I was measuring performance, it was "trained". After some hour, the responsiveness of S3 bucket jumped up, AWS have probably detected higher load and spin up some more instances serving it.

Try latest stable version of boto

Other thing is, that boto is trying to retry in many cases, so many failures are hidden to our calls. Sometime I got a bit better with upgrading to the latest stable version.

My conclusion are:

• try upgrading to the latest stable boto
• when error rate grows up, lower the pressure
• accept the fact, that AWS S3 is distributed service having rare performance problems

In your code, I would definitely recommend adding some sleep, (at least 5, but 30 s would seem fine to me), otherwise you are just pushing harder and harder to a system, which might be in shaky situation at the moment.

这篇关于错误处理：宝途：[错误104]被连接重置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！