当使用s3，boto，django-storage进行静态文件时，由对等体重新连接 [英] Connection reset by peer when using s3, boto, django-storage for static files

问题描述

我正在尝试切换到使用amazon s3来托管我们的django项目的静态文件。我使用django，boto，django-storage和django-compression。当我在我的开发服务器上运行收集静态时，我得到错误

  socket.error：[Errno 104] 
  

我的所有静态文件的大小是74MB，这似乎太大了。有没有人看过这个，或有任何调试提示？

这是完整的跟踪。

 追溯（最近的最后一次呼叫）：
文件./manage.py，第10行，< ;模块> 
 execute_from_command_line（sys.argv）
文件/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py，第443行，execute_from_command_line 
 utility.execute（）
文件/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py，第382行，执行
 self.fetch_command （子命令）.run_from_argv（self.argv）
文件/usr/local/lib/python2.7/dist-packages/django/core/management/base.py，第196行，在run_from_argv 
 self.execute（* args，** options .__ dict__）
文件/usr/local/lib/python2.7/dist-packages/django/core/management/base.py，第232行，执行
 output = self.handle（* args，** options）
文件/usr/local/lib/python2.7/dist-packages/django/core/management/base.py，行371，在句柄
返回self.handle_noargs（**选项）
文件/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic .py，第163行，在handle_noargs 
 colle中cted = self.collect（）
文件/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py，第113行，收集
 handler（path，prefixed_pa​​th，storage）
文件/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py，第303行， copy_file 
 self.storage.save（prefixed_pa​​th，source_file）
文件/usr/local/lib/python2.7/dist-packages/django/core/files/storage.py，第45行，在save 
 name = self._save（name，content）
文件/usr/local/lib/python2.7/dist-packages/storages/backends/s3boto.py，第392行， _save 
 self._save_content（key，content，headers = headers）
文件/usr/local/lib/python2.7/dist-packages/storages/backends/s3boto.py，第403行，在_save_content 
 rewind = True，** kwargs）
文件/usr/local/lib/python2.7/dist-packages/boto/s3/key.py，第1222行，set_contents_from_file 
 chunked_transfer = chunked_transfer，siz e = size）
文件/usr/local/lib/python2.7/dist-packages/boto/s3/key.py，第714行，在send_file 
 chunked_transfer = chunked_transfer，size = size ）
文件/usr/local/lib/python2.7/dist-packages/boto/s3/key.py，第890行，_send_file_internal 
 query_args = query_args 
文件/ usr / local / lib / python2.7 / dist-packages / boto / s3 / connection.py，第547行，make_request 
 retry_handler = retry_handler 
文件/ usr / local / lib / python2 7 / dist-packages / boto / connection.py，第966行，make_request 
 retry_handler = retry_handler）
文件/usr/local/lib/python2.7/dist-packages/boto/connection .py，第927行，_mexe 
 raise e 
 socket.error：[Errno 104]连接由对等体
  

更新：我没有解决如何调试此错误的答案，但后来这只是停止发生，这让我觉得它可能与S3上的某些事情有关。 / strong>

解决方案

tl; dr

如果您的存储区不在默认区域，则需要告知boto要连接的区域，例如。如果您的存储桶位于我们-west-2中，则需要将以下行添加到settings.py：

  AWS_S3_HOST ='s3 -us-west-2.amazonaws.com'
  

长说明：

这不是一个权限问题，您不应将桶权限设置为已验证的用户。

如果创建桶，则会出现此问题在一个不是默认的区域（在我的情况下，我使用的是us-west-2）。

如果您不使用默认区域，并且不知道Buto在哪个区域中存在，boto将连接到默认区域，S3将回复一个307重定向到桶属于的区域。

不幸的是，由于boto中的这个错误：

https://github.com/boto/boto/issues/2207

如果307回复在boto完成上传文件之前到达，boto将不会看到重定向，并将继续上传到默认区域。
最终，S3关闭套接字，导致由对等体重新连接。

这是一种竞争条件，取决于上传的对象的大小和您的互联网连接的速度，这就解释了为什么它会随机发生。

有两个可能的原因，OP停止在一段时间后看到错误：

   - 他后来在默认区域中创建了一个新的桶，问题本身就消失了。 
  - 他开始上传只有小的文件，足够快，以便在S3回复307 
  

时完全上传

I'm trying to switch to use amazon s3 to host our static files for our django project. I am using django, boto, django-storage and django-compressor. When I run collect static on my dev server, I get the error

socket.error: [Errno 104] Connection reset by peer 

The size of all of my static files is 74MB, which doesnt seem too large. Has anyone seen this before, or have any debugging tips?

Here is the full trace.

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 163, in handle_noargs
    collected = self.collect()
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 113, in collect
    handler(path, prefixed_path, storage)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 303, in copy_file
    self.storage.save(prefixed_path, source_file)
  File "/usr/local/lib/python2.7/dist-packages/django/core/files/storage.py", line 45, in save
    name = self._save(name, content)
  File "/usr/local/lib/python2.7/dist-packages/storages/backends/s3boto.py", line 392, in _save
    self._save_content(key, content, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/storages/backends/s3boto.py", line 403, in _save_content
    rewind=True, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1222, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 714, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 890, in _send_file_internal
    query_args=query_args
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 547, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 966, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 927, in _mexe
    raise e
socket.error: [Errno 104] Connection reset by peer

UPDATE: I don't have the answer to how to debug this error, but later this just stopped happening which makes me think it may have to do with something on S3.

解决方案

tl;dr

If your bucket is not in the default region, you need to tell boto which region to connect to, e.g. if your bucket is in us-west-2 you need to add the following line to settings.py:

 AWS_S3_HOST = 's3-us-west-2.amazonaws.com'

Long explanation:

It's not a permission problem and you should not set your bucket permissions to 'Authenticated users'.

This problem happens if you create your bucket in a region which is not the default one (in my case I was using us-west-2).

If you don't use the default region and you don't tell boto in which region your bucket resides, boto will connect to the default region and S3 will reply with a 307 redirect to the region where the bucket belongs.

Unfortunately, due to this bug in boto:

https://github.com/boto/boto/issues/2207

if the 307 reply arrives before boto has finished uploading the file, boto won't see the redirect and will keep uploading to the default region. Eventually S3 closes the socket resulting into a 'Connection reset by peer'.

It's a kind of race condition which depends on the size of the object being uploaded and the speed of your internet connection, which explains why it happens randomly.

There are two possible reasons why the OP stopped seeing the error after some time:

- he later created a new bucket in the default region and the problem went away by itself. 
- he started uploading only small files, which are fast enough to be fully uploaded by the time S3 replies with 307

这篇关于当使用s3，boto，django-storage进行静态文件时，由对等体重新连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！