default_storage.exists极其缓慢，经常超时 [英] default_storage.exists extremely slow and frequently times out

问题描述

我们刚刚迁移一个Django项目，Heroku和（使用Django，存储器和S3-博托）把我们所有的媒体和静态文件在Amazon S3的。

We have just migrated a Django project to Heroku and put all of our media and static files on Amazon S3 (using django-storages and s3-boto).

尽管一切，我听说过Amazon S3的速度非常快，并转向了很少的结果在性能降低，我们的图片加载已经放缓至一个绝对的抓取，并经常超时。的code，它是超时的一个例子是这个属性对我们的车型之一，它试图选择一个合适的图像，最终回落至返回无：

Despite everything I've heard about Amazon S3 being very fast, and turning up very few results on slow performance, our image loading has slowed to an absolute crawl and frequently times out. An example of the code that is timing out is this property on one of our models, which tries to select an appropriate image, ultimately falling back to returning None:

@property
def photo(self):
    """Transparently serve the best available image for templates"""
    if self.model_shot.storage.exists(self.model_shot.name):
        return self.model_shot
    elif self.image.storage.exists(self.image.name):
        return self.image
    else:
        return None

在我测试的这是造成问题的一个模式，我想这样的：

When I tested on a model which was causing problems, I tried this:

$ heroku run python manage.py shell
...
>>> design = Design.objects.get(pk=10210)
>>> design.photo

此命令导致壳，最后返回一个 ImageFieldFile 对象挂几秒钟。随后调用它立即返回，这使我相信结果会被缓存。

This command caused the shell to hang for several seconds before finally returning an ImageFieldFile object. Subsequent calls to it returned instantaneously, which makes me believe the result is cached.

我的问题是，什么是处理这一目标的最佳方式是什么？我听到了很多关于在这样的情况下使用CloudFront的，但这绝对不是由于高流量（不应该基本上在我们网站上的任何流量还）。其他的一些缓存框架？别的东西完全？

My question is, what is the best way to handle this? I have heard a lot about using CloudFront in situations like this, but this is definitely not due to high traffic (shouldn't have basically any traffic on our site yet). Some other caching framework? Something else entirely?

最有问题的图像是1000×1000像素至少。

Most of the images in question are 1000x1000 px at least.

推荐答案

切换到CloudFront的彻底解决这个问题，而且是相对容易（无需code的变化只是多与亚马逊控制台胡闹左右），所以我决定回答我的问题。

Switching to CloudFront completely solved this issue, and was relatively easy (no code changes just more monkeying around with the Amazon Console), so I decided to answer my own question.

步骤0：如果您还没有准备好，确保你的水桶名符合的最佳实践命名桶。他们并不一定能使这个明显的在所有的地方，他们应该，但一个不好斗的名字完全可以打破它的互操作性与其他Amazon Web服务。做的最好的事情就是命名你的水桶东西全部小写，这不是太长（小于= 60字左右）。

Step 0: If you haven't already, make sure your bucket name complies with the "best practices" for naming buckets. They don't necessarily make this obvious in all the places they should, but a bad bucket name can completely break its interoperability with other Amazon Web Services. The best thing to do is name your bucket something all lowercase that's not too long (<= 60 characters or so).

第1步：为了获得CloudFront的，从你的水桶提供文件，你需要将其设置为，如果服务于静态的网站。你可以从你的水桶的权限标签亚马逊AWS控制台上做到这一点。亚马逊有几个地方有说明/此文件; IMO最明显的是这些。 重要的：请确保您设置了默认根对象为 index.html的 - 该文件不连必须存在，但设置的确实的。

Step 1: In order to get CloudFront to serve files from your bucket, you need to set it up as if to serve a static website. You can do this on the Amazon AWS console from your bucket's Permissions tab. Amazon has several places where there are instructions/documentation for this; IMO the clearest are these. IMPORTANT: Make sure you set up the Default Root Object to index.html -- that file doesn't even have to exist, but that setting does.

步骤1.5 [可能可选]：确保在你的水桶权限是否正确。即使我的发球文件从S3没有问题，更改为CloudFront的为他们服务变成一切都变成 403：禁止访问错误。如果有疑问，你的文件是不敏感的，你可以右键点击你的水桶在AWS控制台的文件夹，然后点击设为公开。 的警告的：这可能是一个非常耗费时间的过程中，对于一些愚蠢的原因（即使它是服务器端），您的浏览器会话已经留打开。为此，首先，不要关闭您的会话。对于我们的水桶，这花了约16小时。 ：/

Step 1.5 [possibly optional]: Make sure the permissions on your bucket are correct. Even though I was serving files from S3 no problem, changing to CloudFront to serve them turned everything into a 403: Access Forbidden error. If in doubt, and your files are not sensitive, you can right click on folders of your bucket in the AWS Console and click Make Public. WARNING: This can be a very time intensive process, and for some stupid reason (even though it's server side) your browser session has to stay open. Do this first and don't close your session. For our bucket, this took about 16 hours. :/

第二步：转到在AWS控制台中的亚马逊的CloudFront的部分，然后点击创建通讯按钮。使之成为Web分布（默认值），并使用通过设置你的水桶在previous步骤为原点静态网页分配所产生的域。此外，国际海事组织，这些是在AWS文档清晰，最直接的说明。你可以一走了之一切默认位置。一旦它的创建，只是等待，直到它的上市控制台的部署的。

Step 2: Go to the Amazon CloudFront section in the AWS Console and click the Create Distribution button. Make it a web distribution (default) and use the domain you generated by setting your bucket up for static web distribution in the previous step as the origin. Again, IMO, these are the clearest and most straightforward instructions in the AWS docs. You can leave just about everything default here. Once it's created, just wait until it's listed on the console as "Deployed".

第三步：配置你的应用程序从CloudFront的，而不是S3服务。这是最简单的部分，因为URL是从 https://bucketname.s3.amazonaws.com/path 透明地移动到 https：//开头somerandomstring .cloudfront.net /路径（奖金：您可以设置后者为 CNAME 记录指向类似 media.yourdomain.tld ，我们没有这样做，所以我不打算在这里）。由于我使用的Django与 Django的货仓和 S3-博托的组合，这最终是一个简单的的事情建立在 settings.py 的Cloudfront域：

Step 3: Configure your app to serve from CloudFront rather than S3. This is the easiest part because the URLs are transparently moved from https://bucketname.s3.amazonaws.com/path to https://somerandomstring.cloudfront.net/path (bonus: you can set up the latter as a CNAME record to point to something like media.yourdomain.tld; we didn't do this so I won't go into it here). Since I'm using Django with a combination of django-storages and s3-boto, this ended up being a simple matter of setting up that Cloudfront domain in settings.py:

AWS_S3_CUSTOM_DOMAIN = 'd2ynhpzeiwwiom.cloudfront.net'

这就是它！有了这些变化，我们所有的速度困境走了，我们的富媒体页（每页的图像6-20 MP值）突然负荷比以往更快！

And that's it! With these changes, all of our speed woes went away, and our media-rich pages (6-20 MP worth of images per page) suddenly load faster than ever!

这篇关于default_storage.exists极其缓慢，经常超时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！