在django中将图像和缩略图存储在s3上 [英] Storing images and thumbnails on s3 in django

问题描述

我正在尝试使用django-storages，boto和sorl-thumbnail将我的图像缩小并存储在s3上。我有它的工作，但它很慢，即使是小图像。我不介意，当我保存表单并将图像上传到s3时，速度很慢，但我希望在此之后快速显示图像。

这个SO问题的答案解释说，直到首次访问之前，缩略图才会被创建，但是可以使用get_thumbnail（）事先创建它。

Django + S3（boto）+ Sorl缩略图：优化建议

我在做，现在似乎创建了thumbnail_kvstore表中的所有条目当上传图像时，而不是显示图像。

问题是显示图像的页面仍然很慢。查看调试工具栏中的日志记录面板，看起来还是有很多与s3的通信。看起来像上传和缓存图像和缩略图后，页面应该快速呈现，而无需与s3通信。

我做错了什么？谢谢！

更新：weak hack似乎已经开始工作，但我很想知道如何正确地执行此操作： p>

https://github.com / asciitaxi / sorl-thumbnail / commit / 545cce3f5e719a91dd9cc21d78bb973b2211bbbf

更新：@sorl的更多信息

我正在使用两个视图：

添加视图：在此视图中，我提交表单以创建其中的图像模型。图像上传到s3。在post_save信号中，我需要调用get_thumbnail（）来生成缩略图：

  im = get_thumbnail（instance.image， '360x360'）
  

显示视图：在此视图中，我显示在添加视图中生成的缩略图： / p>

  {％thumbnail object.image360x360as im％} 
< img src ={{im。 url}}width ={{im.width}}height ={{im.height}}> 
 {％endthumbnail％} 
  

没有补丁：

添加视图：在kvstore表中创建3个条目，访问高速缓存10次（6组，4次获取），调试工具栏的日志记录选项卡显示建立HTTP连接12次

显示视图：kvstore表中只有3个条目，只有1个缓存，但调试工具栏中显示建立HTTP连接3次仍然

只有在第122行的更改：

添加视图：与上述相同，但日志记录仅说建立HTTP连接2次
显示视图：与上述相同，但日志记录仅说建立HTTP连接1次

同时在行118上添加更改：

添加视图：与上述相同，但现在我们至少2个建立HTTP连接消息
显示视图：与上述相同，没有记录消息

更新：看起来像storage._setup（）被调用两次，并且storage.url（）被调用一次。根据时间，我会说每个人都连接到s3：

  1304711315.4 
 _setup 
 1304711317.84 
 1304711317.84 
 _setup 
 1304711320.3 
 1304711320.39 
 _url 
 1304711323.66 
  

这似乎被boto日志记录反映出来，它说建立HTTP连接3次。

解决方案

作为sorl缩略图的作者，我真的很想解决这个问题，如果它不按照我的意图工作。如果键值sotre被填充，它将当前存储：名称，存储和大小。我已经假设网址是基于该名称，因此不应该导致任何存储呼叫。查看django存储库， https：// github.com/e-loue/django-storages/blob/master/storages/backends/s3boto.py#L214 似乎是一个安全的假设。在你的补丁中，由于某些原因您已经修补了读取方法。当创建缩略图时，ImageFile实例从缓存中获取（如果不是创建它），那么您当然可以调用read来读取该文件，但预期的用途是.url，它使用缓存的名称调用url，其中应该是inturn成为非存储访问操作。你可以尝试将你的问题隔离在你的代码中，这个存储访问发生在哪里？

还要确保你有THUMBNAIL_DEBUG，你的密钥值存储正确设置

I'm trying to get my images thumbnailed and stored on s3 using django-storages, boto, and sorl-thumbnail. I have it working, but it's very slow, even with small images. I don't mind it being slow when I save the form and upload the images to s3, but I'd like it to display the image quickly after that.

The answer to this SO question explains that the thumbnail won't be created until first access, but that you can use get_thumbnail() to create it beforehand.

Django + S3 (boto) + Sorl Thumbnail: Suggestions for optimisation

I'm doing that, and now it seems that all entries into the thumbnail_kvstore table are created when uploading the image, rather than when it is displayed.

The problem is that the page displaying the image is still really slow. Looking at the logging panel in the debug toolbar, it looks like there is still lots of communication with s3. It seems like after the image and thumbnails are uploaded and cached, page should render quickly without communicating with s3.

What am I doing wrong? Thanks!

Update: weak hack seems to have gotten it working, but I'd love to know how to do this properly:

https://github.com/asciitaxi/sorl-thumbnail/commit/545cce3f5e719a91dd9cc21d78bb973b2211bbbf

Update: more information for @sorl

I'm working with 2 views:

ADD VIEW: In this view I submit the form to create the model with the image in it. The image is uploaded to s3. In a post_save signal, I call get_thumbnail() to generate the thumbnail before it's needed:

im = get_thumbnail(instance.image, '360x360')

DISPLAY VIEW: In this view I display the thumbnail generated in the add view:

    {% thumbnail object.image "360x360" as im %}
    <img src="{{ im.url }}" width="{{ im.width }}" height="{{ im.height }}">
    {% endthumbnail %}

Without the patch:

ADD VIEW: creates 3 entries in the kvstore table, accesses the cache 10 times (6 sets, 4 gets), logging tab of debug toolbar says "establishing HTTP connection" 12 times

DISPLAY VIEW: still just 3 entries in the kvstore table, just 1 get from cache, but debug toolbar says "establishing HTTP connection" 3 times still

With only the change on line 122:

ADD VIEW: same as above, except the logging only says "establishing HTTP connection" 2 times DISPLAY VIEW: same as above, except the logging only says "establishing HTTP connection" 1 time

Also adding the change on line 118:

ADD VIEW: same as above, but now we are down to 2 "establishing HTTP connection" messages DISPLAY VIEW: same as above, with no logging messages at all

UPDATE: It looks like storage._setup() is called twice, and storage.url() is called once. Based on the timing, I'd say each one makes connections to s3:

1304711315.4
_setup
1304711317.84
1304711317.84
_setup
1304711320.3
1304711320.39
_url
1304711323.66

This seems to be reflected by the boto logging, which says "establishing HTTP connection" 3 times.

解决方案

As the author of sorl thumbnail I am really interested in solving this if it is not working as I intended. If the key value sotre is populated it will currently store: name, storage and size. I have made the assumption that the url is based on the name and thus should not cause any storage calls. Looking at django storages, https://github.com/e-loue/django-storages/blob/master/storages/backends/s3boto.py#L214 it seems like a safe assumption to make. In your patch you have patched the read method for some reason. When creating a thumbnail a ImageFile instance is fetched from cache (if not create it) then you can of course call read which will read the file, but the intended use is .url which calls url on the storage with the cached name which inturn should be a non storage access op. Could you try to isolate your problem to exacly where in your code this storage access happends?

Also make sure you have THUMBNAIL_DEBUG on and that you have the key value store properly set up.

这篇关于在django中将图像和缩略图存储在s3上的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！