有没有办法将数据直接从 python 请求流到 minio 存储桶 [英] Is there a way to stream data directly from python request to minio bucket

问题描述

我正在尝试向服务器发出 GET 请求以检索 tiff 图像.然后我想使用 MinIO python SDK 中的 put_object 方法将它直接流式传输到 MinIO.

I am trying to make a GET request to a server to retrieve a tiff image. I then want to stream it directly to MinIO using the put_object method in the MinIO python SDK.

我知道我可以通过将图像保存到一个临时文件，然后上传来做到这一点，但我想看看我是否可以跳过这一步.

I know I could do this by saving the image to a temp file, then uploading but I wanted to see if I could skip that step.

我尝试直接插入字节响应并使用 BytesIO 对其进行包装，但我想我遗漏了一些东西.

I've tried inserting the byte response directly and using BytesIO to wrap it but I think I am missing something.

r = requests.get(url_to_download, stream=True)
Minio_client.put_object("bucket_name", "stream_test.tiff", r.content, r.headers['Content-length'])

我返回以下错误

AttributeError: 'bytes' 对象没有属性 'read'

AttributeError: 'bytes' object has no attribute 'read'

非常感谢任何帮助！

推荐答案

阅读 关于put_object 的 MinIO 文档，有一些示例如何将新对象添加到对象存储服务器.这些示例仅说明了如何添加文件.

Reading documentation on MinIO about put_object, there are examples how to add a new object to the object storage server. Those examples only explain how to add a file.

这是put_object函数的定义:

put_object(bucket_name, object_name, data, length, content_type='application/octet-stream', metadata=None, progress=None, part_size=510241024)

我们对 data 参数感兴趣.它指出:

We are interested in data parameter. It states:

任何实现 io.RawIOBase 的 Python 对象.

Any python object implementing io.RawIOBase.

RawIOBase 是原始二进制 I/的基类哦.它还定义了方法read.

RawIOBase is base class for raw binary I/O. It also defines method read.

如果我们使用 dir() 内置函数尝试返回 r.content 的有效属性列表，然后我们可以检查 read 是否存在:

If we were to use dir() built-in function to attempt to return a list of valid attributes for r.content, we could then check if read is there:

'read' in dir(r.content) -> return False

'read' in dir(r.content) -> return False

这就是您得到 AttributeError: 'bytes' object has no attribute 'read' 的原因.这是因为 type(r.content) 是 bytes 类.

That's the reason why you get AttributeError: 'bytes' object has no attribute 'read'. It's because type(r.content) is bytes class.

您可以将r.content 转换为从RawIOBase 继承的类.也就是说，使用 io.BytesIO 类.要以字节为单位获取对象的大小，我们可以使用 io.BytesIO(r.content).getbuffer().nbytes.

You can convert r.content into class that inherits from RawIOBase. That is, using io.BytesIO class. To get size of an object in bytes, we could use io.BytesIO(r.content).getbuffer().nbytes.

因此，如果您想将原始数据字节流式传输到存储桶，请将 bytes 类转换为 io.BytesIO 类:

So if you want to stream raw bytes of data to your bucket, convert bytes class to io.BytesIO class:

import io
import requests

r = requests.get(url_to_download, stream=True)
raw_img = io.BytesIO(r.content)
raw_img_size = raw_img.getbuffer().nbytes

Minio_client.put_object("bucket_name", "stream_test.tiff", raw_img, raw_img_size)

<小时>

注意:示例显示了从文件中读取二进制数据并通过从使用 返回的 stat_result 中读取 st_size 属性来获取其大小>os.stat() 函数.

---

NOTE: Examples show reading binary data from file and getting its size by reading st_size attribute from stat_result which is returned by using os.stat() function.

st_size 等价于 io.BytesIO(r.content).getbuffer().nbytes.

这篇关于有没有办法将数据直接从 python 请求流到 minio 存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！