我可以从AWS s3恢复下载吗? [英] Can I resume a download from aws s3?

问题描述

我正在使用python boto3库将文件从s3下载到蜂窝连接上的IOT设备上，该连接通常很慢且不稳定.

I am using the python boto3 library to download files from s3 to an IOT device on a cellular connection which is often slow and shaky.

某些文件很大(250Mb，在这种情况下该文件很大)，并且网络出现故障，并且下载时设备重新启动.

Some files are quite large (250Mb, which for this scenario is large) and the network fails and the device reboots while downloading.

我想从重新启动设备时结束的地方恢复下载.有什么办法吗?

I would like to resume the download from the place it ended when the device rebooted. Is there any way to do it?

中止的下载似乎在下载时将下载的数据保存在一个临时文件中，因此数据在那里.

The aborted download does seem to keep downloaded data in a temporary file while downloading so the data is there.

目标是节省数据传输，并使下载更具弹性.

The goal is to economize data transfer and make the download more resilient.

我正在使用分段上传，但本身没有简历.

I am using multipart uploads but no resume happens by itself.

我正在做的事情是这样的:

What i'm doing is something like this:

s3 = boto.resource('s3')
session = boto.session.Session(region_name='eu-central-1', profile_name=profile)
s3client = session.client( 's3', config=boto.session.Config(signature_version='s3v4'))
MB = 1024 ** 2

config = TransferConfig(
    multipart_threshold=10*MB,
    num_download_attempts=100)

def upload():
    s3client.upload_file(Filename=localfile, Bucket=bucket, Key=key, Config=config)

def download():
    s3client.download_file(bucket, key, localfile, Config=config )

# upload from server...
upload()

# .... later, from IOT device
download()

推荐答案

我不认为boto3具有可恢复的下载功能.

I don't believe that boto3 has a resumable download feature.

您可以通过使用远程获取来实现自己.使用

You could potentially implement one yourself by making use of ranged gets. Find the size of the object upfront using head_object, then split that into N ranges, download them individually (maybe K chunks in parallel, depending on your hardware), store them on the local file system as chunks, and re-compose them into the final download when all chunks complete.

response = client.get_object(
    Bucket='mybucket',
    Key='mykey',
    Range='bytes=10001-20000'
)

这篇关于我可以从AWS s3恢复下载吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！