S3ToSFTP:将多个文件从同一S3密钥移到SFTP路径 [英] S3ToSFTP: Move multiple files from same S3 key to SFTP path

问题描述

要求:将多个文件从同一S3密钥移至SFTP

Requirement: Move multiple files from the same S3 Key to SFTP

下面是代码的一部分，我能够实现将一个文件移动到SFTP位置. 如果s3_key位置具有以下多个文件示例，则需要将这两个文件都从/path/output获取到SFTP位置

Below is the part of the code, I was able to achieve moving one file into SFTP location. If the s3_key location has more than 1 file example as below, I need to get both files from /path/output to SFTP Location

/path/output/abc.csv
/path/output/def.csv

尝试过:但两个文件均未发布

Tried: But both files are not posted

Tried passing s3_key as '/path/output/*.csv'

代码

with sftp.open(sftp_path + key_name, 'wb') as f:
     s3_client.download_fileobj(s3_bucket,s3_key, f)

推荐答案

请参见 Boto3从S3存储桶下载所有文件.

除了将文件下载到本地文件之外:

Except that instead of downloading the files to a local file:

client.download_file(bucket, k, dest_pathname)

...您会将它们流式传输到SFTP:

... you will stream them to SFTP:

with sftp.open(sftp_path + file.get('Key'), 'wb', 32768) as f:
    client.download_file(bucket, k, f)

请注意，此简单更改不会处理子文件夹".

出于32768参数的目的，请参见在使用pysftp"open"打开的SFTP服务器上写入文件;方法很慢 .

For the purpose of the 32768 argument, see Writing to a file on SFTP server opened using pysftp "open" method is slow.

这篇关于S3ToSFTP:将多个文件从同一S3密钥移到SFTP路径的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！