使用 Boto 3 将文件从 AWS S3 传输到 SFTP [英] Transfer file from AWS S3 to SFTP using Boto 3

问题描述

我是使用 Boto3 的初学者，我想将文件从 S3 存储桶直接传输到 SFTP 服务器.

I am a beginner in using Boto3 and I would like to transfer a file from an S3 bucket to am SFTP server directly.

我的最终目标是为 AWS Glue 编写 Python 脚本.

My final goal is to write a Python script for AWS Glue.

我找到了一些文章，其中展示了如何将文件从 SFTP 传输到 S3 存储桶:
https://medium.com/better-programming/transfer-file-from-ftp-server-to-a-s3-bucket-using-python-7f9e51f44e35

I have found some article which shows how to transfer a file from an SFTP to an S3 bucket:
https://medium.com/better-programming/transfer-file-from-ftp-server-to-a-s3-bucket-using-python-7f9e51f44e35

不幸的是，我找不到任何做相反动作的东西.您有什么建议/想法吗?

Unfortunately I can't find anything which does the opposite action. Do you have any suggestions/ideas?

我的第一次错误尝试如下.

My first wrong attempt is below.

但我想避免将文件下载到本地内存，以便将其移动到 SFTP.

But I would like to avoid downloading while file to my local memory in order to move it then to SFTP.

import pysftp
import boto3

# get clients
s3_gl = boto3.client('s3', aws_access_key_id='', aws_secret_access_key='')

# parameters
bucket_gl = ''
gl_data = ''
gl_script = ''

source_response = s3_gl.get_object(Bucket=bucket_gl,Key=gl_script+'file.csv')
print(source_response['Body'].read().decode('utf-8'))

#---------------------------------

srv = pysftp.Connection(host="", username="", password="")

with srv.cd('relevant folder in sftp'): 
    srv.put(source_response['Body'].read().decode('utf-8')) 

# Closes the connection
srv.close()

推荐答案

transfer ... direct" 可能意味着许多不同的事情.

"transfer ... directly" can mean number of different things.

假设您想通过本地机器(运行 Python 代码的地方)传输文件，而不实际将文件的临时副本存储到本地文件系统.

Let's assume that you want to transfer the file via the local machine (where the Python code runs), without actually storing a temporary copy of the file to the local file system.

对于 SFTP 上传，您可以使用 Paramiko 库.

For SFTP upload, you can use Paramiko library.

假设您已经拥有 Paramiko SFTPClient (sftp) 和 Boto 3 client (s3) 实例准备好(您在问题中链接的文章中涵盖了哪些内容)，你可以简单地粘合"它们一起使用类文件对象:

Assuming you already have your Paramiko SFTPClient (sftp) and Boto 3 client (s3) instances ready (what is covered in the article you have linked in your question), you can simply "glue" them together using file-like objects:

with sftp.open('/sftp/path/filename', 'wb', 32768) as f:
    s3.download_fileobj('mybucket', 'mykey', f)

对于 32768 参数的目的，请参阅写入打开的 SFTP 服务器上的文件使用 pysftp 打开"方法很慢.

For the purpose of the 32768 argument, see Writing to a file on SFTP server opened using pysftp "open" method is slow.

这篇关于使用 Boto 3 将文件从 AWS S3 传输到 SFTP的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！