使用Boto 3将文件从AWS S3传输到SFTP [英] Transfer file from AWS S3 to SFTP using Boto 3
问题描述
我是使用Boto3的初学者,我想将文件从S3存储桶直接传输到SFTP服务器.
I am a beginner in using Boto3 and I would like to transfer a file from an S3 bucket to am SFTP server directly.
我的最终目标是为AWS Glue编写Python脚本.
My final goal is to write a Python script for AWS Glue.
我找到了一些文章,展示了如何将文件从SFTP传输到S3存储桶:
https://medium.com/better-programming/transfer-file-from-ftp-server-to-a-s3-bucket-using-python-7f9e51f44e35
I have found some article which shows how to transfer a file from an SFTP to an S3 bucket:
https://medium.com/better-programming/transfer-file-from-ftp-server-to-a-s3-bucket-using-python-7f9e51f44e35
不幸的是,我找不到任何相反的动作.您有什么建议/想法吗?
Unfortunately I can't find anything which does the opposite action. Do you have any suggestions/ideas?
我的第一个错误尝试是在下面.
My first wrong attempt is below.
但是我要避免将文件下载到本地内存中,然后再将其移动到SFTP.
But I would like to avoid downloading while file to my local memory in order to move it then to SFTP.
import pysftp
import boto3
# get clients
s3_gl = boto3.client('s3', aws_access_key_id='', aws_secret_access_key='')
# parameters
bucket_gl = ''
gl_data = ''
gl_script = ''
source_response = s3_gl.get_object(Bucket=bucket_gl,Key=gl_script+'file.csv')
print(source_response['Body'].read().decode('utf-8'))
#---------------------------------
srv = pysftp.Connection(host="", username="", password="")
with srv.cd('relevant folder in sftp'):
srv.put(source_response['Body'].read().decode('utf-8'))
# Closes the connection
srv.close()
推荐答案
直接传输..."可以表示许多不同的事物.
"transfer ... directly" can mean number of different things.
让我们假设您想通过本地计算机(运行Python代码的地方)传输文件,而没有实际将文件的临时副本存储到本地文件系统中.
Let's assume that you want to transfer the file via the local machine (where the Python code runs), without actually storing a temporary copy of the file to the local file system.
对于SFTP上传,您可以使用 Paramiko库.
For SFTP upload, you can use Paramiko library.
假设您已经拥有 Paramiko SFTPClient
(sftp
)和Boto 3 client
(s3
)实例就绪(问题链接的文章中对此进行了介绍),您可以简单地胶粘"使用类似文件的对象将它们一起使用:
Assuming you already have your Paramiko SFTPClient
(sftp
) and Boto 3 client
(s3
) instances ready (what is covered in the article you have linked in your question), you can simply "glue" them together using file-like objects:
with sftp.open('/sftp/path/filename', 'wb', 32768) as f:
s3.download_fileobj('mybucket', 'mykey', f)
出于32768
参数的目的,请参见在使用pysftp"open"打开的SFTP服务器上写入文件;方法很慢 .
For the purpose of the 32768
argument, see Writing to a file on SFTP server opened using pysftp "open" method is slow.
这篇关于使用Boto 3将文件从AWS S3传输到SFTP的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!