是否有必要在多线程Paramiko中为每个线程打开一个SFTPClient? [英] Is it necessary to open a SFTPClient per one thread in Paramiko with multi-threading?
问题描述
我想使用 Paramiko 和多线程从远程服务器下载文件.
I want to download files from remote server using Paramiko with multithreading.
我想到了两个解决方案,但我不确定哪个是正确的(或更好).
There are two solution came into my mind, but I'm not sure which is right (or better).
解决方案 1:假设 SFTPClient.get
是线程安全的(但我找不到任何提到的文档),一个简单的:
Solution 1:
Assuming that the SFTPClient.get
is thread safe (But I can't find any document mentioned that), a simple one would as:
from paramiko import SSHClient, AutoAddPolicy, SFTPClient
from concurrent.futures import ThreadPoolExecutor
from typing import List
client = SSHClient()
ciient.set_missing_host_key_policy(AutoAddPolicy())
client.connect( ... )
sftp = client.open_sftp()
files_to_download: List[str] = ...
with ThreadPoolExecutor(10) as pool:
pool.map(lambda fn: sftp.get(fn, fn), files_to_download)
解决方案 2:解决方案 1
- Paramiko 的 API 真的是线程安全的吗?
- 通过单个连接下载多个文件是否有效?
这是我的第二个解决方案:
So here is my second solution:
from paramiko import SSHClient, AutoAddPolicy, SFTPClient
from concurrent.futures import ThreadPoolExecutor
from threading import Lock, local
from typing import List
client = SSHClient()
ciient.set_missing_host_key_policy(AutoAddPolicy())
client.connect( ... )
thread_local = local()
thread_lock = Lock()
files_to_download: List[str] = ...
def download(fn: str) -> None:
"""
thread-safe and each thread has its own SFTPClient
"""
if not hasattr(thread_local, 'sftp'):
with thread_lock:
thread_local.sftp = client.open_sftp()
thread_local.sftp.get(fn, fn)
with ThreadPoolExecutor(10) as pool:
pool.map(download, files_to_download)
哪种解决方案更好?
推荐答案
Paramiko 不是线程安全的.
Paramiko is not thread safe.
在一个连接上使用多个线程可能无论如何都无法提供您希望的性能.您必须为每个线程打开一个单独的连接 (SSHClient
/SFTPClient
).
Using multiple threads over one connection might not give you the performance you hope for anyway. You would have to open a separate connection (SSHClient
/SFTPClient
) per thread.
只有在传输大量小文件之类的情况下,才能通过一个连接获得更好的性能.为此,请参阅使用 SFTP 缓慢上传许多小文件.
With one connection, you would have better performance, only with scenarios like a transfer of large amount of small files. For that, see Slow upload of many small files with SFTP.
这篇关于是否有必要在多线程Paramiko中为每个线程打开一个SFTPClient?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!