为Paramiko模块的sftp.get()设置时间限制 [英] Setting timelimit for sftp.get() of Paramiko module

问题描述

我使用Paramiko的SFTP客户端从远程服务器下载文件到客户端(即get操作)
要传输的文件有点大~1GB.
所以如果时间超过10s，我希望get操作超时.

I am using Paramiko's SFTP client to download a file from remote server to a client(i.e. get operation)
The file to be transferred is a bit huge ~1GB.
So I would like the get operation to timeout if the time is more than 10s.

但是为connect设置超时值不起作用，它似乎只是创建SSH连接的超时，而不是整个ssh连接的超时.

But setting the timeout value for connect doesn't work, It appears to be the timeout for just creating the SSH connection and not the timeout for the whole ssh connection.

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=username, password=password, timeout=10.0)
sftp = ssh.open_sftp()
start_time = time.time()
sftp.get(local_path,remote_path)
elapsed_time = time.time()-start_time
print elapsed_time
sftp.close()

我也试过为频道设置超时值，但它也不起作用

I also tried setting the timeout value for the channel but it doesn't work too

sftp.get_channel.settimeout(10.0)

但是这个超时再次仅用于读/写操作

But this timeout is again just for read/write operations

有一个类似的问题 paramiko (python) 超时 但它只有在创建 SSH 连接时有超时的答案

There is a similar question Timeout in paramiko (python) but it only has answer for timeout in creation of SSH-connection

更新1:根据@Martin的评论，我实现了一个回调函数，用于检查sftp的get操作的时间限制:

Update 1: Following the comments of @Martin I implemented a callback function which checks for the time-limit for the get operation of sftp:

import paramiko
import time

Class TimeLimitExceeded(Exception):
    pass

timelimit = 10
start_time = time.time()

def _timer():
    elapsed_time = time.time()-start_time
    if elapsed_time > timelimit:
        raise TimeLimitExceeded

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=username, password=password, timeout=10.0)
sftp = ssh.open_sftp()
try:
    sftp.get(local_path,remote_path,_timer)
except TimeLimitExceeded:
    print "The operation took too much time to complete"
sftp.close()

但是排除异常的时间很多，代码在某处阻塞.我深入研究了 Paramiko 源代码，发现其背后的罪魁祸首是 sftp_file.py 的 _close(self,async=False) 方法有什么帮助来解决这个问题吗?

But the time to except the exception is a lot, the code is blocking somewhere. I dived into Paramiko source code and found the culprit behind it was the _close(self,async=False) method of the sftp_file.py Any help to get around this?

更新 2:如果超过时间限制，尝试关闭频道本身.然后异常被刷新到控制台，因为 prefetch 是由单独的守护线程实现的

Update 2: Trying to close the channel itself if timelimit is exceeded. Then the exception is flushed to console as the prefetch is implemented by separate daemon thread

 File "/scratch/divjaisw/python2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/scratch/divjaisw/python2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp_file.py", line 488, in _prefetch_thread
    num = self.sftp._async_request(self, CMD_READ, self.handle, long(offset), int(length))
  File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp_client.py", line 754, in _async_request
    self._send_packet(t, msg)
  File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
    self._write_all(out)
  File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp.py", line 133, in _write_all
    n = self.sock.send(out)
  File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/channel.py", line 715, in send
    return self._send(s, m)
  File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/channel.py", line 1081, in _send
    raise socket.error('Socket is closed')
error: Socket is closed

推荐答案

你所要求的，并不是真正的超时.术语超时"用于限制等待响应.

What you ask for, is not really a timeout. The term "timeout" is used for a limit of waiting for a response.

但是您的服务器并没有停止响应.通信处于活动状态.

But your server does not stop responding. The communication is active.

您要求的是操作持续时间的限制.您几乎不能指望这会为您轻松实现.而是一个具体的要求.您必须自己实施.

What you ask for, is rather a limit for a duration of an operation. You can hardly expect this to be implemented readily for you. It's rather a specific requirement. You have to implement it yourself.

您可以使用 get 方法:

You can use the callback argument of the get method:

def get(self, remotepath, localpath, callback=None):

在回调中，检查传输的持续时间并抛出异常，如果时间限制到期.

In the callback, check the duration of the transfer and throw an exception, if the time limit expires.

这不会立即取消转移.为了优化传输性能，Paramiko 将最多 100 个读取请求排入服务器队列(请参阅 sftp_file._write 中的条件).一旦您尝试取消传输，Paramiko 必须等待(最多 100 个)对这些请求的响应以清除队列.

This won't cancel the transfer immediately. To optimize transfer performance, Paramiko queues up to 100 read requests to the server (see the condition in the sftp_file._write). Once you attempt to cancel the transfer, Paramiko has to wait for (up to 100) responses to those requests to clear the queue.

这篇关于为Paramiko模块的sftp.get()设置时间限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！