Ruby Net::FTP 超时线程 [英] Ruby Net::FTP Timeout Threads

问题描述

我试图通过使用线程 FTP 连接来加速多个 FTP 下载.我的问题是我总是有线程挂起.我正在寻找一种干净的方式来告诉 FTP 它需要重试 ftp 事务，或者至少知道 FTP 连接何时挂起.

I was trying to speed up multiple FTP downloads by using threaded FTP connections. My problem is that I always have threads hang. I am looking for a clean way of either telling FTP it needs to retry the ftp transaction, or at least knowing when the FTP connection is hanging.

在下面的代码中，我正在处理 5/6 个单独的 FTP 连接，其中每个线程都有一个预期下载的文件列表.脚本完成后，一些线程挂起并且无法加入.我使用变量@last_updated 来表示最后一次成功的下载时间.如果当前时间 + 20 秒超过 @last_updated，则终止剩余线程.有没有更好的办法?

In the code below I am threading 5/6 separate FTP connections where each thread has a list of files it is expected to download. When the script completes, a few of the threads hang and can not be joined. I am using the variable @last_updated to represent the last successful download time. If the current time + 20 seconds exceeds @last_updated, kill the remaining threads. Is there a better way?

threads = []
max_thread_pool = 5
running_threads = 0
Thread.abort_on_exception = true

existing_file_count = 0
files_downloaded = 0

errors = []
missing_on_the_server = []
@last_updated = Time.now

if ids.length > 0
    ids.each_slice(ids.length / max_thread_pool) do |id_set|
        threads << Thread.new(id_set) do |t_id_set|
            running_threads += 1
            thread_num = running_threads
            thread_num.freeze
            puts "making thread # #{thread_num}"
            begin
                ftp = Net::FTP.open(@remote_site)
                ftp.login(@remote_user, @remote_password)
                ftp.binary = true
                #ftp.debug_mode = true
                ftp.passive = false
            rescue
                raise "Could not establish FTP connection"
            end
            t_id_set.each do |id|
                @last_updated = Time.now
                rmls_path = "/_Photos/0#{id[0,2]}00000/#{id[2,1]}0000/#{id[3,1]}000/#{id}-1.jpg"
                local_path = "#{@photos_path}/01/#{id}-1.jpg"
                progress += 1
                unless File.exist?(local_path)
                    begin
                        ftp.getbinaryfile(rmls_path, local_path)
                        puts "ftp reponse: #{ftp.last_response}"
                        # find the percentage of progress just for fun
                        files_downloaded += 1
                        p = sprintf("%.2f", ((progress.to_f / total) * 100))
                        puts "Thread # #{thread_num} > %#{p} > #{progress}/#{total} > Got file: #{local_path}"
                    rescue
                        errors << "#{thread_num} unable to get file > ftp response: #{ftp.last_response}"
                        puts errors.last
                        if ftp.last_response_code.to_i == 550
                            # Add the missing file to the missing list
                            missing_on_the_server << errors.last.match(/d{5,}-d{1,2}.jpg/)[0]
                        end
                    end
                else
                    puts "found file: #{local_path}"
                    existing_file_count += 1
                end
            end
            puts "closing FTP connection #{thread_num}"
            ftp.close
        end # close thread
    end
end

# If @last_updated has not been updated on the server in over 20 seconds, wait 3 seconds and check again
while Time.now < @last_updated + 20 do
    sleep 3
end
# threads are hanging so joining the threads does not work.
threads.each { |t| t.kill }

推荐答案

对我来说有效的技巧是使用 ruby​​ 的 Timeout.timeout 来确保 FTP 连接没有挂起.

The trick for me that worked was to use ruby's Timeout.timeout to ensure the FTP connection was not hanging.

begin
    Timeout.timeout(10) do
        ftp.getbinaryfile(rmls_path, local_path)
    end
    # ...
rescue Timeout::Error
    errors << "#{thread_num}> File download timed out for: #{rmls_path}"
    puts errors.last
rescue
    errors << "unable to get file > ftp reponse: #{ftp.last_response}"
    # ...
end

挂起的 FTP 下载导致我的线程似乎挂起.现在线程不再挂起，我可以使用更合适的方式处理线程:

Hanging FTP downloads were causing my threads to appear to hang. Now that the threads are no longer hanging, I can use the more proper way of dealing with threads:

threads.each { |t| t.join }

而不是丑陋的:

# If @last_updated has not been updated on the server in over 20 seconds, wait 3 seconds and check again
while Time.now < @last_updated + 20 do
    sleep 3
end
# threads are hanging so joining the threads does not work.
threads.each { |t| t.kill }

这篇关于Ruby Net::FTP 超时线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！