我收到错误:在 Ruby 中写入大量文件后打开的文件太多 [英] I am getting the error: Too many open files after writing a lot of files in Ruby

问题描述

我有一个脚本，可以生成 16000 个 html 页面并将其保存在系统中.在 1013 页后，我收到错误消息:打开的文件太多.

I have a script which generates like 16000 html pages and saves it in the system. after 1013 pages i get the error: Too many open files.

这是生成文件的 Ruby 代码

This is the Ruby code which generates the files

FileUtils.mkdir_p "public/users_directory/#{DEFAULT_COUNTRY_CODE}/#{prefix}"
FileUtils.mkdir_p "public/users_directory/#{DEFAULT_COUNTRY_CODE}/#{prefix}/#{n/1000}"

html_file = File.new("public/users_directory/#{DEFAULT_COUNTRY_CODE}/#{prefix}/#{n/1000}/#{n}.html", "w")
html_file.write(html)
html_file.close

如你所见，我在最后一行关闭了文件......

as you can see i close the file in the last line....

有人知道我在这里做错了什么吗?我有 Ubuntu 8.04.4 LTS

Does somebody know what i am doing wrong here? I have Ubuntu 8.04.4 LTS

非常感谢

这是整个脚本

    def self.fetching_directory_page(n=1, letter = nil)
      id = letter == '' ? "" : "/#{letter.upcase}"
      url = "this is a valid url :)"
      agent = WWW::Mechanize.new
      page = agent.get(url)
      html = page.search('div#my_profile_body').to_html

      prefix = id == '' ? 'all' : letter
      FileUtils.mkdir_p "public/users_directory/#{DEFAULT_COUNTRY_CODE}/#{prefix}"
      FileUtils.mkdir_p "public/users_directory/#{DEFAULT_COUNTRY_CODE}/#{prefix}/#{n/1000}"

      html_file = File.new("public/users_directory/#{DEFAULT_COUNTRY_CODE}/#{prefix}/#{n/1000}/#{n}.html", "w")
      html_file.write(html)
      html_file.close

      puts "+ CREATED #{prefix}/#{n/1000}/#{n}.html" 

      new_url = page.parser.xpath("//a[@class='next_page']")[0]['href'] rescue nil

      if new_url.present?
        self.fetching_directory_page(n+1, letter)
      end
    end

它正在获取我的用户目录的所有用户并出于缓存原因保存页面.一共生成16000个文件.

It is fetching all the users of my users directory and saves the page for caching reasons. It generates 16000 files in total.

这是 ulimit-a 的结果

This is results for ulimit-a

    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 24640
    max locked memory       (kbytes, -l) 32
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 24000
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 8192
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 24640
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited

编辑/etc/security/limits 后，我没有收到错误 打开的文件太多 但它只是卡住了

After editing /etc/security/limits i dont get the error Too many open files but it just gets stuck

lsof -u username 返回一个包含或多或少 600 个条目的列表，并且在执行脚本时不会改变

lsof -u username returns a list of more or less 600 entries and it doesnt change while doing the script

推荐答案

打开文件不是问题的原因.这是递归方法.我改变了它，一切都很好.

Open files weren't causing the problem. It was the recursive method. I changed that and things work great.

这篇关于我收到错误:在 Ruby 中写入大量文件后打开的文件太多的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！