在Rails中从CSV导入.如何批量处理，而不是一次全部处理? [英] Importing from CSV in Rails. How to do per batches, not all at once?

问题描述

def self.import file, organization
  counter = 0
  CSV.foreach(file.path, encoding: 'windows-1251:utf-8', headers: true) do |row|
    name = (row["First Name"].to_s + " " + row["Last Name"].to_s).titleize
    customer = Customer.create(
      name: name,
      phone: row["Main Phone"],
      email: row["Main Email"],
      address: row["Address"],
      repair_shop: repair_shop
    )
    puts "#{name} - #{customer.errors.full_messages.join(',')}" if   customer.errors.any?
    counter += 1 if customer.persisted?
  end
  message = "Imported #{counter} users."
end

这是我到目前为止的代码.我要导入的文件具有10,000行，因此在处理过程中不堪重负.

我该如何批量执行此操作?

解决方案

来自 https://satishonrails.wordpress.com/2007/07/18/how-to-import-csv-file-in-rails/

只需添加一个定期的显式垃圾回收:

def self.import file, organization
  counter = 0
  CSV.foreach(file.path, encoding: 'windows-1251:utf-8', headers: true).with_index do |row, i|
    name = (row["First Name"].to_s + " " + row["Last Name"].to_s).titleize
    customer = Customer.create(
      name: name,
      phone: row["Main Phone"],
      email: row["Main Email"],
      address: row["Address"],
      repair_shop: repair_shop
    )
    puts "#{name} - #{customer.errors.full_messages.join(',')}" if   customer.errors.any?
    counter += 1 if customer.persisted?
    GC.start if i % 100 == 0 # forcing garbage collection
  end
  message = "Imported #{counter} users."
end

这样，您将保证服务器不会耗尽内存.我已经在实践中对其进行了检查，它确实有效.

def self.import file, organization
  counter = 0
  CSV.foreach(file.path, encoding: 'windows-1251:utf-8', headers: true) do |row|
    name = (row["First Name"].to_s + " " + row["Last Name"].to_s).titleize
    customer = Customer.create(
      name: name,
      phone: row["Main Phone"],
      email: row["Main Email"],
      address: row["Address"],
      repair_shop: repair_shop
    )
    puts "#{name} - #{customer.errors.full_messages.join(',')}" if   customer.errors.any?
    counter += 1 if customer.persisted?
  end
  message = "Imported #{counter} users."
end

This is the code I have so far. I'm importing files with 10,000 rows, so it overwhelms my production server in processing.

How could I do this in batches?

解决方案

Taken from https://satishonrails.wordpress.com/2007/07/18/how-to-import-csv-file-in-rails/

Simply add a periodic explicit garbage collection:

def self.import file, organization
  counter = 0
  CSV.foreach(file.path, encoding: 'windows-1251:utf-8', headers: true).with_index do |row, i|
    name = (row["First Name"].to_s + " " + row["Last Name"].to_s).titleize
    customer = Customer.create(
      name: name,
      phone: row["Main Phone"],
      email: row["Main Email"],
      address: row["Address"],
      repair_shop: repair_shop
    )
    puts "#{name} - #{customer.errors.full_messages.join(',')}" if   customer.errors.any?
    counter += 1 if customer.persisted?
    GC.start if i % 100 == 0 # forcing garbage collection
  end
  message = "Imported #{counter} users."
end

This way you will guarantee that your server will not run out of memory. I have checked it in practice, it really worked.

这篇关于在Rails中从CSV导入.如何批量处理，而不是一次全部处理?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！