在Rails中批量导入CSV文件? [英] Import CSV in batches of lines in Rails?

问题描述

我正在使用FasterCSV将上载的文件导入到模型中，它对于小型文件非常有用.但是，当我尝试导入一个大型数据集(21,000行)时，它会花一些时间，并且在实时服务器上也会出现浏览器超时.

I'm using FasterCSV to import an uploaded file to a model, and it's working great for small files. However when I try to import a large dataset (21,000 lines) it takes ages and I get browser timeouts on the live server.

这是我当前的工作代码:

This is my current working code:

  logcount=0
  Attendee.transaction do
    FCSV.new(file, :headers => true).each do |row|
      row[1] = Date.strptime(row[1], '%m/%d/%Y')
      record = @event.attendees.new(:union_id => row[0], :dob => row[1], :gender => row[2])
      if record.save
        logcount += 1
      end
    end
  end

我想使用后台进程，但是用户需要先查看导入了多少行，然后才能移至系统的下一步.

I'd love to use a background process, but the user needs to see how many lines were imported before they can move to the next step of the system.

因此，我当时认为我应该使用动作分块，并且只能读取行数较小的情况下，设置一个计数器，然后以某种进度更新视图，然后以上一个计数器为起点再次运行该方法.

So, I was thinking that I should use action chunking and only read a smaller number of lines, set a counter, then update the view with some kind of progress, then run the method again using the previous counter as start point.

我似乎看不到如何使FasterCSV只读取一定数量的行，以及为起点设置偏移量.

I can't seem to see how to get FasterCSV to read only a set number of lines, and also set an offset for the start point.

有人知道该怎么做吗?还是有更好的方法来解决这个问题?

Does anyone know how to do this? Or is there a better way to handle this?

推荐答案

尝试 AR导入

旧答案

您是否尝试过使用 AR扩展进行批量导入? 当您向数据库插入1000行时，您将获得令人印象深刻的性能提升. 请访问其网站以获取更多详细信息.

Have you tried to use AR Extensions for bulk import? You get impressive performance improvements when you are inserting 1000's of rows to DB. Visit their website for more details.

这篇关于在Rails中批量导入CSV文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！