从Rails应用程序将csv导入多个mysql数据库 [英] Importing csv into multiple mysql databases from rails app

问题描述

我有一个包含78,000条记录的CSV文件.我使用的是smarter_csv( https://github.com/tilo/smarter_csv )来解析csv文件.我想将其从我的rails应用程序导入到MySQL数据库中.我有以下两个问题

I have a CSV file consisting of 78,000 records.Im using smarter_csv (https://github.com/tilo/smarter_csv) to parse the csv file. I want to import that into a MySQL database from my rails app. I have the following two questions

1. 从我的rails应用程序快速将如此大的数据集导入MySQL的最佳方法是什么?使用resque或sidekiq创建多个工作程序是一个好主意吗?

1. What would be the best approach to quickly importing such a large data-set into MySQL from my rails app ?. Would using resque or sidekiq to create multiple workers be a good idea ?

我需要将此数据插入到存在于多个数据库中的给定表中.在Rails中，我仅与一个数据库进行模型对话.那么，如何扩展解决方案以与模型中的多个mysql数据库对话?

I need to insert this data into a given table which is present in multiple databases. In Rails, i have a model talk to only one database. So how can i scale the solution to talk to multiple mysql databases from my model ?

谢谢

推荐答案

一种方法是使用数据库应用程序本身的本机接口进行导入和导出.它将针对该特定目的进行优化.

One way would be to use the native interface of the database application itself for importing and exporting; it would be optimised for that specific purpose.

对于MySQL， mysqlimport 提供了该接口.请注意，导入也可以作为SQL语句完成，并且此可执行文件为基础SQL命令提供了更为精巧的界面.

For MySQL, the mysqlimport provides that interface. Note that the import can also be done as an SQL statement and that this executable provides a much saner interface for the underlying SQL command.

就实施而言，如果这是一次频繁的导入操作，sidekiq/resque/cron作业是最好的方法.

As far as implementation goes, if this is a frequent import exercise, the sidekiq/resque/cron job is the best possible approach.

上面提到的SQL命令是 LOAD DATA INFILE 其他答案指出.

The SQL command referred to above is the LOAD DATA INFILE as the other answer points out.

这篇关于从Rails应用程序将csv导入多个mysql数据库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！