SOLR - 从 csv 文件导入 2000 万个文档的最佳方法 [英] SOLR - Best approach to import 20 million documents from csv file

问题描述

我目前手头的任务是找出在 solr 中加载数百万文档的最佳方法.数据文件是从 DB 导出的 csv 格式.

My current task on hand is to figure out the best approach to load millions of documents in solr. The data file is an export from DB in csv format.

目前，我正在考虑将文件拆分为较小的文件并编写一个脚本，同时使用 curl 发布这些较小的文件.

Currently, I am thinking about splitting the file into smaller files and having a script while post this smaller ones using curl.

我注意到如果您发布大量数据，大多数情况下请求会超时.

I have noticed that if u post high amount of data, most of the time the request times out.

我正在研究数据导入器，这似乎是一个不错的选择

I am looking into Data importer and it seems like a good option

高度赞赏任何其他想法

谢谢

推荐答案

除非数据库已经是您解决方案的一部分，否则我不会为您的解决方案增加额外的复杂性.引用 SOLR FAQ 这是您的 servlet 容器发出会话超时.

Unless a database is already part of your solution, I wouldn't add additional complexity to your solution. Quoting the SOLR FAQ it's your servlet container that is issuing the session time-out.

在我看来，您有几个选择(按我的偏好顺序):

As I see it, you have a couple of options (In my order of preference):

增加容器超时.(maxIdleTime"参数，如果您使用嵌入式 Jetty 实例).

Increase the container timeout. ("maxIdleTime" parameter, if you're using the embedded Jetty instance).

我假设您只是偶尔索引这么大的文件?暂时增加超时可能只是最简单的选择.

I'm assuming you only occasionally index such large files? Increasing the time-out temporarily might just be simplest option.

这是一个简单的 unix 脚本，可以完成这项工作(将文件拆分为 500,000 行块):

Here's the simple unix script that will do the job (Splitting the file in 500,000 line chunks):

split -d -l 500000 data.csv split_files.
for file in `ls split_files.*`
do  
curl 'http://localhost:8983/solr/update/csv?fieldnames=id,name,category&commit=true' -H 'Content-type:text/plain; charset=utf-8' --data-binary @$file
done

解析文件并分块加载

以下 groovy 脚本使用 opencsv 和 solrj 来解析 CSV 文件并每 500,000 行向 Solr 提交更改.

Parse the file and load in chunks

The following groovy script uses opencsv and solrj to parse the CSV file and commit changes to Solr every 500,000 lines.

import au.com.bytecode.opencsv.CSVReader

import org.apache.solr.client.solrj.SolrServer
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
import org.apache.solr.common.SolrInputDocument

@Grapes([
    @Grab(group='net.sf.opencsv', module='opencsv', version='2.3'),
    @Grab(group='org.apache.solr', module='solr-solrj', version='3.5.0'),
    @Grab(group='ch.qos.logback', module='logback-classic', version='1.0.0'),
])

SolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr/");

new File("data.csv").withReader { reader ->
    CSVReader csv = new CSVReader(reader)
    String[] result
    Integer count = 1
    Integer chunkSize = 500000

    while (result = csv.readNext()) {
        SolrInputDocument doc = new SolrInputDocument();

        doc.addField("id",         result[0])
        doc.addField("name_s",     result[1])
        doc.addField("category_s", result[2])

        server.add(doc)

        if (count.mod(chunkSize) == 0) {
            server.commit()
        }
        count++
    }
    server.commit()
}

这篇关于SOLR - 从 csv 文件导入 2000 万个文档的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！