弹性搜索-Spark中将多个字段用作映射ID [英] Elastic Search - Multiple Field as mapping ID in Spark

问题描述

我对弹性搜索非常陌生.我正在使用 elasticsearch-hadoop 6.2.4 版本，并且正在从 HDFS 中读取文件，转换为Bean对象并写入弹性搜索.我正在使用Spark结构化流式传输.

I am pretty new to elastic search. I am using elasticsearch-hadoop 6.2.4 version and I am reading the files from HDFS, converting to bean object and writing to elastic search. I am using Spark Structured streaming.

StreamingQuery query = dataSet
                        .writeStream()
                        .format("org.elasticsearch.spark.sql")
                        //.outputMode(OutputMode.Append())
                        .option("checkpointLocation", "\tmp\ckpt1")
                        .option("es.nodes","abc.dev.cm.par.xy.hp")
                        .option("es.port","9200")
                        .option("es.mapping.id", "CustomerID")
                        .option("es.resource", "testIndex/testType")
                        .start();

在撰写本文时，我将pojo类中的一个字段( CustomerID )作为映射iD.我们可以给多个字段或字段组合作为映射ID吗?例如，我的文件包含客户ID以及订单ID字段.我们可以将这两个字段合并为 CustomerID + OrderID 吗?

While writing i am giving one of the field (CustomerID)in the pojo class as mapping iD. Can we give multiple fields or combination of fields as mapping ID? For example, my file contains customer id as well as order id fields. Can we combine these both fields as CustomerID+OrderID something like that?

推荐答案

否，您不能将多个属性设置为 "es.mapping.id" .您可以做的一件事就是，无论您想要什么复合ID，创建它并将其附加到数据框并使用它.

No, You can't set multiple attributes as "es.mapping.id" . One Thing you can do is, what ever composite Id you want, Create it and append it to the Dataframe and use the same.

这篇关于弹性搜索-Spark中将多个字段用作映射ID的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！