org.apache.solr.common.SolrException:TransactionLog 不知道如何序列化类 org.bson.types.ObjectId；尝试实现 ObjectResolver? [英] org.apache.solr.common.SolrException: TransactionLog doesn't know how to serialize class org.bson.types.ObjectId; try implementing ObjectResolver?
问题描述
从mongodb导入数据时，Solr抛出如下错误:
When performing a data import from mongodb, Solr throws the following error:
org.apache.solr.common.SolrException: TransactionLog doesn't know how to serialize class org.bson.types.ObjectId; try implementing ObjectResolver?
at org.apache.solr.update.TransactionLog$1.resolve(TransactionLog.java:100)
at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:234)
at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:589)
at org.apache.solr.update.TransactionLog.write(TransactionLog.java:395)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:320)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80)
at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:254)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:526)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:415)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
at java.lang.Thread.run(Thread.java:748)
我的 Solr 版本是 6.6.0.错误的原因是什么?如何解决?
My Solr version is 6.6.0. What could be the reason for the error and how can it be resolved?
推荐答案
我在尝试从 mongoDB 中的多个集合导入数据时遇到了这个问题.
I came across this issue while trying to import data from multiple collections in mongoDB.
假设您没有使用 mongo-connector，我使用以下方法导入数据.
Assuming you are not using mongo-connector, I used the following to import data.
- Solr-6.6.0
- solr-dataimporthandler-6.6.0
- mongo-java-driver-3.5.0
- Solr Mongo 导入器
由于返回的 '_id' 是 ObjectId 类型，我的解决方案是将 '_id' 转换为 String，然后将其索引到 solr 中，并在查询 '_id' 时，在运行之前将其转换为 ObjectId 类型查询.
Since the returned '_id' is of type ObjectId, my work around solution was to convert the '_id' to String before indexing it into solr and while querying with respect to '_id', convert it to type ObjectId before running the query.
下载 solr mongo 导入器并进行以下更改.
Download the solr mongo importer and make the following changes.
MongoMapperTransformer.java
MongoMapperTransformer.java
public class MongoMapperTransformer extends Transformer {
@Override
public Object transformRow(Map<String, Object> row, Context context) {
for (Map<String, String> map : context.getAllEntityFields()) {
String mongoFieldName = map.get(MONGO_FIELD);
String mongoId = map.get(MONGO_ID);
if (mongoFieldName == null)
continue;
String columnFieldName = map.get(DataImporter.COLUMN);
//If the field is ObjectId convert it into String
if (mongoId != null && Boolean.parseBoolean(mongoId)) {
Object srcId = row.get(columnFieldName);
row.put(columnFieldName, srcId.toString());
}
else{
row.put(columnFieldName, row.get(mongoFieldName));
}
}
return row;
}
public static final String MONGO_FIELD = "mongoField";
//To identify the _id field
public static final String MONGO_ID = "objectIdToString";
}
接下来，替换函数
public Iterator <Map<String, Object>> getData(String query){...}
在 MongoDataSource.java 中使用以下内容:
in MongoDataSource.java with the following:
@Override
public Iterator<Map<String, Object>> getData(String query) {
DBObject queryObject = new BasicDBObject();
/* If querying by _id, since the id is a string now,
* it has to be converted back to type ObjectId() using the
* constructor
*/
if(query.contains("_id")){
@SuppressWarnings("unchecked")
Map<String, String> queryWithId = (Map<String, String>) JSON.parse(query);
String id = queryWithId.get("_id");
queryObject = new BasicDBObject("_id", new ObjectId(id));
}
else{
queryObject = (DBObject) JSON.parse(query);
}
LOG.debug("Executing MongoQuery: " + query.toString());
long start = System.currentTimeMillis();
mongoCursor = this.mongoCollection.find(queryObject);
LOG.trace("Time taken for mongo :"
+ (System.currentTimeMillis() - start));
ResultSetIterator resultSet = new ResultSetIterator(mongoCursor);
return resultSet.getIterator();
}
进行这些更改后，您可以使用 ant 构建 jar.
After these changes you can build the jar using ant.
将 jars(solr mongo importer 和 mongo-java-driver)复制到 lib 目录中.我将它们复制到 ${solr-install-dir}/contrib/dataimport-handler/lib
Copy the jars (solr mongo importer and the mongo-java-driver) into the lib directory. I copied them into ${solr-install-dir}/contrib/dataimport-handler/lib
在 solr-config.xml 中为上述 jar 添加 lib 指令:
Add the lib directives in solr-config.xml for the above jars:
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" />
最后，这里是 mongo 集合和 data-config.xml 的示例
Finally, here's an example of the mongo collections and data-config.xml
User collection
{
"_id" : ObjectId("56e9c892e4b0355017b2fa0f"),
"name" : "User1",
"phone" : "123456789"
}
Address collection
{
"_id" : ObjectId("56e9c892e4b0355017b2fa0f"),
"address" : "#666, Maiden street"
}
data-config.xml
data-config.xml
不要忘记为 _id 字段提及 objectIdToString="true" 以便 MongoMapperTransformer 可以将 id 字符串化.
Do not forget to mention objectIdToString="true" for the _id field so that the MongoMapperTransformer can stringify the id.
<dataConfig>
<dataSource name="MyMongo"
type="MongoDataSource"
database="test"
/>
<document name="UserDetails">
<!-- if query="" then it imports everything -->
<entity name="users"
processor="MongoEntityProcessor"
query=""
collection="user"
datasource="MyMongo"
transformer="MongoMapperTransformer">
<field column="_id" name="id" mongoField="_id" objectIdToString="true" />
<field column="phone" name="phone" mongoField="phone"/>
<entity name="address"
processor="MongoEntityProcessor"
query="{_id:'${users._id}'}"
collection="address"
datasource="MyMongo"
transformer="MongoMapperTransformer">
<field column="address" name="adress" mongoField="address"/>
</entity>
</entity>
</document>
</dataConfig>
托管模式将 id 字段作为字符串.此外，如果您在 mongodb 中有嵌套对象，则必须使用脚本转换器在 solr 中为它们编制索引.
The managed-schema will have the id field as string. Also, if you have nested objects in mongodb you will have to use script transformers to index them in solr.
希望这有帮助，祝你好运！
Hope this helps, Good luck !
这篇关于org.apache.solr.common.SolrException:TransactionLog 不知道如何序列化类 org.bson.types.ObjectId；尝试实现 ObjectResolver?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
