无法运行mongo连接器 [英] unable to run mongo-connector

问题描述

我已经在mongodb服务器中安装了mongo连接器。

我正在执行命令

  mongo-connector  - m [remote mongo server IP]：[remote mongo server port] -t [elastic search server IP]：[弹性搜索服务器端口] -d elastic_doc_manager.py 
 p> 
 
 
我也尝试过这样，因为mongo与默认端口在同一台服务器上运行。
 code> mongo-connector -t [弹性搜索服务器IP]：[弹性搜索服务器端口] -d elastic_doc_manager.py 
  
我收到错误
 追溯（最近最近通话）：
文件/ usr / local / bin / mongo-connector，第9行，< module> 
 load_entry_point（'mongo-connector == 2.3.dev0'，'console_scripts'，'mongo-connector'）（）
 
文件/usr/local/lib/python2.7/ dist-packages / mongo_connector-2.3.dev0-py2.7.egg / mongo_connector / util.py，第85行，包裹
 func（* args，** kwargs）
 
文件/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py，第1037行，主要是
 conf.parse_args（）
 
文件/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/config.py，第118行，在parse_args 
 option，（（k，values.get（k））for k.在option.cli_names））
 
文件/usr/local/lib/python2.7/dist-packages/mongo_connector -2.3.dev0-py2.7.egg / mongo_connector / connector.py，第820行，apply_doc_managers 
 module = import_dm_by_name（dm ['docManager']）
 
文件/ usr /local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py，l ine 810，in import_dm_by_name 
无法导入％s。 ％full_name）
 
 ** mongo_connector.errors.InvalidConfiguration：无法导入mongo_connector.doc_managers.elastic_doc_manager.py。** 
  
注意：我使用python2.7 
和mongo-connector 2.3 
 
 
 
弹性搜索服务器是2.2 
任何建议？
 
 
 
    
申请 Val 的建议：
 
 
 2016-02-29 19：56：59,519 [CRITICAL] mongo_connector.oplog_manager：549  -  
收集转储中的异常
 
 
 
追溯（最近的最后一次呼叫）：
 
 
 
文件
/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/oplog_manager.py，
 line 501，in do_dump 
 
 
 
 upsert_all（dm）
 
 
 
文件
/ usr / local / lib / python2。 7 / dist-packages / mongo_connector-2.3.dev0-py2.7.egg / mongo_connector / oplog_manager.py，
 line 485，insertsert_all dm.bulk_upsert（docs_to_dump（namespace），
 mapped_ns，long_ts）
 
 
 
文件
/usr/local/lib/python2.7/dist packages / mongo_connector-2.3.dev0-py2.7.egg / mongo_connector / util.py，第32行，包装
 
 
 
 return f（* args，**文件
/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/ mongo_connector / doc_managers / elastic_doc_manager.py，第190行，bulk_upsert 
 
 
 
为了确定，在响应中：
 
 
 
文件
/usr/local/lib/python2.7/dist-packages/elasticsearch-1.9.0-py2.7.egg/elasticsearch/helpers/init.py，
第160行，在streaming_bulk中
 
 
 
导致_process_bulk_chunk（客户端，bulk_actions，
 raise_on_exception，raise_on_error，** kwargs）：
 
 $ b $文件
/usr/local/lib/python2.7/dist-packages/elasticsearch-1.9.0-py2.7.egg/elasticsearch/helpers/init.py，
第132行，在_process_bulk_chunk 
 
 $中b $ b 
 raise BulkIndexError（'％i文件无法索引'％len（errors），
 errors）
 
 
 
 BulkIndexError： （u'2 document（s）failed to 
 index。'，.. document_class = dict，tz_aware = False，connect = True，
 replicaset = u'mss'），u'local'）， u'oplog.rs'）
 
 
 
 2016-02-29 19：56：59,835 [错误] mongo_connector.connector：302  -  
 MongoConnector：OplogThread意外停止！关闭
 
 
 
 
 
 
 
 
 Hi Val，
 
 $ b $我连接另一个mongodb实例，它只有一个数据库，有一个集合有30,000多个记录，我能够成功执行。以前的mongodb集合有多​​个数据库（大约7个），内部有多个集合（每个数据库大约5到15个），并且在集合中都有很多文档（范围从500到50,000）。 
 
 
 
由于mongo数据库中存在巨大数据，Mongo连接器是否发生故障？
 
 
 
我有进一步的查询
 
 
 
 a。是否有可能仅在mongodb中的特定集合完成索引，驻留在不同的数据库中？我只想索引特定的集合（而不是整个数据库）。如何实现？
 
 
 
 b。在弹性搜索中，我可以看到一个集合的重复索引。第一个是数据库名称（如预期），另一个名称为mongodb_meta，它们都具有相同的数据，如果我正在更改集合，则更新在集合中发生。 
 
 
 
 c。可以配置输出索引名称或任何其他参数吗？
解决方案
我认为唯一的问题是你有文档管理器上的 .py 扩展名（mongo-connector 2.0之前需要），您只需要删除它：
 
 
 [pre>  mongo-connector -m [远程mongo服务器IP]：[远程mongo服务器端口] -t [弹性搜索服务器IP]：[弹性搜索服务器端口] -d elastic_doc_manager 
  

I have installed mongo-connector in the mongodb server.

I am executing by giving the command

mongo-connector -m [remote mongo server IP]:[remote mongo server port] -t [elastic search server IP]:[elastic search server Port]  -d elastic_doc_manager.py

I also tried with this since mongo is running in the same server with the default port.

mongo-connector -t [elastic search server IP]:[elastic search server Port]  -d elastic_doc_manager.py

I am getting error

Traceback (most recent call last):
  File "/usr/local/bin/mongo-connector", line 9, in <module>
    load_entry_point('mongo-connector==2.3.dev0', 'console_scripts', 'mongo-connector')()

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/util.py", line 85, in wrapped
    func(*args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py", line 1037, in main
    conf.parse_args()

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/config.py", line 118, in parse_args
    option, dict((k, values.get(k)) for k in option.cli_names))

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py", line 820, in apply_doc_managers
    module = import_dm_by_name(dm['docManager'])

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py", line 810, in import_dm_by_name
    "Could not import %s." % full_name)

**mongo_connector.errors.InvalidConfiguration: Could not import mongo_connector.doc_managers.elastic_doc_manager.py.**

NOTE: I am using python2.7 and mongo-connector 2.3

Elastic search server is 2.2

Any suggestions ?

[edit] After applying Val's suggestion:

2016-02-29 19:56:59,519 [CRITICAL] mongo_connector.oplog_manager:549 - Exception during collection dump

Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/oplog_manager.py", line 501, in do_dump

upsert_all(dm)

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/oplog_manager.py", line 485, in upsert_all dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/util.py", line 32, in wrapped

return f(*args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/doc_managers/elastic_doc_manager.py", line 190, in bulk_upsert

for ok, resp in responses:

File "/usr/local/lib/python2.7/dist-packages/elasticsearch-1.9.0-py2.7.egg/elasticsearch/helpers/init.py", line 160, in streaming_bulk

for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):

File "/usr/local/lib/python2.7/dist-packages/elasticsearch-1.9.0-py2.7.egg/elasticsearch/helpers/init.py", line 132, in _process_bulk_chunk

raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)

BulkIndexError: (u'2 document(s) failed to index.',..document_class=dict, tz_aware=False, connect=True, replicaset=u'mss'), u'local'), u'oplog.rs')

2016-02-29 19:56:59,835 [ERROR] mongo_connector.connector:302 - MongoConnector: OplogThread unexpectedly stopped! Shutting down

---

Hi Val,

I connected with another mongodb instance, which had only one database, having one collection with 30,000+ records and I was able to execute it succesfully. The previous mongodb collection has multiple databases (around 7), which internally had multiple collections (around 5 to 15 per databases) and all were having good amount of documents (ranging from 500 to 50,000) in the collections.

Was Mongo-connector failing because of huge data residing in the mongo database ?

I have further queries

a. Is is possible to get indexing done of only specific collections in the mongodb, residing in different databases? I wan to index only specific collections (not the entire database). How can I achieve this ?

b. In elasticsearch i can see duplicate indexes for one collection. First one is with the database name (as expected), other one with the name mongodb_meta, both of them having same data, if I am changing the collection, the update is happening in both the collections.

c. Is it possible to configure the output index name or any other parameters any how?

解决方案

I think the only issue is that you have the .py extension on the doc manager (it was needed before mongo-connector 2.0), you simply need to remove it:

mongo-connector -m [remote mongo server IP]:[remote mongo server port] -t [elastic search server IP]:[elastic search server Port]  -d elastic_doc_manager

这篇关于无法运行mongo连接器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！