solr ReplicationHandler - SnapPull 无法下载文件 [英] solr ReplicationHandler - SnapPull failed to download files
问题描述
在从 master 到 slave 的复制过程中，我们不断收到此异常.我们的索引大小是 9.7 G，我们正在尝试从头开始复制一个从站.
we are continuously getting this exception during replication from master to slave. our index size is 9.7 G and we are trying to replicate a slave from scratch.
2013 年 10 月 30 日 18:22:16,996 [explicit-fetchindex-cmd] 错误 ReplicationHandler - SnapPull 失败:org.apache.solr.common.SolrException:无法完全下载 _41c_Lucene41_0.doc.已下载 0!=107464871在 org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1266)在 org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1146)在 org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:741)在 org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)在 org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:319)在 org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:220)
30 Oct 2013 18:22:16,996 [explicit-fetchindex-cmd] ERROR ReplicationHandler - SnapPull failed :org.apache.solr.common.SolrException: Unable to download _41c_Lucene41_0.doc completely. Downloaded 0!=107464871 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1266) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1146) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:741) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:319) at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:220)
我在一些帖子中读到 solr 4.1 中有一个相关的错误，但我们正在使用 solr 4.3 并尝试使用 4.5.1.看来DirectoryFileFetcher有时不能下载文件，文件以零大小下载到salve.
I read in some thread that there was a related bug in solr 4.1, but we are using solr 4.3 and tried with 4.5.1 also. It seams that DirectoryFileFetcher can not download a file sometimes , the files is downloaded to the salve in size zero.
这是主设置:
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="confFiles">stopwords.txt,spellings.txt,synonyms.txt,protwords.txt,elevate.xml,currency.xml</str>
<str name="commitReserveDuration">00:00:50</str>
</lst>
</requestHandler>
和从站设置:
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="confFiles">stopwords.txt,spellings.txt,synonyms.txt,protwords.txt,elevate.xml,currency.xml</str>
<str name="commitReserveDuration">00:00:50</str>
</lst>
</requestHandler>
推荐答案
问题似乎与 httpclient 有关.我打开了所有库的调试日志记录，并在失败前看到来自 httpclient 的消息响应垃圾".
The problem appeared to be with httpclient. I turned on debug logging for all libraries and saw a message "Garbage in response" coming from httpclient just before the failure.
这是一个日志片段:
31 Oct 2013 18:10:40,360 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Sending request: GET /solr-master/replication?comman
d=filecontent&generation=6814&qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.12013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - >> "GET/solr-master/replication?command=filecontent&generation=6814&qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - >>用户代理:Solr[org.apache.solr.client.solrj.impl.HttpSolrServer]1.0[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - >>主机:solr-master.saltdev.sealdoc.com:8081[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - >>连接:保持活动 [\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - >> "[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试标头 - >> GET/solr-master/replication?command=filecontent&generation=6814&qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.12013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试标头 - >> 用户代理:Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.02013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试标头 - >> 主机:solr-master.saltdev.sealdoc.com:80812013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] DEBUG 标头 - >> 连接:Keep-Alive2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - <<"[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - 垃圾响应:2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - <<"4[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - 垃圾响应:42013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - <<"[0x0][0x0][0x0][0x0][\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - 垃圾响应:^@^@^@^@2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - <<"0[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - 垃圾响应:02013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] 调试线 - <<"[\r][\n]"2013 年 10 月 31 日 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - 垃圾响应:2013 年 10 月 31 日 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 关闭2013 年 10 月 31 日 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 关闭2013 年 10 月 31 日 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 关闭2013 年 10 月 31 日 18:10:40,398 [explicit-fetchindex-cmd] DEBUG PoolingClientConnectionManager - 连接发布:[id: 0][route: {}->http://solr-master.saltdev.sealdoc.com:8081][总存活数:1；分配的路线:10000 中的 1；总分配:10000]2013 年 10 月 31 日 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 发布目录:/opt/watchdox/solr-slave/data/index 2 false2013 年 10 月 31 日 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 重用缓存目录:CachedDir<>2013 年 10 月 31 日 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 发布目录:/opt/watchdox/solr-slave/data 0 false2013 年 10 月 31 日 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 重用缓存目录:CachedDir<>2013 年 10 月 31 日 18:10:40,427 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 发布目录:/opt/watchdox/solr-slave/data 0 false2013 年 10 月 31 日 18:10:40,428 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 使用目录完成:CachedDir<>2013 年 10 月 31 日 18:10:40,428 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 发布目录:/opt/watchdox/solr-slave/data/index.20131031180837277 0 true2013 年 10 月 31 日 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - 希望关闭/opt/watchdox/solr-slave/data/index.20131031180837277 [CachedDir<>]2013 年 10 月 31 日 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - 关闭目录:/opt/watchdox/solr-slave/data/index.201310311808372772013 年 10 月 31 日 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - 在核心关闭前删除目录:/opt/watchdox/solr-slave/data/index.201310311808372772013 年 10 月 31 日 18:10:40,878 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 从缓存中删除:CachedDir<>2013 年 10 月 31 日 18:10:40,878 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - 发布目录:/opt/watchdox/solr-slave/data/index 1 false2013 年 10 月 31 日 18:10:40,879 [explicit-fetchindex-cmd] 错误 ReplicationHandler - SnapPull 失败:org.apache.solr.common.SolrException:无法完全下载 _aa7_Lucene41_0.pos.已下载 0!=1081710在 org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1212)在 org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1092)在 org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:719)在 org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:397)在 org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:317)在 org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:218)
d=filecontent&generation=6814&qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "GET /solr-master/replication?command=filecontent&generation=6814&qt =%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "Host: solr-master.saltdev.sealdoc.com:8081[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "Connection: Keep-Alive[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - >> "[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> GET /solr-master/replication?command=filecontent&generation=6814& qt=%2Freplication&file=_aa7_Lucene41_0.pos&checksum=true&wt=filestream HTTP/1.1 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer ] 1.0 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> Host: solr-master.saltdev.sealdoc.com:8081 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG headers - >> Connection: Keep-Alive 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "4[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: 4 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "[0x0][0x0][0x0][0x0][\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: ^@^@^@^@ 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "0[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: 0 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG wire - << "[\r][\n]" 31 Oct 2013 18:10:40,361 [explicit-fetchindex-cmd] DEBUG DefaultHttpResponseParser - Garbage in response: 31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 closed 31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 shut down 31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG DefaultClientConnection - Connection 0.0.0.0:55266<->172.16.77.121:8081 closed 31 Oct 2013 18:10:40,398 [explicit-fetchindex-cmd] DEBUG PoolingClientConnectionManager - Connection released: [id: 0][route: {}->http://solr-master.saltdev.sealdoc.com:8081][total kept alive: 1; route allocated: 1 of 10000; total allocated: 1 of 10000] 31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data/index 2 false 31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Reusing cached directory: CachedDir<> 31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data 0 false 31 Oct 2013 18:10:40,425 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Reusing cached directory: CachedDir<> 31 Oct 2013 18:10:40,427 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data 0 false 31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Done with dir: CachedDir<> 31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data/index.20131031180837277 0 true 31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - looking to close /opt/watchdox/solr-slave/data/index.20131031180837277 [CachedDir<>] 31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - Closing directory: /opt/watchdox/solr-slave/data/index.20131031180837277 31 Oct 2013 18:10:40,428 [explicit-fetchindex-cmd] INFO CachingDirectoryFactory - Removing directory before core close: /opt/watchdox/solr-slave/data/index.20131031180837277 31 Oct 2013 18:10:40,878 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Removing from cache: CachedDir<> 31 Oct 2013 18:10:40,878 [explicit-fetchindex-cmd] DEBUG CachingDirectoryFactory - Releasing directory: /opt/watchdox/solr-slave/data/index 1 false 31 Oct 2013 18:10:40,879 [explicit-fetchindex-cmd] ERROR ReplicationHandler - SnapPull failed :org.apache.solr.common.SolrException: Unable to download _aa7_Lucene41_0.pos completely. Downloaded 0!=1081710 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1212) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1092) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:719) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:397) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:317) at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:218)
2013 年 10 月 31 日 18:10:40,910 [http-bio-8080-exec-8] DEBUG CachingDirectoryFactory - 重用缓存目录:CachedDir<>
31 Oct 2013 18:10:40,910 [http-bio-8080-exec-8] DEBUG CachingDirectoryFactory - Reusing cached directory: CachedDir<>
所以我将 httpcomponents jar 升级到最新的 4.3.x 版本，问题就消失了.httpcomponents jar 是 solrj 的依赖项，在 4.2.x 版本中，我升级到 httpclient-4.3.1、httpcore-4.3 和 httpmime-4.3.1我现在运行了几次复制，完全没有问题，它现在按预期工作.看来只有slave端才需要升级，不过我也要升级master.
So I upgraded the httpcomponents jars to their latest 4.3.x version and the problem disappeared. the httpcomponents jars which are dependencies of solrj where in the 4.2.x version, I upgraded to httpclient-4.3.1 , httpcore-4.3 and httpmime-4.3.1 I ran the replication a few times now and no problem at all, it is now working as expected. It seams that the upgrade is necessary only on the slave side but I'm going to upgrade the master too.
这篇关于solr ReplicationHandler - SnapPull 无法下载文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
