elasticsearch批量索引变慢随着时间的推移索引和文件的常数 [英] elasticsearch bulk indexing gets slower over time with constant number of indexes and documents
问题描述
我遇到该散装索引性能使用.NET NEST客户端和ElasticSearch降低随时间索引的恒定量的文件和数目
I am experiencing that bulk indexing performance using the .NET NEST client and ElasticSearch degrades over time with a constant amount of indexes and number of documents.
我们正在运行 ElasticSearch版本：0.19.11，JVM：23.5-B02 与Ubuntu的服务器12.04.1 LTS 64位和Sun Java一m1.large亚马逊实例7.有没有其他的，除了什么来与Ubuntu的安装此实例中运行。
We are running ElasticSearch Version: 0.19.11, JVM: 23.5-b02on a m1.large Amazon instance with Ubuntu Server 12.04.1 LTS 64 bit and Sun Java 7. There is nothing else running on this instance except what comes along with the Ubuntu install.
亚马逊M1大实例：从的http：// AWS。 amazon.com/ec2/instance-types/
7.5 GiB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage
64-bit platform
I/O Performance: High
EBS-Optimized Available: 500 Mbps
API name: m1.large
ES_MAX_MEM设为4g和ES_MIN_MEM设为2克
ES_MAX_MEM is set to 4g and ES_MIN_MEM is set to 2g
每天晚上我们的索引/重新索引我们的.NET应用程序中使用NEST〜15000的文件。在任何给定时间仅存在一个索引以＆lt; = 15000文档
Every night we index/reindex ~15000 documents using NEST in our .NET application. At any given time there is only one index with <= 15000 documents.
当服务器第一次安装的索引和搜索是快速的头几天，然后索引开始变得越来越慢。大宗索引的索引的时间和一段时间后，将需要长达15秒的批量操作，完成100份文件。之后，我们开始看到很多下面的异常和索引的磨停了下来。
when the server was first installed the indexing and search was fast for the first couple of days, then indexing started to get slower and slower. the bulk indexing indexes 100 documents at a time and after a while it would take up to 15s for a bulk operation to finish. after that we started to see alot of the following exception and the indexing grinding to a halt.
System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) :
在builk索引实现看起来像这样
The builk indexing implementation looks like this
private ElasticClient GetElasticClient()
{
var setting = new ConnectionSettings(ConfigurationManager.AppSettings["elasticSearchHost"], 9200);
setting.SetDefaultIndex("products");
var elastic = new ElasticClient(setting);
return elastic;
}
private void DisableRefreshInterval()
{
var elasticClient = GetElasticClient();
var s = elasticClient.GetIndexSettings("products");
var settings = s != null && s.Settings != null ? s.Settings : new IndexSettings();
settings["refresh_interval"] = "-1";
var result = elasticClient.UpdateSettings(settings);
if (!result.OK)
_logger.Warn("unable to set refresh_interval to -1, {0}", result.ConnectionStatus == null || result.ConnectionStatus.Error == null ? "" : result.ConnectionStatus.Error.ExceptionMessage);
}
private void EnableRefreshInterval()
{
var elasticClient = GetElasticClient();
var s = elasticClient.GetIndexSettings("products");
var settings = s != null && s.Settings != null ? s.Settings : new IndexSettings();
settings["refresh_interval"] = "1s";
var result = elasticClient.UpdateSettings(settings);
if (!result.OK)
_logger.Warn("unable to set refresh_interval to 1s, {0}", result.ConnectionStatus == null || result.ConnectionStatus.Error == null ? "" : result.ConnectionStatus.Error.ExceptionMessage);
}
public void Index(IEnumerable<Product> products)
{
var enumerable = products as Product[] ?? products.ToArray();
var elasticClient = GetElasticClient();
try
{
DisableRefreshInterval();
_logger.Info("Indexing {0} products", enumerable.Count());
var status = elasticClient.IndexMany(enumerable as IEnumerable<Product>, "products");
if (status.Items != null)
_logger.Info("Done, Indexing {0} products, duration: {1}", status.Items.Count(), status.Took);
if (status.ConnectionStatus.Error != null)
{
_logger.Error(status.ConnectionStatus.Error.OriginalException);
}
}
catch(Exception ex)
{
_logger.Error(ex);
}
finally
{
EnableRefreshInterval();
}
}
重新启动elasticsearch守护进程似乎没有做任何任何区别，但删除索引和重新索引一切一样。但过了几天，我们也有同样的慢索引的问题。
Restarting the elasticsearch daemon does not seem to make any difference whatsoever, but deleting the index and re-indexing everything does. But after a few days we would have the same slow-indexing problem.
我刚刚删除索引后，希望这会保证指数的降低每个批量索引操作后重新启用刷新间隔增加了优化。
I just deleted the index and added an Optimize after the re-enabling of the refresh interval after each bulk-index operation in the hope that this might keep the index from degrading.
...
...
finally
{
EnableRefreshInterval();
elasticClient.Optimize("products");
}
我做得可怕的错误在这里？
Am I doing something horribly wrong here?
推荐答案
对不起 - 刚开始写另一相当长的注释，以为我只是坚持这一切的答案，以防它的好处别人...
Sorry - just started writing another quite long comment and thought I'd just stick it all in an answer in case it benefits someone else...
ES_HEAP_SIZE
我注意到这里的第一件事是，你说你设置的最大值和最小值的堆值elasticsearch为不同的值。这些应该是相同的。在配置/的init.d脚本，应该有一个EX_HEAP_SIZE，您可以设置。请务必只设置这个（而不是最小值和最大值），因为它会设置最小值和最大值为相同的值，它是你想要的。如果当你开始需要更多的内存不这样做的JVM将阻止Java进程 - 看到停电在GitHub上的这篇大文章最近（这里的引号）：
The first thing I noticed here is that you said you set the max and min heap values for elasticsearch to different values. These should be the same. In the configuration / init.d script there should be an EX_HEAP_SIZE that you can set. Be sure to only set this (and not the min and max values) as it will set the min and max values to the same value which is what you want. If you don't do this the JVM will block java processes when you start to need more memory - see this great article of an outage at github very recently (here's a quote):
设置ES_HEAP_SIZE环境变量来让JVM使用的最小和最大内存相同的值。配置JVM具有不同的最低和最高值意味着每当JVM需要附加存储器（直到最大），它会阻止在Java进程分配它。结合旧的Java版本，这说明我们的节点展出当介绍到更高的负载，当他们被打开了公开搜查连续的内存分配的停顿。该elasticsearch小组建议的系统内存50％的设置。
Set the ES_HEAP_SIZE environment variable so that the JVM uses the same value for minimum and maximum memory. Configuring the JVM to have different minimum and maximum values means that each time the JVM needs additional memory (up to the maximum), it will block the Java process to allocate it. Combined with the old Java version, this explains the pauses that our nodes exhibited when introduced to higher load and continuous memory allocation when they were opened up to public searches. The elasticsearch team recommends a setting of 50% of system RAM.
另外，请查阅这个伟大的职位了解elasticsearch配置从战壕。
Also check out this great post for more elasticsearch config from the trenches.
锁定内存停止交换
从我的研究，我发现，你也应该锁定的可用内存java程序的金额，以避免内存交换。我不是这方面的专家，但我一直在说的是，这也将杀死性能。您可以在elasticsearch.yml配置文件中找到bootstrap.mlockall。
From my research I've found that you should also lock the amount of memory available to the java process to avoid memory swapping. I'm no expert in this field but what I've been told is that this will also kill performance. You can find bootstrap.mlockall in your elasticsearch.yml config file.
升级
Elasticsearch仍是相当新。打算升级亦常为那些你在（0.19.11）版本和当前版本（0.20.4）之间被引入的错误修复是非常显著。请参阅 ES网站了解详细信息。你对Java 7的这绝对是正确的道路要走，我开始在Java 6中，并很快意识到，它只是不够好，尤其是对大容量插入。
Elasticsearch is still quite new. Plan to upgrade fairly frequently as the bug fixes that have been introduced between the version you were on (0.19.11) and the current version (0.20.4) are very significant. See the ES site for details. You're on Java 7 which is definitely the right way to go, I started on Java 6 and realized quickly that it just wasn't good enough, especially for bulk inserting.
插件
最后，其他人谁遇到类似的问题，让安装的节点的概述和JVM一个体面的插件。我建议 bigdesk - 运行bigdesk然后打elasticsearch一些批量插入并观看了怪堆内存模式，一个非常大的线程等的数量，这一切都没有！
Finally, to anyone else who experiences similar issues, get a decent plugin installed for an overview of your nodes and the JVM. I recommend bigdesk - run bigdesk and then hit elasticsearch with some bulk inserts and watch out for strange heap memory patterns, a very large number of threads etc, it's all there!
希望有人认为这有用！
干杯， 詹姆斯
这篇关于elasticsearch批量索引变慢随着时间的推移索引和文件的常数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
