弹性搜索盘空间计算 [英] Elastic search disc space calculation

问题描述

如何计算弹性搜索服务器磁盘空间，每天 4TB 日志需要多少个节点.

How to calculate elastic search server disc space, how much nodes required for 4TB logs per day.

• 弹性搜索索引中需要存储多少磁盘空间?
• 如何计算节点数?
• 索引被压缩了吗?
• 从logstash到行格式的10G日志需要多少磁盘空间?
• 索引是否压缩?

以下是true否是false，以及为什么将节点配置为使用5个分片

Is the following true of false and why, if node is configured to use 5 shards

5 个分片节点是否存储 10G 日志需要 50G 磁盘空间?

Is storing 10G logs needs 50G disc space for 5 shards node?

推荐答案

许多问题需要长答案.简而言之:

Many questions demanding long answers. Here in short:

1. 空格:视情况而定.你有二进制数据吗?还是未压缩的文字?您想要多少个副本-可能是0?如果原始文本和0个副本:您所需的空间很可能<您的原始数据大小.
2. 对于您的方案而言，也许一个节点就足够了.或者您有1000次点击/秒?然后，您可能需要更多的节点. (几乎线性可伸缩=>如果需要，可以添加节点).
3. 是的，从ES版本0.90左右开始，默认设置为压缩
4. 通常，您还希望将数据存储到磁盘，而不仅是存储在内存中(

1. Space: depends. Have you binary data? Or uncompressed text? How many replica do you want - maybe 0? If raw text and 0 replica: space you need is most likely < your original data size.
2. Maybe one node is enough for your scenario. Or You have 1000 hits/s ? Then you need maybe more nodes. (Nearly linear scalable => adding nodes if need arises).
3. Yes, default is set to compressed since ES version 0.90 or so
4. Normally you also want to store the data to disk, not only in memory (although would be possible) => see 1.
5. see 3.

不是space=nodes*rawData.更像是space=compress(rawData)*replicas.

成功的关键是自己进行试验，以找出适合您需求的东西(这些情况因情况而异).

Key to success is to experiment on your own to find out what fits to your needs (these are really varying from scenario to scenario).

另请参阅以下答案以获得更好的理解:

See also these answers for better understanding:

Elasticsearch中的碎片和副本

ElasticSearch-每个节点的最佳分片数量

这篇关于弹性搜索盘空间计算的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！