Logstash / Elasticsearch / Kibana资源规划 [英] Logstash/Elasticsearch/Kibana resource planning

问题描述

如何计算资源（我怀疑，弹性搜索实例）根据负载：

How to plan resources (I suspect, elasticsearch instances) according to load:

加载我的意思是≈500K事件/分钟，每个包含8-10个字段。

With load I mean ≈500K events/min, each containing 8-10 fields.

我应该转动什么配置旋钮？
我是新的这个堆栈。

What are the configuration knobs I should turn? I'm new to this stack.

推荐答案

每分钟500,000事件是每秒8333事件，应该是对于小型集群（3-5台机器）来说很容易处理。

500,000 events per minute is 8,333 events per second, which should be pretty easy for a small cluster (3-5 machines) to handle.

问题在于保持720M的日常文档打开60天（43B文件）。如果10个字段中的每个字节为32字节，则为13.8TB的磁盘空间（接近28TB，单个副本）。

The problem will come with keeping 720M daily documents open for 60 days (43B documents). If each of the 10 fields is 32 bytes, that's 13.8TB of disk space (nearly 28TB with a single replica).

为了比较，我有5个节点在最大（64GB的RAM，31GB的堆），1.2B的文件占用了1.2TB的磁盘空间（双倍的副本）。该集群无法处理每台机器只有32GB内存的负载，但现在使用64GB的速度很快乐。这是我们10天的数据。

For comparison, I have 5 nodes at the max (64GB of RAM, 31GB heap), with 1.2B documents consuming 1.2TB of disk space (double with a replica). This cluster could not handle the load with only 32GB of RAM per machine, but it's happy now with 64GB. This is 10 days of data for us.

大致来说，您希望占用的磁盘空间比我的集群消耗10倍的文件数量的40倍。

Roughly, you're expecting to have 40x the number of documents consuming 10x the disk space than my cluster.

我没有确切的数字在我面前，但是我们使用doc_values的试点项目给了我们像90％的堆积。

I don't have the exact numbers in front of me, but our pilot project for using doc_values is giving us something like a 90% heap savings.

如果所有这些数学成立，并且doc_values都是好的，那么就实际的字节索引而言，可以使用类似的集群。我会征求关于拥有这么多单独文档的开销的其他信息。

If all of that math holds, and doc_values is that good, you could be OK with a similar cluster as far as actual bytes indexed were concerned. I would solicit additional information on the overhead of having so many individual documents.

我们已经做了一些弹性搜索调整，但是可能还有更多的可能

We've done some amount of elasticsearch tuning, but there's probably more than could be done as well.

我建议您从少量的64GB机器开始。您可以根据需要添加更多。将少数（较小的）客户端节点作为索引和搜索请求的前端。

I would advise you to start with a handful of 64GB machines. You can add more as needed. Toss in a couple of (smaller) client nodes as the front-end for index and search requests.

这篇关于Logstash / Elasticsearch / Kibana资源规划的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！