石墨图 - 我们可以多快地更新图表？ [英] Graphite Graph - how fast can we update the graph?

问题描述

我们正在尝试使用Graphite作为（近）实时图形化网络系统。然而，我们似乎无法以更快的速度提升石墨的速度。最终，我们希望有100毫秒的更新

阅读常见问题解答后，它听起来像是石墨速度很快 - 但这或者是误导性的，或者我不了解如何加快速度up stone

耳语的时间信息似乎使用UNIX时间戳

Graphite的可扩展性如何？

从CPU的角度来看，Graphite在前端和后端水平扩展，这意味着您可以简单地将更多机器添加到混合中以获得更高的吞吐量。它也具有容错能力，因为丢失后端机器会导致最小量的数据丢失（无论该机器在内存中缓存），并且如果您有足够的容量来处理负载，则不会中断系统。

从I / O角度来看，在负载下，Graphite可以非常迅速地在许多不同的文件上执行大量微小的I / O操作。这是因为发送给Graphite的每个不同指标都存储在自己的数据库文件中，类似于在RRD工作之上构建的工具（drraw，Cacti，Centreon等）的数量。实际上，Graphite最初确实使用RRD进行存储，直到出现需要新存储引擎的基本限制。

大量（几千个不同的度量标准更新）一个好的RAID阵列。如果磁盘无法跟上发生的大量小写操作（每个数据点只有几个字节，但大多数磁盘每秒不能做超过几千次I / O操作，则石墨的后端会缓存传入数据，即使他们很小）。发生这种情况时，Graphite的数据库引擎悄悄话允许碳一次写入多个数据点，因此只会增加整体吞吐量，但其代价是将多余的数据缓存在内存中，直到写入。

图表的实时性如何？

非常。即使在负载较重的情况下，每个时间间隔内进入的度量数量也远远大于存储系统可以执行的I / O操作速度，并且大量数据点正在缓存在存储管道中（请参阅前面的问题以获得解释），Graphite仍然绘制实时图。诀窍是，当Graphite Web应用程序接收到绘制图形的请求时，它同时从磁盘以及从预存储缓存中检索数据（如果您有多个后端服务器，则可能会分发这些数据），并将两个来源数据来创建一个实时图。

此外，它们只显示秒和没有小数点：$ b​​ $ b http://graphite.readthedocs.org/en/latest/config-carbon.html
和
来自和直到必须是一个符合AT-STYLE时间规范的时间规范这里： http://oss.oetiker.ch/rrdtool/doc/rrdfetch.en.html 。
http://graphite.wikidot.com/url-api-reference

那它是什么？石墨快吗？或者它处理大型数据集的速度很快 - 我们正在寻找一种简单易用的数据包数据网络接收器来直观地显示 - 石墨似乎是一个很好的解决方案，但现在我们已经配置完成并运行了所有配置，我猜测我们只是浪费了一个很多时间

谢谢！

解决方案

每个最精确定义的精度（接收到的额外数据点将被删除）存储在您的storage-schemas.conf中的一个数据点。 最高精度可能为1秒。 retentions = 1s ：6h，1min：7d，10min：5y p为了达到目标，你需要将一个聚合器放在Graphite前面。聚合器将采用所有度量标准并汇总数据，并刷新到Graphite存储以匹配存储模式。聚合器将对指标执行计算（平均值，总和，平均值等）并发送这些值。 e.g。在过去的一秒钟内，你平均需要14ms来处理请求，或者在过去的10秒内请求的总数是4234.

所以，虽然你可以' t报告的时间间隔小于1秒，那么您可以使用聚合器捕获 - 在1秒时间间隔内发生的事件的总数和平均值，并报告。

两种常见的选择是 StatsD 和在碳集合器中提供的Graphite 。

** StatsD，IMO是要走的路。它是一个网络守护进程，您可以单独运行并通过UDP发送给它。也就是说，你可以用carbon-aggregator.py来做同样的事情（例如UDP）。

We are trying to use Graphite for a (near) real-time graphing web system. However we cannot seem to speed graphite faster than 1 second update rates. Ultimately we would like to have 100ms updates

From reading the FAQ it makes it sound like graphite is fast - but this is either very misleading or I am not understanding how to speed up graphite

the timing information for whisper appears to use UNIX time stamps

How scalable is Graphite?

From a CPU perspective, Graphite scales horizontally on both the frontend and the backend, meaning you can simply add more machines to the mix to get more throughput. It is also fault tolerant in the sense that losing a backend machine will cause a minimal amount of data loss (whatever that machine had cached in memory) and will not disrupt the system if you have sufficient capacity remaining to handle the load.

From an I/O perspective, under load Graphite performs lots of tiny I/O operations on lots of different files very rapidly. This is because each distinct metric sent to Graphite is stored in its own database file, similar to how many tools (drraw, Cacti, Centreon, etc) built on top of RRD work. In fact, Graphite originally did use RRD for storage until fundamental limitations arose that required a new storage engine.

High volume (a few thousand distinct metrics updating minutely) pretty much requires a good RAID array. Graphite's backend caches incoming data if the disks cannot keep up with the large number of small write operations that occur (each data point is only a few bytes, but most disks cannot do more than a few thousand I/O operations per second, even if they are tiny). When this occurs, Graphite's database engine, whisper, allows carbon to write multiple data points at once, thus increasing overall throughput only at the cost of keeping excess data cached in memory until it can be written.

How real-time are the graphs?

Very. Even under heavy load, where the number of metrics coming in each time interval is much greater than the rate at which your storage system can perform I/O operations and lots of data points are being cached in the storage pipeline (see previous question for explanation), Graphite still draws real-time graphs. The trick is that when the Graphite webapp receives a request to draw a graph, it simultaneously retrieves data off the disk as well as from the pre-storage cache (which may be distributed if you have multiple backend servers) and combines the two sources of data to create a real-time graph.

Also They only show seconds and no fractional points here: http://graphite.readthedocs.org/en/latest/config-carbon.html and from and until must be a time specification conforming to the AT-STYLE time specification described here: http://oss.oetiker.ch/rrdtool/doc/rrdfetch.en.html. http://graphite.wikidot.com/url-api-reference

So what is it? Is graphite fast? or is it just fast to process large datasets - we are looking for a simple to use web receiver of packet data to display visually - Graphite seemed like a great solution but now that we have it all configured and running I am guessing we just wasted a lot of time

Thanks!

解决方案

Graphite will store at most one data point per the finest defined precision (additional data points received will be dropped) in your storage-schemas.conf. The finest precision possible is 1 second. e.g. retentions = 1s:6h,1min:7d,10min:5y

In order to meet your goals you'll need to put an aggregator in front of Graphite. The aggregator will take in all the metrics and aggregate the data, flushing to Graphite storage in order to match the storage schema. The aggregator will perform calculations (avg, sum, mean, etc) on the metrics and send those on. e.g. over the last second you averaged 14ms to process the request OR for the last 10 seconds the total number of requests was 4234.

So, while you can't report at a finer granularity than 1 second, you can use the aggregator to capture -the sum and average of- what has happened during the 1 second time interval and report that.

Two common choices are StatsD and the Graphite provided in carbon aggregator.

** StatsD, IMO is the way to go. It is a network daemon that you run separately and send to it over UDP. That said, you can do much the same (e.g UDP) with carbon-aggregator.py.

这篇关于石墨图 - 我们可以多快地更新图表？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！