石墨.某些指标丢失,但在tcpdump中可见 [英] Graphite. Some metrics lost, but seen in tcpdump
问题描述
我使用石墨的时间很长,第一次遇到一些指标丢失…的问题?
通过tcpdump -nA dst port 2003
,我可以看到度量标准已传递到Graphite节点.
另外,其中一些是在耳语数据库中创建的,并在/var/log/carbon/updates.log
中可见
但是其中大多数都不会出现在任何地方.
I'm using graphite for pretty long time, and first time facing issue with some metrics getting… lost?
Through tcpdump -nA dst port 2003
I can see that metrics are delivered to Graphite node.
Also, some of them are getting created in whisper database, and seen in /var/log/carbon/updates.log
But most of them are not appearing anywhere.
所以我的问题是:如何调试?如何证明Graphite确实从eth0接收了这些指标?
So my question is: how do I debug it? How do I prove that Graphite really receives these metrics from eth0?
除碳中的updates.log
外,我找不到任何调试日志.
I couldn't find any debug logs except for updates.log
in carbon.
日志:
sudo tcpdump -An dst port 2003 | grep 172_31_00_01 | grep requests
backend.dev.172_31_00_01.requests.max 60554.34 1453734067 backend.dev.172_31_00_01.requests.mean 16714.87 1453734067 backend.dev.172_31_00_01.requests.min 2.93 1453734067 backend.dev.172_31_00_01.requests.stddev 12185.74 1453734067 backend.dev.172_31_00_01.requests.p50 16415.87 1453734067 backend.dev.172_31_00_01.requests.p75 20314.51 1453734067 backend.dev.172_31_00_01.requests.p95 41526.36 1453734067 backend.dev.172_31_00_01.requests.p98 54370.59 1453734067 backend.dev.172_31_00_01.requests.p99 60368.68 1453734067 backend.dev.172_31_00_01.requests.p999 60553.31 1453734067 backend.dev.172_31_00_01.requests.count 3141 1453734067 backend.dev.172_31_00_01.requests.m1_rate 2.02 1453734067 backend.dev.172_31_00_01.requests.m5_rate 1.95 1453734067 backend.dev.172_31_00_01.requests.m15_rate 1.20 1453734067 backend.dev.172_31_00_01.requests.mean_rate 0.66 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.max 152.59 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.mean 41.86 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.min 0.82 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.stddev 24.84 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.p75 57.51 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.p95 85.78 1453734067
backend.dev.172_31_00_01.requests.max 60554.34 1453734067 backend.dev.172_31_00_01.requests.mean 16714.87 1453734067 backend.dev.172_31_00_01.requests.min 2.93 1453734067 backend.dev.172_31_00_01.requests.stddev 12185.74 1453734067 backend.dev.172_31_00_01.requests.p50 16415.87 1453734067 backend.dev.172_31_00_01.requests.p75 20314.51 1453734067 backend.dev.172_31_00_01.requests.p95 41526.36 1453734067 backend.dev.172_31_00_01.requests.p98 54370.59 1453734067 backend.dev.172_31_00_01.requests.p99 60368.68 1453734067 backend.dev.172_31_00_01.requests.p999 60553.31 1453734067 backend.dev.172_31_00_01.requests.count 3141 1453734067 backend.dev.172_31_00_01.requests.m1_rate 2.02 1453734067 backend.dev.172_31_00_01.requests.m5_rate 1.95 1453734067 backend.dev.172_31_00_01.requests.m15_rate 1.20 1453734067 backend.dev.172_31_00_01.requests.mean_rate 0.66 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.max 152.59 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.mean 41.86 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.min 0.82 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.stddev 24.84 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.p75 57.51 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.p95 85.78 1453734067
$ pwd
/var/lib/graphite/whisper/backend/dev/172_31_00_01/requests
$ ls -Rl
.:
total 1796
drwxr-xr-x 2 _graphite _graphite 4096 Jan 25 14:25 mark_sessionid_active
-rw-r--r-- 1 _graphite _graphite 1831744 Jan 25 15:05 mean.wsp
./mark_sessionid_active:
total 3584
-rw-r--r-- 1 _graphite _graphite 1831744 Jan 25 15:05 min.wsp
-rw-r--r-- 1 _graphite _graphite 1831744 Jan 25 15:05 stddev.wsp
PS:这不是新安装,它已经可以使用几个月了,直到今天都没有丢失任何指标.
PS: It's not new installation, it works for several months now, and no metrics was lost until today.
推荐答案
carbon.conf
中有MAX_CREATES
设置.将其设置为较高的值(例如1000)或inf
即可解决此问题.
There is MAX_CREATES
setting in carbon.conf
. Setting it to high value (like 1000) or inf
solves this.
这篇关于石墨.某些指标丢失,但在tcpdump中可见的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!