使用NGINX将跟踪数据转发到Flume [英] Using NGINX to forward tracking data to Flume

问题描述

我正在根据我们通过简单的图像信标收集的检测数据为我们的网络媒体资源提供分析.我们的数据管道从Flume开始，我需要最快的方法来解析查询字符串参数，形成一条简单的文本消息并将其推入Flume.

I am working on providing analytics for our web property based on instrumentation data we collect via a simple image beacon. Our data pipeline starts with Flume, and I need the fastest possible way to parse query string parameters, form a simple text message and shove it into Flume.

出于性能方面的考虑，我倾向于使用nginx.由于已经支持从内存提供静态图像，因此我的任务简化为处理查询字符串并将消息转发到Flume.因此，问题是:

For performance reasons, I am leaning towards nginx. Since serving static image from memory is already supported, my task is reduced to handling the querystring and forwarding a message to Flume. Hence, the question:

将Nginx与Flume集成的最简单可靠的方法是什么?我正在考虑使用syslog(Flume支持syslog侦听器)，但是我在如何配置nginx以便将自定义日志消息转发到在远程服务器和自定义端口上运行的syslog(或只是TCP)侦听器方面感到困惑.现有的用于Nginx的第三方模块是否可行，还是我必须编写自己的第三方模块?

What is the simplest reliable way to integrate nginx with Flume? I am thinking about using syslog (Flume supports syslog listeners), but I struggle with how to configure nginx to forward custom log messages to a syslog (or just TCP) listener running on a remote server and on a custom port. Is it possible with existing 3rd party modules for nginx or would I have to write my own?

另外，任何您可以推荐的用于编写快速$ args解析器的东西都将不胜感激.

Separately, anything existing you can recommend for writing a fast $args parser would be much appreciated.

如果您认为我走在完全错误的道路上，并且可以推荐一些更好的性能，请随时告诉我.

If you think I am on a completely wrong path and can recommend something better performance-wise, feel free to let me know.

提前谢谢！

推荐答案

您应该像 tail -f 一样解析nginx日志文件，然后将结果传递给Flume.这将是最简单，最可靠的方法.syslog的问题在于它会阻塞nginx并可能在高负载下完全卡住，或者出现问题(这就是nginx不支持它的原因).

You should parse nginx log file like tail -f do and then pass results to Flume. It will be the most simple and reliable way. The problem with syslog is that it blocks nginx and may completely stuck under high-load or if something goes wrong (this is why nginx doesn't support it).

这篇关于使用NGINX将跟踪数据转发到Flume的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！