存储和显示统计数据 [英] Storing And Displaying Stats

问题描述

我将用 PHP 编写一些软件来解析日志文件并聚合数据，然后将它们显示在图表中(如条形图，而不是顶点和边).

I am going to be writing some software in PHP to parse log files and aggregate the data then display them in graphs (like bar graphs, not vertices and edges).

是的，它基本上是商业智能软件，我的公司有一个完整的团队，但显然他们做得并不好(加载页面 10 分钟就是不行).

Yeah, it's basically business intelligence software which my company has an entire team for but apparently they don't do a great job (10 minutes to load a page just doesn't do it).

这是我必须做的:

• 日志文件是数据文件，用于存储我们在办公室运行的统计服务器的原始数据(我们向统计服务器发送异步调用，有点像谷歌分析).它以 csv 格式存储数据.
• 编写脚本来解析文件并将数据聚合到数据库中(或者我正在考虑使用 redis)
• 将有数以百万计的内容需要汇总，因此在显示统计数据时必须快速

我知道数据库的 OLAP，但如果我想使用 redis，你认为它可以扩展到大量数据吗?要解析文件，您认为 PHP 脚本就足够了，还是我应该使用 C/C++ 之类的更快的东西?

I know about OLAP for the DB, but if i want to go with redis do you think it would scale for large volumes of data? To parse the files do you think a PHP script would suffice or should i go with something faster like C/C++?

基本上，我想获得一些关于完成任务的不同方式的有趣想法.它必须快速且可扩展.

Basically i would like to get some interesting ideas about different ways to accomplish my task. It must be fast and scale.

有什么想法吗?

推荐答案

听起来像在您谈论的尺度上，您需要将数据聚合和显示分开.也就是说，您应该有一些进程在生成日志文件时接收它们，解析它们并将数据插入数据库；这将是一项漫长而复杂的任务.然后当用户想要显示数据的图形时，他们可以向 PHP 服务器发出请求，PHP 服务器将从数据库中提取数据并构建他们想要的显示.通过这种方式，您的解析与您的显示请求分离(尽管它仍然是串行相关的，您的解析可以在日志文件可用时开始，因此解析它们所涉及的延迟在显示时被隐藏).

It sounds like at the scales you're talking about, you need to separate the data aggregation and display. That is, you should have some process working to receive the log files when they're generated, parse them and insert the data into the database; that will be a long, complicated task. Then when a user wants to display a graph of the data, they can make a request to the PHP server, which will pull the data from the database and construct the display they want. In this way, your parsing is separated from your display request (although it's still serially dependent, your parsing can begin when the logfiles become available, and therefore, the lag involved in parsing them is hidden at display time).

这篇关于存储和显示统计数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！