记录到数据库而不是日志文件 [英] Log to database instead of log files
问题描述
我有兴趣将所有Rails应用程序日志记录发送到数据库（MySQL或MongoDB），不管是日志文件还是日志文件。有几个原因，其中大多数都关心日志文件分析。我们已经使用Google Analytics（分析），但我们希望做的各种事情在Google Analytics（分析）中不可行。
I'm interested in sending all Rails application logging to a database (MySQL or MongoDB) either in addition to or instead of to a log file. There are a few reasons, most of which are concerned about log file analysis. We already use Google Analytics, but there are a variety of things we want to do that aren't as workable in Analytics.
此外，我想通过查看日志来实时调查问题。筛选日志文件是一个乏味的方式来做到这一点，我想做的更好的搜索和过滤比日志文件（轻松）允许。
Furthermore, I'd like to do "real time" investigation of issues by looking at logs. Sifting through a log file is a tedious way to do that, and I'd like to do better searching and filtering than a log file (easily) allows for.
最后，我经常想检查更接近网站访问者行为的东西：例如跟踪网站的路径，以便我可以看到最后一页是什么用户在看错误发生之前。由于我们有多个应用程序服务器，单独的日志文件使这是一个真正的痛苦。如果所有数据都在数据库中，那么我可以很容易地看到给定访问者的正确页面顺序。我知道Syslog将是解决这个特殊的事情（单个日志文件/存储库）的一种方法，但我想把它与数据库搜索更好的搜索能力相结合。
Finally, I often want to examine something closer to site visitor behavior: tracing the path through the site for example, so that I can see what the last page was that a user was looking at before an error occurred. Given we have multiple app servers, the separate log files make this a real pain. If all the data were in a database, I could then easily see the proper sequence of pages for a given visitor. I know that Syslog would be one way to solve this particular thing (single log file/repository), but I want to combine that with better searching abilities that I associate with database searches.
我想知道有什么人推荐解决这个。你直接日志到数据库，或者你把日志文件转储到一个数据库（但你的方法是什么，所以它本质上是实时的/作为日志文件本身的最新）？
I'm wondering what folks recommend to solve this. Do you directly log to a database, or do you dump log files into a DB (but what's your approach for that so that it's essentially realtime/as up to date as the logfile itself)?
我目前正在确定我希望这个日志记录的级别，因为我注意到的另一件事是编写一个小的Rack过滤器，它会记录所有请求。这将丢失正常Rails日志转储的所有额外输出（所有SQL和缓存命中和未命中的输出等），但它会实现我的目标的很大一部分，并且似乎具有不打扰的优势
I am currently determining at what level I'd like this logging, because another thing I looked at is writing a small Rack filter that would log all requests. This would miss all the extra output that the normal Rails logging dumps out (all the SQL and output on cache hits and misses, etc.), but it would achieve a big part of my goal, and seems to have the advantage of not disturbing anything else in the system.
无论如何，我不是寻找一个正确的答案，更多的讨论和信息，任何其他人可能在这一灯光下做的。
Anyway, I am not looking for one right answer, more of a discussion and information on what anyone else might be doing in this same light.
推荐答案
我公司已经将一些结构化流量信息直接记录到MySQL日志数据库中。此数据库将下游复制到另一个数据库。所有分析都运行于最终数据库复制。我们的网站维持相当多的流量。到目前为止，它似乎没有任何大的问题。然而，我们的IT部门对当前设置的可扩展性有一些日益增长的关注，并建议我们将日志信息卸载到正确日志文件。然后，日志文件将重新插入到相同的下游数据库表中。这使我想到这个问题。 :)
My company have been logging some structured traffic info straight into a MySQL log database. This database is replicated downstream to another database. All analytics run off the final database replication. Our site sustain quite a bit of traffic. So far, it doesn't seem to have any major problems. However, our IT department have some growing concerns regarding to the scalability of the current setup and is suggesting that we offload the log info onto "proper" log-files. The log-files will then be reinserted back into the same downstream database tables. Which brings me to this question. :)
这里有一些关于日志文件vs log-db（关系）主题的优点和缺点：
Here are some of pros and cons that I see regarding to the subject of log-files vs log-db (relational):
- 日志文件是快速，可靠和可扩展的（至少我听说过Yahoo!大量使用日志文件进行点击跟踪分析）。
- 日志文件可以非常灵活，因为您可以向其中写几乎任何内容。
- 日志文件需要大量解析，并且可能需要进行映射缩减类型的数据提取设置。
- 日志数据库结构到你的应用程序，使一些功能的周转时间短得多。这可能是一个祝福或诅se。
- log-db可以减少日志记录的噪音和冗余，因为日志文件是一个非常复杂的应用程序和分析代码库。插入只有在log-db允许执行update和关联插入（如果你敢的标准化）的情况下。
- 如果你使用数据库，log-db可以快速和可扩展分区和/或多日志数据库（通过下游复制重新加入数据）
- log-files are fast, reliable, and scalable (At least I have heard Yahoo! makes heavy uses of log files for their click tracking analytics).
- log-files are easy for sys-admin to maintain.
- log-files can be very flexible since you can write almost anything to it.
- log-files requires heavy parsing and potentially a map-reduced type of setup for data-extraction.
- log-db structures are a lot closer to your application, making some feature's turn around time a lot shorter. This can be a blessing or a curse. Probably a curse in the long run since you'll most likely end up with a highly coupled application and analytic code base.
- log-db can reduce logging noises and redundancies since log-files are insert only where as log-db gives you the ability to do update and associated-insert (normalization if you dare).
- log-db can be fast and scalable too if you go with database partitioning and/or multi-log databases (rejoin data via downstream replications)
我认为需要对日志数据库进行一些压力测试我的情况。这样至少我知道有多少余量。
I think some stress tests on the log database are needed in my situation. This way at least I know how much headroom I have.
最近，我一直在研究一些基于键值/文档的数据库，如Redis，Tokyo Cabinet和MongoDB。这些快速插入数据库可能是最佳点，因为它们提供持久性，高（写）吞吐量和不同程度的查询能力。他们可以使数据提取过程比通过日志文件的分析解析和减少地图更简单。
Recently, I've been looking into some key-value / document-based databases like Redis, Tokyo Cabinet, and MongoDB. These fast inserting databases can potentially be the sweet spot since they provide persistence, high (write) throughputs, and querying capabilities to varying degrees. They can make the data-extraction process much simpler than parsing and map-reducing through gigs of log files.
从长远来看，我相信拥有强大的分析数据仓库至关重要。从分析数据中释放应用程序数据，反之亦然可以是一个大赢家。
In the long run, I believe it is crucial to have a robust analytics data warehouse. Freeing application data from analytic data and vice versa can be a big WIN.
最后，我想指出，StackOverflow上有很多类似/
Lastly, I would just like to point out there are many similar / closely related questions here on StackOverflow in case you want to broaden your discussion.
- Storage of many log files
- Is writing server log files to a database a good idea?
- Using a SQL Server for application logging. Pros/Cons?
- Fast Search in Logs
- Separate production database for logging
- You Log to Your DB, Where Do You Log When Your DB is Down?
编辑：
rsyslog 看起来很有趣。它使您能够直接写入MySQL。如果你使用Ruby，你应该看看logging gem。它提供多目标日志记录功能。这太好了。
rsyslog looks very interesting. It gives you the ability to write directly to MySQL. If you are using Ruby, you should have a look at the logging gem. It provides multi-target logging capabilities. It's really nice.
这篇关于记录到数据库而不是日志文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
