您将使用什么数据库用于日志记录（即als日志文件替换） [英] What database would you use for logging (i.e. als logfile replacement)

问题描述

在用grep等分析了一些千兆字节的日志文件之后，我想知道如何通过使用数据库来记录这些东西。什么数据库适合这个purpuse？当然，vanillia SQL数据库工作，但提供了大量的事务性保证等，你不需要在这里，如果你使用千兆字节的数据和非常快的插入率，这可能会使它很慢。因此，NoSQL数据库可能是正确的答案（比较此答案一些建议）。数据库的一些要求是：

After analyzing some gigabytes of logfiles with grep and the like I was wondering how to make this easier by using a database to log the stuff into. What database would be appropiate for this purpuse? A vanillia SQL database works, of course, but provides lots of transactional guarantees etc. which you don't need here, and which might make it slow if you work with gigabytes of data and very fast insertion rates. So a NoSQL database that could be the right answer (compare this answer for some suggestions). Some requirements for the database would be:

• 能够处理千兆字节或甚至数TB的数据


• 快速插入


• 应该可以对每个条目设置多个缩写（例如时间，会话ID，网址等）。

• Ability to cope with gigabytes or maybe even terabytes of data
• Fast insertion
• Multiple indizes on each entry should be possible (e.g. time, session id, URL etc.)
• If possible, it store the data in a compressed form, since logfiles are usually extremely repetitive.

更新：以下是一些SO问题：处理/报告的数据库建议大量日志文件类型数据和什么是好的NoSQL和非关系数据库解决方案的审计/日志数据库。

Update: There are already some SO-questions for this: Database suggestion for processing/reporting on large amount of log file type data and What are good NoSQL and non-relational database solutions for audit/logging database . However, I am curious which databases fulfill which requirements.

推荐答案

在尝试了很多nosql解决方案之后，我最好的投注是：

After having tried a lot of nosql solutions, my best bets would be:

• riak + riak搜索大可扩展性


• mysql / postgresql中的非规范化数据


• mongoDB如果你不介意等待


• couchdb如果你知道你正在搜索什么

• riak + riak search for great scalability
• unnormalized data in mysql/postgresql
• mongoDB if you don't mind waiting
• couchdb if you KNOW what you're searching for

Riak + Riak轻松搜索标度（REALLY！），并允许您对数据进行形式查询。您也可以轻松地混合数据模式，甚至可以使用innostore作为后端来压缩数据。

Riak + Riak Search scale easily (REALLY!) and allow you free form queries over your data. You can also easily mix data schemas and maybe even compress data with innostore as a backend.

如果您真的想使用索引，MongoDB很难扩展到几千兆字节的数据而不是慢下来爬行。考虑单节点性能并提供索引创建，它真的很快。一旦你的工作数据集不适合在内存中，它就成为一个问题...

MongoDB is annoying to scale over several gigabytes of data if you really want to use indexes and not slow down to a crawl. It is really fast considering single node performance and offers index creation. As soon as your working data set doesn't fit in memory anymore, it becomes a problem...

mysql / postgresql仍然相当快，允许自由形式查询感谢到通常的b +树索引。如果某些字段未显示在每个字段中，请查看postgres的部分索引记录。他们还提供压缩表，由于模式是固定的，你不会一遍又一遍地保存你的行名（这是很多nosql解决方案通常发生的情况）。

mysql/postgresql is still pretty fast and allows free form queries thanks to the usual b+tree indexes. Look at postgres for partial indexes if some of the fields don't show up in every record. They also offer compressed tables and since the schema is fixed, you don't save your row names over and over again (that's what usually happens for a lot of the nosql solutions)

CouchDB很好，如果你已经知道你想要查看的查询，他们增量地图/基于减少的视图是一个伟大的系统。

CouchDB is nice if you already know the queries you want to see, their incremental map/reduce based views are a great system for that.

这篇关于您将使用什么数据库用于日志记录（即als日志文件替换）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！