rsyslog性能优化 [英] rsyslog performance optimization
问题描述
如何设置 rsyslog 以获得最佳性能?
What should be done to setup rsyslog for best performance?
- 我们可以允许丢失服务器崩溃一些项目或只是失去了.
- 我们将日志保存到MySQL数据库.
- 我们希望能够每秒至少处理100次日志写入，并且延迟为0.001-0.005秒.
- 我们正在从PHP应用程序编写日志.
感谢您的帮助.
推荐答案
我们刚刚使用MongoDB作为数据库进行了类似的练习，所以我将在文档中记录我们所做的事情，希望对您有所帮助.
We just went through a similar exercise using MongoDB as the database, so I'll document what we did and hope it helps you.
这是我们第一次使用rsyslog，因此花费了一些精力来找到正确的文档并将所有内容整理在一起.最后，我们的测试驱动程序(我们使用的是SoapUI)能够通过使用rsyslog编写交易摘要记录的php Web服务获得1000 TPS.
It was our first time using rsyslog, so it took a bit of effort to find the right documentation and piece everything together. In the end, our test drivers (we're using SoapUI) are able to get 1000 TPS through a php web service that uses rsyslog to write a summary record of the transaction.
我们找到了以下使我们入门的文章:
We found the following articles that got us started:
- http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
- http://www.rsyslog.com/tag/mongodb/
概述是，您将启用rsyslog的队列基础结构，以在守护程序的内存队列已满时将传入消息写入磁盘.在我们的例子中，我们启用了$ ActionQueueSaveOnShutdown，这听起来像您不需要.然后，您将配置rsyslog规则集以解析传入的消息，并将它们传递给MySQL的输出处理程序.最后，您的php脚本将使用openlog()和syslog()编写要记录的任何数据.哦，为了启用json/mongo插件，我们还必须从源代码编译rsyslog，这本身就是一个练习.我们正在Ubuntu 12.04上使用rsyslog 7.4.5.
The overview is that you'll enable rsyslog's queue infrastructure for writing incoming messages to disk when the daemon's memory queue is full. In our case, we enabled $ActionQueueSaveOnShutdown, which it sounds like you don't need. Then you'll configure the rsyslog ruleset to parse the incoming messages and pass them to an output handler for MySQL. Finally, your php script will use openlog() and syslog() to write whatever data you want to log. Oh, we also had to compile rsyslog from source, in order to enable the json/mongo plugins, and that was an exercise in itself. We're using rsyslog 7.4.5 on Ubuntu 12.04.
我当然不是rsyslog的专家，但是可以为您提供我们的配置文件和代码作为起点.再次，它们是针对MongoDB的，希望它可以使您对实现中的操作和更改内容有所了解.
I'm certainly not an expert on rsyslog, however can give you our config files and code as a starting point. Again, they're for MongoDB, hopefully it gives you an idea of what to do and where to change things for your implementation.
祝你好运！
/etc/rsyslog.conf:
/etc/rsyslog.conf:
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
# Load modules for MongoDB integration: json parser and MongoDB output driver
module(load="mmjsonparse")
module(load="ommongodb")
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
# Set the default permissions for all log files.
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
# Where to place spool files
$WorkDirectory /var/spool/rsyslog
# use queue to decouple the db writes from default message handling
# From http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
$MainMsgQueueFileName mainq # set file name for main queue, also enables disk mode
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName mongodbq # set file name for mongo db queue, enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # write all queue data to disk when rsyslogd is
# terminated (default is off)
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
/etc/rsyslog.d/10-mongo.conf:
/etc/rsyslog.d/10-mongo.conf:
input(type="imuxsock" socket="/dev/log")
template(name="mongodblocal" type="subtree" subtree="$!")
# use json parser for all "local0" facility messages,
# if parsed successfully run the template to load the
# message into the MongoDB database.
if $syslogfacility-text == 'local0' then {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
# set some local vars that are appended onto the
# document that's written to MongoDB
set $!time = $timestamp;
set $!sys = $hostname;
set $!procid = $syslogtag;
set $!syslog_fac = $syslogfacility;
set $!syslog_sever = $syslogpriority;
set $!pid = $procid;
action(type="ommongodb" server="127.0.0.1" db="test" collection="syslog" template="mongodblocal")
}
}
/etc/rsyslog.d/50-default.conf:注意:这将禁用"local0"消息以进行默认处理.
/etc/rsyslog.d/50-default.conf: Note: this disables "local0" messages for default handling.
# First some standard log files. Log by facility.
auth,authpriv.* /var/log/auth.log
# don't write "local0" messages to syslog,
# as they're processed using ommongodb (see 10-mongo.conf)
*.*;local0,auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
mail.err /var/log/mail.err
# Logging for INN news system.
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
# Emergencies are sent to everybody logged in.
*.emerg :omusrmsg:*
与PHP Web服务相关的调用:
php web service related calls:
// open syslog, include the process ID and open the connection to the logger
// immediately, and use a user defined logging mechanism Local0
openlog($SCRIPT_NAME, LOG_PID | LOG_NDELAY, LOG_LOCAL0);
// note: calling closelog() is optional, and we don't use it
...
// construct $doc, which is what will be logged, change this as appropriate
// for your implementation; here $ary_headers is the request's HTTP headers,
// and $request/$response are what was posted/returned
$doc = array("headers" => $ary_headers
,"request" => $request
,"response" => $response
);
...
// write the log entry to syslog, where it queues it and writes it to MongoDB
// NOTE: need the '@cee: ' prefix so the rsyslog json parser will process it
// See: http://www.rsyslog.com/doc/rsyslog_conf_modules.html/mmjsonparse.html
// JSON_BIGINT_AS_STRING = Encodes large integers as their original string value.
// JSON_NUMERIC_CHECK = Encodes numeric strings as numbers.
// JSON_UNESCAPED_SLASHES = Don't escape "/".
syslog(LOG_INFO, '@cee: ' . json_encode($doc, JSON_BIGINT_AS_STRING | JSON_NUMERIC_CHECK | JSON_UNESCAPED_SLASHES));
这篇关于rsyslog性能优化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
