带有错误时间戳和时间键的日志被忽略 [英] Logs shipped with wrong timestamp and timekey ignored

问题描述

我想将我的保险柜日志发送到s3.基于此问题，我这样做了:

I want to ship my Vault logs to s3. Based on this issue I did this:

## vault input
<source>
  @type tail
  path /var/log/vault_audit.log
  pos_file /var/log/td-agent/vault.audit_log.pos
  <parse>
    @type json
  </parse>
  tag s3.vault.audit
</source>

## s3 output
<match s3.*.*>
  @type s3

  s3_bucket vault
  path logs/

  <buffer time>
    @type file
    path /var/log/td-agent/s3
    timekey 30m
    timekey_wait 5m
    chunk_limit_size 256m
  </buffer>

  time_slice_format %Y/%m/%d/%H%M
</match>

我希望每30分钟将日志发送到S3，并在目录中将其格式化为:logs/2019/05/01/1030

What I'd expect is for my logs to be shipped to S3 every 30 minutes, and be formatted in directories as ie: logs/2019/05/01/1030

实际上，我的日志平均每2-3ish分钟发送一次，并且S3中的输出时间格式是从纪元开始的，即:logs/1970/01/01/0030_0.gz

Instead my logs are shipped every 2-3ish minutes on average, and the output time format in S3 is starting from the epoch ie: logs/1970/01/01/0030_0.gz

(时间已在我的系统上正确设置)

(the time is correctly set on my system)

推荐答案

以下是适合我的示例配置.

Here is sample configuration which worked fine for me.

您需要确保将time传递到缓冲区部分，并尝试提供应明确显示的format类型.

You need to make sure, you pass time to buffer section and also try to provide what kind of format it should be explicitly.

通过检查代理启动日志来检查您的匹配表达式是否工作正常.另外，尝试使用<match s3.**>

Check whether your match expression is working fine by checking agent start up logs. Also, try with <match s3.**>

<match>
  @type s3

  s3_bucket somebucket
  s3_region "us-east-1"
  path "logs/%Y/%m/%d/%H"
  s3_object_key_format "%{path}/%{time_slice}_%{index}.%{file_extension}"
  include_time_key true
  time_format "%Y-%m-%dT%H:%M:%S.%L"

  <buffer tag,time>
    @type file
    path /fluentd/buffer/s3
    timekey_wait 5m
    timekey 30m
    chunk_limit_size 64m
    flush_at_shutdown true
    total_limit_size 256m
    overflow_action block
  </buffer>
  <format>
    @type json
  </format>
  time_slice_format %Y%m%d%H%M%S
</match>

这篇关于带有错误时间戳和时间键的日志被忽略的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！