Logstash-CSV输出标头 [英] Logstash - csv output headers

问题描述

我正在尝试使用 logstash jdbc插件，并返回带有 logstash csv插件。

I'm trying to request database with logstash jdbc plugins and returns a csv output file with headers with logstash csv plugin.

我在Logstash文档上花了很多时间，但是我仍然缺少一点。

I spent a lot of time on logstash documentation but I'm still missing a point.

使用以下logstash配置，结果将为我提供一个文件，其中每行都有标题。我找不到在logstash配置中仅添加第一行标题的方法。

With the following logstash configuration, the results give me a file with headers for each row. I couldn't find a way to add the headers for only the first row in the logstash configuration.

非常感谢帮助。

_object$id;_object$name;_object$type;nb_surveys;csat_score
2;Jeff Karas;Agent;2;2  
_object$id;_object$name;_object$type;nb_surveys;csat_score
3;John Lafer;Agent;2;2;2;2;$2;2
_object$id;_object$name;_object$type;nb_surveys;csat_score
4;Michele Fisher;Agent;2;2
_object$id;_object$name;_object$type;nb_surveys;csat_score
5;Chad Hendren;Agent;2;78

文件：简单-out.conf

file: simple-out.conf

input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/postgres"
        jdbc_user => "postgres"
        jdbc_password => "postgres"
        jdbc_driver_library => "/tmp/drivers/postgresql/postgresql_jdbc.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        statement_filepath => "query.sql"
    }
}
output {
    csv {
        fields => ["_object$id","_object$name","_object$type","nb_surveys","csat_score"]
        path => "output/%{team}/output-%{team}.%{+yyyy.MM.dd}.csv"
        csv_options => {
        "write_headers" => true
        "headers" =>["_object$id","_object$name","_object$type","nb_surveys","csat_score"]
        "col_sep" => ";"
        }
    }
}

谢谢

推荐答案

在输出中获得多个标头的原因是因为Logstash在事件之间没有全局/共享状态的概念，因此每个项目都在隔离，因此每次CSV输出插件运行时，其行为都与第一个类似，并写入标题。

The reason why you are getting multiple headers in the output is because Logstash has no concept of global/shared state between events, each item is handled in isolation so every time the CSV output plugin runs it behaves like the first one and writes the headers.

我遇到了同样的问题，并使用 init 选项过滤器以在logstash启动时执行一些代码。

I had the same issue and found a solution using the init option of the ruby filter to execute some code at logstash startup-time.

以下是logstash配置示例：

Here is an example logstash config:

# csv-headers.conf

input {
    stdin {}
}
filter {
    ruby {
        init => "
            begin
                @@csv_file    = 'output.csv'
                @@csv_headers = ['A','B','C']
                if File.zero?(@@csv_file) || !File.exist?(@@csv_file)
                    CSV.open(@@csv_file, 'w') do |csv|
                        csv << @@csv_headers
                    end
                end
            end
        "
        code => "
            begin
                event['@metadata']['csv_file']    = @@csv_file
                event['@metadata']['csv_headers'] = @@csv_headers
            end
        "
    }
    csv {
        columns => ["a", "b", "c"]
    }
}
output {
    csv {
        fields => ["a", "b", "c"]
        path   => "%{[@metadata][csv_file]}"
    }
    stdout {
        codec => rubydebug {
            metadata => true
        }
    }
}

如果运行Logstash该配置：

If you run Logstash with that config:

echo "1,2,3\n4,5,6\n7,8,9" | ./bin/logstash -f csv-headers.conf

您将获得 output.csv 文件，内容如下：

You will get an output.csv file with this content:

A,B,C
1,2,3
4,5,6
7,8,9

这也是线程安全的，因为它仅在启动时运行代码，因此您可以使用多个工作程序。

This is also thread-safe because it runs the code on startup only, so you can use multiple workers.

希望它会有所帮助！

这篇关于Logstash-CSV输出标头的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！