使用Logstash在ElasticSearch中加载CSV [英] Loading csv in ElasticSearch using logstash

问题描述

我有一个csv，其中一列可能包含多行值.

I have a csv in which one column may contain multi-line values.

ID,Name,Address
1, ABC, "Line 1
Line 2
Line 3"

(据我所知)以上根据CSV标准写入的数据是一条记录.

The data written above as per CSV standard is one record (to my knowledge).

我有以下用于logstash的过滤器

I have following filter for logstash

filter {
  csv {
      separator => ","
      quote_char => "\""
     columns => ["ID","Name", "Address"]
  }
}
output {
   elasticsearch {
     host => "localhost"
     port => "9200"
     index => "TestData"
     protocol => "http"
  }
stdout {}
}

但是当我执行它时，它会创建三个记录. (原则上都是错误的，因为前一个包含两个列数据ID和Name以及Address的部分数据，接下来的两个记录包含第2行和第3行，但没有ID和Name

But when I execute it, it creates three records. (All are wrong in principle as first one contains two column data ID and Name and partial data for Address and next two records contain Line 2 and Line 3 but no ID and Name

我该如何解决?我在文件解析中丢失了什么吗?

How can I fix this? Am I missing something in the file parsing?

推荐答案

您是否尝试过您应该在输入插件中添加以下内容:

You should add something like this in your input plugin:

codec => multiline {
      pattern => "^[0-9]"
      negate => "true"
      what => "previous"
    }

它告诉logstash，每条不以数字开头的行都应与前一行合并

it tells logstash that every line not starting with a number should be merged with the previous line

这篇关于使用Logstash在ElasticSearch中加载CSV的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！