使用多个csv生成多种类型 [英] Generate multiple types with multiple csv

问题描述

我试图基于各种csv在同一索引中生成各种类型.由于我不知道csv的数量，因此无法为每个输入csv.那么，有谁知道如何使用文件名生成类型，并在其中分别引入csv?

I am trying to generate various types in the same index based on various csv. As I don´t know the amount of csv, making an input for each one would be non-viable. So does anyone know how to generate types with the names of the files and in those, introduce the csv respectively?

input {
  file {
    path => "/home/user/Documents/data/*.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  } 
}


filter { 
    
    csv {
        separator => ","
        skip_header => "true"
        autodetect_column_names => true
        autogenerate_column_names => true
    }
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "final_index"

  }

stdout {}

}

非常感谢您

推荐答案

自版本6开始，在Elasticsearch索引中已删除了同一索引中的多个文档结构，如果文档的外观与索引的模板化方式不同将无法向其发送数据，您可以做的是确保所有字段都已知，并且您有一个包含所有可能字段的通用模板.

Having multiple document structures in the same index has been removed in Elasticsearch indices since version 6, if a document is not looking the same way as the index is templated it will not be able to send the data to it, what you can do is make sure that all fields are known and you have one general template containing all possible fields.

您是否有理由将所有索引合并在一个索引中?

Is there a reason why you want all of it in one index?

如果它是用于查询或Kibana的，请知道您可以在搜索时使用通配符，并且具有Kibana的模式.

If it is for querying purposes or Kibana, do know you can wildcard when searching and have patterns for Kibana.

发表评论后更新:

使用过滤器使用grok提取文件名

Use a filter to extract the filename using grok

filter {
  grok {
    match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.csv"]
  }
}

并在输出中使用文件名，如下所示:

And use the filename in your output like this:

elasticsearch {
   hosts => "http://localhost:9200"
   index => "final_index-%{[filename]}"
}

这篇关于使用多个csv生成多种类型的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！