logstash jdbc上有多个输入 [英] multiple inputs on logstash jdbc

问题描述

我使用logstash jdbc来保持mysql和elasticsearch之间的事物同步。它的工作罚款一张桌子。但是现在我想为多个表做它。我需要在终端中打开多个

  logstash agent -f /Users/logstash/logstash-jdbc.conf 
  

每个都有一个选择查询，或者我们有一个更好的方法，所以我们可以更新多个表。 / p>

我的配置文件

  input {
 jdbc {
 jdbc_driver_library => /Users/logstash/mysql-connector-java-5.1.39-bin.jar
 jdbc_driver_class => com.mysql.jdbc.Driver
 jdbc_connection_string => jdbc：mysql：// localhost：3306 / database_name
 jdbc_user => root
 jdbc_password => password
 schedule => * * * * *
 statement => select * from table1
} 
} 
 output {
 elasticsearch {
 index => testdb
 document_type => table1
 document_id => ％{table_id}
 hosts => localhost：9200
} 
} 
  

解决方案>

你可以一个配置多个 jdbc 输入，然后参数化索引和 document_type 在 elasticsearch 中，取决于事件来自哪个表。

  input {
 jdbc {
 jdbc_driver_library => /Users/logstash/mysql-connector-java-5.1.39-bin.jar
 jdbc_driver_class => com.mysql.jdbc.Driver
 jdbc_connection_string => jdbc：mysql：// localhost：3306 / database_name
 jdbc_user => root
 jdbc_password => password
 schedule => * * * * *
 statement => select * from table1
 type => table1
} 
 jdbc {
 jdbc_driver_library => /Users/logstash/mysql-connector-java-5.1.39-bin.jar
 jdbc_driver_class => com.mysql.jdbc.Driver
 jdbc_connection_string => jdbc：mysql：// localhost：3306 / database_name
 jdbc_user => root
 jdbc_password => password
 schedule => * * * * *
 statement => select * from table2
 type => table2
} 
＃添加更多jdbc输入以满足您的需要
} 
输出{
 elasticsearch {
 index => testdb
 document_type => ％{type}＃<  - 使用每个输入的类型
 hosts => localhost：9200
} 
} 
  

I am using logstash jdbc to keep the things syncd between mysql and elasticsearch. Its working fine for one table. But now I want to do it for multiple tables. Do I need to open multiple in terminal

logstash  agent -f /Users/logstash/logstash-jdbc.conf 

each with a select query or do we have a better way of doing it so we can have multiple tables being updated.

my config file

input {
  jdbc {
    jdbc_driver_library => "/Users/logstash/mysql-connector-java-5.1.39-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/database_name"
    jdbc_user => "root"
    jdbc_password => "password"
    schedule => "* * * * *"
    statement => "select * from table1"
  }
}
output {
    elasticsearch {
        index => "testdb"
        document_type => "table1"
        document_id => "%{table_id}"
        hosts => "localhost:9200"
    }
}

解决方案

You can definitely have a single config with multiple jdbc input and then parametrize the index and document_type in your elasticsearch output depending on which table the event is coming from.

input {
  jdbc {
    jdbc_driver_library => "/Users/logstash/mysql-connector-java-5.1.39-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/database_name"
    jdbc_user => "root"
    jdbc_password => "password"
    schedule => "* * * * *"
    statement => "select * from table1"
    type => "table1"
  }
  jdbc {
    jdbc_driver_library => "/Users/logstash/mysql-connector-java-5.1.39-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/database_name"
    jdbc_user => "root"
    jdbc_password => "password"
    schedule => "* * * * *"
    statement => "select * from table2"
    type => "table2"
  }
  # add more jdbc inputs to suit your needs 
}
output {
    elasticsearch {
        index => "testdb"
        document_type => "%{type}"   # <- use the type from each input
        hosts => "localhost:9200"
    }
}

这篇关于logstash jdbc上有多个输入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！