logstash http_poller第一个URL请求的响应应该输入到第二个URL的请求参数中 [英] logstash http_poller first URL request's response should be input to second URL's request param

问题描述

我有两个URL(出于安全考虑，我将使用虚拟对象进行解释)

I have two URLs (due to security concern i will explain by using dummy)

 a> https://xyz.company.com/ui/api/token
 b> https://xyz.company.com/request/transaction?date=2016-01-21&token=<tokeninfo>

当您点击"a"点提到的网址时，它将生成一个令牌，使其为16个字符的字符串

When you hit url mentioned in point 'a' it will generate a token let it be a string of 16 characters

然后在令牌参数中对令牌点'b'的第二次请求中应使用令牌

Then that token should be used in making second request of point 'b' in token param

 The second url response is important to me i.e is a JSON response, I need       
 to filter the json data and extract required data and output it to standard 
 output and elastic search.    

在logstash中可以使用插件"http_poller"或任何其他插件来执行此操作吗.

is there any way of doing so in logstash using plugin "http_poller" or any other plugins.

注意:这些请求网址应一个接一个地执行，即，在接收到新令牌之后，应首先执行"a"网址，然后应执行"b"网址.

Note : these request urls should be executed one after another, i.e point 'a' url should be executed first and point 'b' url should be executed next after receiving new token.

请提出建议.

推荐答案

是的，可以混合使用http_poller输入和http输出.

Yes, it's possible with a mix of an http_poller input and an http output.

这是我想到的配置:

input {
   # 1. trigger new token requests every hour
   http_poller {
     urls => {
       token => "https://xyz.company.com/ui/api/token"
     }
     interval => 3600
     add_field => {"token" => "%{message}"}
   }
}
filter {
}
output {
   # 2. call the API
   http {
     http_method => "get"
     url => "https://xyz.company.com/request/transaction?date=2016-01-21&token=%{token}"
   }
}

更新

如果您希望能够获取API调用的内容并将其存储在ES中，则需要一个混合解决方案.您需要设置一个cron来调用一个脚本，该脚本运行两个HTTP调用并将结果存储在文件中，然后可以让logstash尾随该文件并将结果转发给ES.

If you want to be able to get the content of the API call and store it in ES, you need a hybrid solution. You need to set up a cron that will call a script that runs the two HTTP calls and stores the results in a file and then you can let logstash tail that file and forward the results to ES.

要放在cron上的Shell脚本:

Shell script to put on cron:

#!/bin/sh

# 1. Get the token
TOKEN=$(curl -s -XGET https://xyz.company.com/ui/api/token)

# 2. Call the API with the token and append JSON to file
curl -s -XGET "https://xyz.company.com/request/transaction?date=2016-01-21&token=$TOKEN" >> api_calls.log

可以使用crontab(或类似工具)在cron上设置上述脚本，其中

The above script can be set on cron using crontab (or similar), there are plenty of examples out there on how to achieve this.

然后，logstash配置可以非常简单.它只需要拖尾api_calls.log文件并将文档发送到ES

Then the logstash config can be very simple. It just needs to tail the api_calls.log file and send the document to ES

input {
    file {
        path => "api_calls.log"
        start_position => "beginning"
    }
}
filter {
    json {
        source => "message"
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "my_index"
        document_type" => "my_type"
    }
    stdout {
        codec => "rubydebug"
    }
}

这篇关于logstash http_poller第一个URL请求的响应应该输入到第二个URL的请求参数中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！