Logstash HTTP输出无法发布到需要客户端证书的HTTPS端点 [英] Logstash HTTP output can't post to to HTTPS endpoint requiring client certificates

问题描述

我当前正在尝试通过http输出将我们生产环境中的Logstash接收服务器的一些示例事件发送到测试环境.
接收端的服务器是一个自定义的Nginx HTTPS端点，该端点接受发布数据(单个事件的端点和批量事件的端点，以支持Elasticsearch批量索引格式)并将其放入Redis队列，最终由Logstash处理服务器读取.

I'm currently attempting to send some sample events from Logstash receiving servers on our production environment to a testing env via the http output.
The server on the receiving end is a custom Nginx HTTPS endpoint which accepts post data (endpoints for both single events, and bulk events to support Elasticsearch bulk indexing format) and places it into a redis queue, which is eventually read by Logstash processing servers.

logstash接收服务器上的当前http输出看起来像这样:

The current http output on the logstash receiving server looks something like this:

 http {
   url => "https://json-logs-endpoint.example.com:8443/event"
   http_method => "post"
   format => "json"
   ssl_certificate_validation => false
   client_cert => "/etc/filebeat/ssl/filebeat.crt"
   client_key => "/etc/filebeat/ssl/filebeat.key"
   cacert => "/etc/filebeat/ssl/filebeat.chain"
 }

与cert相关的选项通过elasticsearch输出使用与filebeat日志传送程序(安装在所有服务器上)相同的文件. /etc/filebeat/ssl/中文件的权限确实允许logstash读取证书.我的filebeat配置看起来像这样:

The cert related options are using the same files as the filebeat log shipper (installed on all servers), via elasticsearch output. The permissions on the files in /etc/filebeat/ssl/ does allow logstash to read the certs. My filebeat config ressembles something like this:

filebeat:
  prospectors: []
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat/conf.d"
output:
  elasticsearch:
    enabled: true
    hosts:
    - json-logs-endpoint.example.com:8443
    protocol: https
    path: "/multi-event"
    tls:
      certificate_authorities:
      - "/etc/filebeat/ssl/filebeat.chain"
      certificate: "/etc/filebeat/ssl/filebeat.crt"
      certificate_key: "/etc/filebeat/ssl/filebeat.key"

所有服务器上的所有filebeat实例均已通过/multi-event端点通过https成功将数据发送到此Nginx服务器.

All filebeat instances on all servers are successfully sending data via https to this Nginx server, via the /multi-event endpoint.

/event和/multi-event端点之间的唯一区别是，前者接受单个JSON事件，而后者接受此堆栈跟踪，以详细模式运行logstash会提供更多信息.

The only difference between the /event and /multi-event endpoints are that the former accepts a single JSON event and the latter accepts data in the format of an elasticsearch bulk indexing request.
Both endpoints are secured via SSL in the same fashion. When Logstash is started, there's no indication of any error relating to the http output, although after a very short period of time pipeline essentially halts. Running logstash in verbose mode then provides a bit more information, with this stack trace.

我猜认为问题与所提供证书的格式有关?如果有人可以指出我的问题，我将不胜感激.

I'm guessing the problem is related to the format of the provided certs? I would really appreciate if someone could point out my issue.

谢谢！

推荐答案

出现问题的原因是在http输出中指定了客户端密钥文件:

It appears the problem was due to the fact that the client key file specified in the http output:

client_key => "/etc/filebeat/ssl/filebeat.key"

应为 PKCS8 格式.一旦我运行以下命令，将其转换为正确的格式:

should have been in a PKCS8 format. Once I ran the following command to convert it to the proper format:

openssl pkcs8 -topk8 -nocrypt -in '/etc/filebeat/ssl/filebeat.key' -out '/etc/filebeat/ssl/pkcs8-filebeat.key'

，然后将client_key参数更新为指向PKCS8格式的密钥:

and then updated the client_key parameter to point to the PKCS8 formatted key:

client_key => "/etc/filebeat/ssl/pkcs8-filebeat.key"

这解决了该问题，并且HTTPS POST请求无法正常运行.希望这对将来对其他人有用.

This solved the issue and the HTTPS POST requests are no functioning as expected. Hopefully this ends up being useful to someone else in the future.

这篇关于Logstash HTTP输出无法发布到需要客户端证书的HTTPS端点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！