Logstash的流式传输行为不一致-ELK [英] Inconsistency streaming behavior with Logstash - ELK

问题描述

我有一个包含几个平面字段和几个嵌套字段的索引.我正在尝试通过Logstash将SQL Server中的信息通过特定的ID通过Logstash流到嵌套字段中.当我仅流传输一个 Id ID的数据时，它就可以完全成功地通过，而不会出现任何问题.另一方面，当我尝试流式传输多个ID的数据时，由于某种原因，插入到索引中的信息是不完整的.注意:查询按ID排序.而且，在流式传输数据的不同尝试中，获得了不同数量的信息.例如，假设完整信息包含15行.一次尝试-仅获得2行，而另一次尝试-获得14行，看似完全任意.有谁知道会导致这种奇怪行为的原因吗?我很乐意提供任何帮助.谢谢！

I have an index with several flat fields and several nested fields. I am trying to stream info from SQL Server through the Logstash into a nested field by a specific Id. When I stream the data for only one Id then it passes fully and successfully without any problem. On the other hand, when I try to stream the data for more than one id - the info that is inserted to the index is partial for some reason. Note: The query is sorted by id. Moreover, in different tries streaming the data, a different amount of information is obtained. For example, suppose the full info contains 15 rows. In one try - only 2 rows is obtained, but in another try - 14 rows is obtained, seemingly completely arbitrarily. Does anyone have any idea what can cause this strange behavior? I would be happy for any help. Thanks!

推荐答案

这是因为

This is because of the Logstash execution model where several workers can work in parallel and your events might be processed by different worker threads.

如果要保持一致的加载行为，则需要使用单个工作程序(命令行上的 -w 1 )执行管道

If you want to have a consistent loading behavior you need to execute your pipeline with a single worker (-w 1 on the command line)

这篇关于Logstash的流式传输行为不一致-ELK的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！