多线程消费时骆驼丢失文件 [英] Losing files in camel when consuming with multiple threads

问题描述

我使用Apache的骆驼2.11.1

I'm using Apache Camel 2.11.1

有这样的路线：

from("file:///somewhere/").
 threads(20).
 to("direct:process")

有的时候我得到这个异​​常： org.apache.camel.InvalidPayloadException 与消息

No body available of type: java.io.InputStream but has value: GenericFile[/somewhere/file.txt] of type: 
org.apache.camel.component.file.GenericFile on: file.txt. Caused by: Error during type conversion from type: 
org.apache.camel.component.file.GenericFile to the required type: byte[] with value GenericFile[/somewhere/file.txt] 
due java.io.FileNotFoundException: /somewhere/file.txt (No such file or directory). 

因为我看到很多.camelLock目录中的文件，我认为这种情况是由于几个线程尝试处理同​​一个文件。如何避免呢？

Since I'm seeing lot of .camelLock files in the directory, I assume this happens due to attempt of few threads to process same file. How to avoid that?

更新1

试过使用 ScheduledExecutorService的并删除螺纹（20）。似乎我失去了较少的文件，但仍然失去他们。如何避免？任何帮助将大大AP preciated。

Tried to use scheduledExecutorService and remove threads(20). Seems I'm losing fewer files, but still losing them. How to avoid? Any help will be greatly appreciated.

推荐答案

我得到了类似的问题，我的是2档处理器从同一目录中检索。结果：失去我的所有文件

I got the similar issue, mine was 2 file processors retrieving from same directory. Result: Losing all my files.

下面是这种情况：

• 线程＃1 检索文件1 ，移动到过程文件夹


• 线程＃2 检索同一个文件：文件1 在同一时间。 文件1 删除


• 线程＃2 找不到文件1 在源目录下，重命名失败。


• 线程＃1 失败归因于线程＃2
删除文件

• Thread#1 retrieves file1, moves to process folder
• Thread#2 retrieves same file: file1 at the same time. file1 is deleted
• Thread#2 cannot find file1 in source directory, rename fails.
• Thread#1 fails due to deleted file by Thread#2

这里的原因是：

如果您检查GenericFileProcessStrategySupport.renameFile方法，你会看到骆驼首先删除目标文件，然后重命名源文件的目标。这就是为什么上述情况发生时

If you check GenericFileProcessStrategySupport.renameFile method, you'll see camel first deletes target file, then renames source file to target. That's why above condition occurs

我不知道一个通​​用的解决方案，无论是分离源消费关系或实施的工作机制，分销商应实施。

I dont know about a generic solution, either separating source-consumer relation or implement a work distributor mechanism should be implemented.

由于您的线程住在同一个JVM中，我建议你实现一个并发负载分销商。这会给请求者1文件名在并发方式时间

Sınce your threads live in same JVM, I suggest you to implement a concurrent load distributor. That'd give requester 1 file name at a time in concurrent way

这篇关于多线程消费时骆驼丢失文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！