目录轮询的最佳实践 [英] best practice for directory polling
问题描述
我必须进行批处理以自动化业务流程。我必须定期轮询目录以检测新文件并进行处理。在处理旧文件时，可以使用新文件。目前，我使用quartz调度程序和线程同步来确保只有一个线程可以处理文件。
I have to do batch processing to automate business process. I have to poll directory at regular interval to detect new files and do processing. While old files is being processed, new files can come in. For now, I use quartz scheduler and thread synchronization to ensure that only one thread can process files.
部分代码是：
Part of the code are:
application-context.xml
application-context.xml
<bean id="methodInvokingJob"
class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean"><br/>
<property name="targetObject" ref="documentProcessor" /><br/>
<property name="targetMethod" value="processDocuments" /><br/>
</bean>
DocumentProcessor
.....
DocumentProcessor
.....
public void processDocuments() {
LOG.info(Thread.currentThread().getName() + " attempt to run.");
if (!processing) {
synchronized (this) {
try {
processing = true;
LOG.info(Thread.currentThread().getName() + " is processing");
List<String> xmlDocuments = documentManager.getFileNamesFromFolder(incomingFolderPath);
// loop over the files and processed unlock files.
for (String xmlDocument : xmlDocuments) {
processDocument(xmlDocument);
}
}
finally {
processing = false;
}
}
}
}
当前代码，我必须阻止其他线程在处理一个线程时处理文件。这是一个好主意吗 ？或者我们支持多线程处理。在这种情况下，我如何知道哪些文件正在处理以及哪些文件刚刚到达？任何想法都非常感激。
For the current code, I have to prevent other thread to process files when one thread is processing. Is that a good idea ? or we support multi-threaded processing. In that case how can I know which files is being process and which files has just arrived ? Any idea is really appreciated.
推荐答案
我会用这些部分构建它：
I would build it with these parts:
- 城堡与TxF的交易
- FileSystemWatcher < a href =http://jnotify.sourceforge.net/ =nofollow noreferrer> JavaVersion
- TransactionScope（除非你经常破解，否则没有java版本）
-
一个无锁队列 *（论文讨论perf Java vs .Net，也许能够得到源来自他们的Java ）基于Java锁定的队列
- Castle Transactions with TxF
- FileSystemWatcher JavaVersion
- TransactionScope (no java version unless you hack it a lot)
A lock-free queue * (Paper discussing perf Java vs .Net, might be able to get source from them for Java) Java lock-based queues
这样：
当有新文件时，文件系统观察程序检测到它（记得放置正确的标志，处理错误条件并设置Enbled< - True并注意双打），将文件路径放入队列。
When there's a new file, the file system watcher detects it (remember to put the correct flags, handle the error condition and set Enbled <- True and watch out for doubles), puts the file path in the queue.
你有一个应用程序线程，n个工作线程。如果这是唯一的应用程序，他们会在队列上旋转等待TryDequeue，否则它们会在监视器上阻塞（！Monitor.Enter（has_items））;
You have an application thread, n worker threads. If this is the only app, they spin-wait on the queue, TryDequeue, otherwise they block on a monitor while(!Monitor.Enter(has_items)) ;
一个工作线程通过de-queue操作得到一个路径，它开始处理它，现在没有其他线程可以对它起作用。如果有两倍的输出（取决于您的设置），则可以在编写输出文件时使用文件事务。如果Commit操作失败，那么您知道另一个线程已经写入了输出文件，并继续轮询队列。
When a worker threads get a path through the de-queue operation, it starts working on it, and now no other thread can work on it. If there are doubles of output (depending on your setup), you can then use a file transaction as you are writing the output file. If the Commit operation fails, then you know another thread has already written the output file, and resume polling the queue.
- Race condition, see: http://groups.google.com/group/lock-free/browse_thread/thread/c3b83466b27f6372
这篇关于目录轮询的最佳实践的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
