如何在 Java 中多线程处理 ANTLR 解析器 [英] How to multi-thread an ANTLR parser in java
问题描述
我的程序在读取文件然后用 antlr 语法解析它时被证明很慢.为了提高性能，我想多线程解析?
I have my program that is proving slow in reading a file and then parsing it with antlr grammar. To improve performance of this I would like to multi-thread the parsing?
读取文件:
LogParser pa = new LogParser();
LogData logrow;
String inputLine;
int a=0;
try {
//feed line by line
FileReader fr = new FileReader(jFileChooser1.getSelectedFile());
BufferedReader reader = new BufferedReader(fr);
while ((inputLine = reader.readLine()) != null)
{
try {
a++;
jProgressBar.setValue(a);
pa.parse(inputLine); //decode the line
} catch ... catches errors and send to logger
} finally {
logrow=new LogData(pa,a);
mLogTable.addRow(logrow);//store the decoded line
}
}
reader.close();
} catch ... catches errors and send to logger
代码使用 pa.parse(inputLine); 解析该行，将输入行发送到 ANTLRStringStream 然后是 CharStream然后被解析.接下来 logrow=new LogData(pa,a); 获取将存储在我的表中的解码值.
The code parses the line with pa.parse(inputLine); which sends the input line in to an ANTLRStringStream and then a CharStream and is then parsed.
Next logrow=new LogData(pa,a); gets the decoded values which will be storred in my table.
我的分析显示了词法分析器和解析器类中的热点，这些类是根据我的语法(即 LogGrammarLexer 和 LogGrammarParser)构建的.希望它足够清楚...
My profiling shows the hotspots being in the lexer and parser classes which are built from my grammar (i.e. LogGrammarLexer and LogGrammarParser). Hope it's clear enough...
解析:
LogGrammarLexer lexer = new LogGrammarLexer(inStream);
CommonTokenStream tokens = new CommonTokenStream(lexer);
decoded = new LogGrammarParser(tokens);
try {
failurePosition="";
decoded.logLine();
} catch (RecognitionException e) {
failurePosition=Integer.toString(e.charPositionInLine);
} catch (Exception e) {
failurePosition="-3";
throw e;
} finally {
return decoded;//TODO: see if return is necessary as decoded is now a field in class
}
我也一直在阅读，现在知道多线程文件 I/O 是无用的...无论如何，这是我的语法在构建类中的表现，我需要改进我需要改进的文件中每一行的复杂解码/解析.
Also I have been reading up and now know that multi-threading file I/O is useless... Anyway it is the performances inside the built classes for my grammar doing complex decoding/parsing of each line in my file what I need to improve.
所以我的问题是如何使它成为多线程的，
So my problem is how to make it multi-threaded,
干杯
推荐答案
如果每个文件行都是一个单独的可解析实体，那么您可以有多个线程，如下所示:
If each file line is a separate parse-able entity, then you can have multiple threads as follows:
- 一个线程(很可能是主线程)从文件中读取行并将结果存储在工作队列中
- 一个线程池，每个线程都会从前面提到的工作队列中提取一个文件行并解析数据
- 线程之间共享的文件行队列必须同步
如果在多核 CPU 上运行，这只会提高性能.
This will only see an increase in performance if run on a multi-core CPU.
此外，这仅在每个文件行是一个单独的可解析实体时才有效，如前所述.如果可解析实体的概念跨越多行，或者是整个文件，那么线程将无济于事.此外，如果文件中的行顺序很重要，那么多线程可能会导致问题，因为这些行可能会被乱序解析.
Additionally, this will only work if each file line is a separate parse-able entity as mentioned before. If the concept of an parse-able entity spans multiple lines, or is the entire file then threading will not help. Additionally, if the order of the lines in the file is important, then multi threading may cause issues, since the lines may be parsed out of order.
这是一个标准的生产者/消费者问题，这里有一些有用的链接:
This is a standard producer/consumer problem, here are some useful links:
这篇关于如何在 Java 中多线程处理 ANTLR 解析器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
