出于语法突出显示的目的，是否建议使用标记? [英] Is it advisable to use tokens for the purpose of syntax highlighting?
问题描述
我正在尝试使用 Xamarin 在 Android 上的 C# 中实现语法突出显示.我正在使用 C# 的 ANTLR v4 库来实现这一点.我的代码，目前使用 这个语法高亮语法a>，不会尝试构建解析树并使用访问者模式.相反，我只是将输入转换为令牌列表:
I'm trying to implement syntax highlighting in C# on Android, using Xamarin. I'm using the ANTLR v4 library for C# to achieve this. My code, which is currently syntax highlighting Java with this grammar, does not attempt to build a parse tree and use the visitor pattern. Instead, I simply convert the input into a list of tokens:
private static IList<IToken> Tokenize(string text)
{
var inputStream = new AntlrInputStream(text);
var lexer = new JavaLexer(inputStream);
var tokenStream = new CommonTokenStream(lexer);
tokenStream.Fill();
return tokenStream.GetTokens();
}
然后我遍历荧光笔中的所有标记，并根据它们的种类为它们分配颜色.
Then I loop through all of the tokens in the highlighter and assign a color to them based on their kind.
public void HighlightAll(IList<IToken> tokens)
{
int tokenCount = tokens.Count;
for (int i = 0; i < tokenCount; i++)
{
var token = tokens[i];
var kind = GetSyntaxKind(token);
HighlightNext(token, kind);
if (kind == SyntaxKind.Annotation)
{
var nextToken = tokens[++i];
Debug.Assert(token.Text == "@" && nextToken.Type == Identifier);
HighlightNext(nextToken, SyntaxKind.Annotation);
}
}
}
public void HighlightNext(IToken token, SyntaxKind tokenKind)
{
int count = token.Text.Length;
if (token.Type != -1)
{
_text.SetSpan(_styler.GetSpan(tokenKind), _index, _index + count, SpanTypes.InclusiveExclusive);
_index += count;
}
}
最初，我认为这是明智的，因为语法突出显示在很大程度上与上下文无关.但是，我已经发现自己需要在 @ 前面使用特殊情况标识符，因为我希望它们像在 GitHub 上一样作为注释突出显示 (示例).GitHub 有在某些上下文中着色标识符的更多示例:此处, List 和 ArrayList 是彩色的，而 mItems 不是.我可能需要添加更多代码来突出显示这些场景中的标识符.
Initially, I figured this was wise because syntax highlighting is largely context-independent. However, I have already found myself needing to special-case identifiers in front of @, since I want those to get highlighted as annotations just as on GitHub (example). GitHub has further examples of coloring identifiers in certain contexts: here, List and ArrayList are colored, while mItems is not. I will likely have to add further code to highlight identifiers in those scenarios.
我的问题是，在这里检查令牌而不是分析树是个好主意吗?一方面，我担心当令牌的邻居改变它应该如何突出显示时，我可能不得不做很多特殊的外壳.另一方面，解析将为内存受限的移动设备增加额外的开销，并使用户在代码编辑器中编辑文本时实现高效的语法突出显示(例如，不重新标记/解析所有内容)变得更加复杂.我还发现处理所有标记类型而不是解析器规则类型要简单得多，因为您只需在 token.Type 上switch 而不是覆盖一堆 <代码>访问*方法.
My question is, is it a good idea to examine tokens rather than a parse tree here? On one hand, I'm worried that I might have to end up doing a lot of special-casing for when a token's neighbors alter how it should be highlighted. On the other, parsing will add additional overhead for memory-constrained mobile devices, and make it more complicated to implement efficient syntax highlighting (e.g. not re-tokenizing/parsing everything) when the user edits text in the code editor. I also found it significantly less complicated to handle all of the token types rather than the parser rule types, because you just switch on token.Type rather than overriding a bunch of Visit* methods.
作为参考，语法高亮器的完整代码可在此处获得.
For reference, the full code of the syntax highlighter is available here.
推荐答案
我最终选择使用解析器，因为临时规则太多.例如，虽然我想将常规标识符着色为白色，但我希望类型声明中的类型(例如 class C 中的 C)为绿色.最终总共有大约 20 个这些特殊规则.此外，与我的应用中的其他瓶颈相比，解析的额外开销被证明是微不足道的.
I ended up choosing to use a parser because there were too many ad hoc rules. For example, although I wanted to color regular identifiers white, I wanted types in type declarations (e.g. C in class C) to be green. There ended up being about 20 of these special rules in total. Also, the added overhead of parsing turned out to be miniscule compared to other bottlenecks in my app.
For those interested, you can view my code here: https://github.com/jamesqo/Repository/blob/e5d5653093861bc35f4c0ac71ad6e27265e656f3/Repository.EditorServices/Internal/Java/Highlighting/JavaSyntaxHighlighter.VisitMethods.cs#L19-L76. I've highlighted all of the ~20 special rules I've had to make.
这篇关于出于语法突出显示的目的，是否建议使用标记?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
