尝试在ANTLR4中使用关键字作为标识符；不工作 [英] Trying to use keywords as identifiers in ANTLR4; not working

问题描述

我正在尝试将一些sql关键字用作标识符时被接受为标识符. Antlr书p210提出了这个技巧:

I'm trying to get some sql keywords to be accepted as identifiers, when used as identifiers. The Antlr book p210 suggests this trick:

id : 'if' | 'call' | 'then' | ID ;

我有类似的东西，但是它不起作用，我认为这是我的误解. regular_ident是标识符的解析规则，因此:

I've got something similar but it's not working and I assume it's a misunderstanding on my part. regular_ident is the parse rule for an identifier thus:

regular_ident :  // (1)
        KEYWORD_AS_IDENT
        |
        REGULAR_IDENT
    ;

REGULAR_IDENT是ident的主要lex规则.大致就是这个(在此简化)，并且可以正常工作:

REGULAR_IDENT is the main lex rule for idents. It's roughly this (simplified here), and it works:

REGULAR_IDENT :
        [a-zA-Z]  ( [a-zA-Z0-9] * )
    ;

KEYWORD_AS_IDENT是特殊词的列表，以下是摘录:

KEYWORD_AS_IDENT is the list of special words, here's an extract:

KEYWORD_AS_IDENT :  // (2)
[...snip...]
  | FILESTREAM
  | SPARSE
  | NO
  | ACTION
  | PERSISTED
  | FILETABLE_DIRECTORY
  | FILETABLE_COLLATE_FILENAME
  | FILETABLE_PRIMARY_KEY_CONSTRAINT_NAME
  | FILETABLE_STREAMID_UNIQUE_CONSTRAINT_NAME
  | FILETABLE_FULLPATH_UNIQUE_CONSTRAINT_NAME
  | COLUMN_SET
  | ALL_SPARSE_COLUMNS
 ;

其中组件在其他位置定义的

where components are defined elsewhere:

SPARSE : 'sparse' ;
NO     : 'no'
(etc)

如果我输入fetch aaa作为输入("aaa"不是关键字)，它将进行解析:

If I give it fetch aaa as input ('aaa' is not a keyword), it parses:

但是如果我给它fetch sparse，它将失败-'sparse'是关键字:

but if I give it fetch sparse it fails - 'sparse' is a keyword:

也许我很傻，但是我不明白为什么，因为SPARSE是KEYWORD_AS_IDENT的成员. 如果我切&将(2)中的一些粘贴到(1)中以得到此结果:

perhaps I'm being dumb but I can't see why, as SPARSE is a member of KEYWORD_AS_IDENT. If I cut & paste some of (2) into (1) to get this:

regular_ident :
    FILESTREAM
  | SPARSE
  | NO
  | ACTION
  | PERSISTED
  | FILETABLE_DIRECTORY
        |
    REGULAR_IDENT
    ;

fetch sparse突然好起来了，因为它现在将'稀疏'视为regular_ident:

it suddenly is ok with fetch sparse as it now treats 'sparse' as an regular_ident:

但是为什么(1)不起作用? 我可以通过内联所有KEYWORD_AS_IDENT来轻松修复它，但是我需要知道我所缺少的内容.

but why does (1) not work? I can fix it trivially by inlining all of KEYWORD_AS_IDENT but I need to know what I'm missing.

感谢所有建议.

推荐答案

来自Google组antlr-discussion的Eric Vergnaud的回复:

Reply from Eric Vergnaud from google group antlr-discussion:

LAST在KEYWORD_AS_IDENT之前声明，所以当词法分析器时 遇到"last"，它会生成一个LAST令牌，而不是 KEYWORD_AS_IDENT.您的开始规则不接受LAST令牌作为 有效输入，因此大喊大叫.您的语法实际上永远不会 产生一个KEYWORD_AS_IDENT令牌，因为另一个有效的令牌将 比赛之前.看来您正在尝试让词法分析器完成以下任务 解析器，即处理多种语义选择，但当时 令牌到达解析器，为时已晚...您是否尝试过 KEYWORD_AS_IDENT是解析器规则(小写)而不是词法分析器规则?

LAST is declared before KEYWORD_AS_IDENT so when the lexer encounters 'last', it generates a LAST token, not a KEYWORD_AS_IDENT. Your start rule does not accept LAST token as a valid input, hence the shouting. Your grammar will actually NEVER produce a KEYWORD_AS_IDENT token, because another valid token will match before. It seems you are trying to get the lexer do the job of the parser i.e. handle multiple semantic alternatives, but at the time the token reaches the parser it's too late... Have you tried making KEYWORD_AS_IDENT a parser rule (lowercase) rather than a lexer rule?

所以我对词法分析器的理解是错误的，他是对的，因为我试图让它完成解析器的工作.

So my understanding of the lexer was faulty, and he's correct that I was trying to get it to do the parser's job.

这篇关于尝试在ANTLR4中使用关键字作为标识符；不工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！