令牌和词位有什么区别? [英] What is the difference between a token and a lexeme?

问题描述

在 Aho Ullman 和 Sethi 的 Compiler Construction 中，给定源程序的输入字符串被划分为具有逻辑意义的字符序列，称为标记，词位是组成令牌那么基本的区别是什么?

In Compiler Construction by Aho Ullman and Set it is given that the input string of characters of the source program are divided into sequence of characters that have a logical meaning, and are known as tokens and lexemes are sequences that make up the token so what is the basic difference?

推荐答案

使用编译器原理、技术和工具，第 2 版" (WorldCat)，作者 Aho、Lam、Sethi 和 Ullman，又名 紫龙书,

Using "Compilers Principles, Techniques, & Tools, 2nd Ed." (WorldCat) by Aho, Lam, Sethi and Ullman, AKA the Purple Dragon Book,

词素 pg.111

词位是源程序中的一个字符序列匹配标记的模式并由词法标识分析器作为该令牌的一个实例.

A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token.

代币 pg.111

标记是由标记名称和可选属性组成的对价值.令牌名称是一个抽象符号，代表一种词汇单元，例如，特定的关键字或输入序列表示标识符的字符.令牌名称是输入解析器处理的符号.

A token is a pair consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a kind of lexical unit, e.g., a particular keyword, or sequence of input characters denoting an identifier. The token names are the input symbols that the parser processes.

模式 pg.111

模式是对标记的词位可能的形式的描述拿.在关键字作为标记的情况下，模式只是构成关键字的字符序列.对于标识符和一些其他标记，模式是更复杂的结构，匹配很多字符串.

A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens, the pattern is more complex structure that is matched by many strings.

图 3.2:标记示例 pg.112

Figure 3.2: Examplesof tokens pg.112

[Token]       [Informal Description]                  [Sample Lexemes]
if            characters i, f                         if
else          characters e, l, s, e                   else
comparison    < or > or <= or >= or == or !=          <=, !=
id            letter followed by letters and digits   pi, score, D2
number        any numeric constant                    3.14159, 0, 6.02e23
literal       anything but ", surrounded by "'s       "core dumped"

为了更好地理解这种与词法分析器和解析器的关系，我们将从解析器开始，然后回溯到输入.

To better understand this relation to a lexer and parser we will start with the parser and work backwards to the input.

为了更容易设计解析器，解析器不直接处理输入，而是接收由词法分析器生成的标记列表.查看图 3.2 中的标记列，我们看到诸如 if、else、comparison、id、<代码>数字和<代码>文字；这些是令牌的名称.通常，对于词法分析器/解析器，标记是一种结构，它不仅包含标记的名称，还包含组成标记的字符/符号以及组成标记的字符串的开始和结束位置，其中用于错误报告、突出显示等的开始和结束位置.

To make it easier to design a parser, a parser does not work with the input directly but takes in a list of tokens generated by a lexer. Looking at the token column in Figure 3.2 we see tokens such as if, else, comparison, id, number and literal; these are names of tokens. Typically with a lexer/parser a token is a structure that holds not only the name of the token, but the characters/symbols that make up the token and the start and end position of the string of characters that make up the token, with the start and end position being used for error reporting, highlighting, etc.

现在词法分析器接受字符/符号的输入，并使用词法分析器的规则将输入的字符/符号转换为标记.现在使用词法分析器/解析器的人对他们经常使用的东西有自己的说法.您认为构成标记的字符/符号序列是使用词法分析器/解析器的人所说的词素.因此，当您看到词素时，只需想到代表令牌的一系列字符/符号.在比较示例中，字符/符号的序列可以是不同的模式，例如 < 或 > 或 else 或 3.14等

Now the lexer takes the input of characters/symbols and using the rules of the lexer converts the input characters/symbols into tokens. Now people who work with lexer/parser have their own words for things they use often. What you think of as a sequence of characters/symbols that make up a token are what people who use lexer/parsers call lexeme. So when you see lexeme, just think of a sequence of characters/symbols representing a token. In the comparison example, the sequence of characters/symbols can be different patterns such as < or > or else or 3.14, etc.

另一种思考两者之间关系的方式是，标记是解析器使用的一种编程结构，它具有一个名为词素的属性，用于保存输入中的字符/符号.现在，如果您查看代码中标记的大多数定义，您可能不会将词位视为标记的属性之一.这是因为标记更有可能保存表示标记和词位的字符/符号的开始和结束位置，字符/符号的序列可以根据需要从开始和结束位置导出，因为输入是静态的.

Another way to think of the relation between the two is that a token is a programming structure used by the parser that has a property called lexeme that holds the character/symbols from the input. Now if you look at most definitions of token in code you may not see lexeme as one of the properties of the token. This is because a token will more likely hold the start and end position of the characters/symbols that represent the token and the lexeme, sequence of characters/symbols can be derived from the start and end position as needed because the input is static.

这篇关于令牌和词位有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！