C中的预处理器命令是否算作令牌? [英] Are the preprocessor commands in C counted as tokens?

问题描述

我正在阅读有关令牌并计算程序中令牌的数量.

I was reading about tokens and counting the number of tokens in a program.

以前，我在某处读到预处理器命令不算作令牌.但是，当我在 Geeksforgeeks 上了解令牌时，它在特殊符号"部分中给出:

Previously I read somewhere that preprocessor commands are not counted as tokens. But when I read about tokens on Geeksforgeeks it is given in section "special symbols":

预处理器(#):预处理器是一个宏处理器，编译器会在自动编译之前自动使用它来转换程序.

pre processor(#): The preprocessor is a macro processor that is used automatically by the compiler to transform your program before actual compilation.

所以我很困惑，在程序中，如果我们编写 #define 会是令牌吗?

So I am confused that in a program, if we write #define will it be a token?

例如:

#include<stdio.h> 
#define max 100 
int main() 
{ 
    printf("max is %d", max); 
    return 0; 
} 

此示例中有多少个令牌??

How many tokens are in this example.?

推荐答案

链接的文章充满了基本错误，因此不应依赖.

The linked article is full of basic errors, and should not be relied upon.

解析C或C ++的过程定义为一系列转换: ¹

The process of parsing C or C++ is defined as a series of transformations:¹

1. 反斜杠换行符什么都不会替换-甚至没有空格.
2. 评论被删除，并用单个空格替换.
3. 尚存的文本将转换为一系列预处理标记.它们比语言本身所使用的标记的具体程度更低:例如，关键字 if 是语言本身的IF标记，而仅仅是预处理器的IDENT标记.
4. 执行预处理指令并扩展宏.
5. 每个预处理令牌都转换为令牌.
6. 令牌流被解析为抽象语法树，其余的编译器从那里获取它.

1. Backslash-newline is replaced with nothing whatsoever -- not even a space.
2. Comments are removed and replaced with a single space each.
3. The surviving text is converted into a series of preprocessing tokens. These are less specific than the tokens used by the language proper: for instance, the keyword if is an IF token to the language proper, but just an IDENT token to the preprocessor.
4. Preprocessing directives are executed and macros are expanded.
5. Each preprocessing token is converted into a token.
6. the stream of tokens is parsed into an abstract syntax tree, and the rest of the compiler takes it from there.

您的示例程序

#include<stdio.h> 
#define max 100 
int main() 
{ 
    printf("max is %d", max); 
    return 0; 
}

在转换3之后，将是这一系列的23个预处理令牌:

will, after transformation 3, be this series of 23 preprocessing tokens:

PUNCT:# IDENT:include INCLUDE-ARG:<stdio.h>
PUNCT:# IDENT:define IDENT:max PP-NUMBER:100
IDENT:int IDENT:main PUNCT:( PUNCT:)
PUNCT:{
IDENT:printf PUNCT:( STRING:"max is %d" PUNCT:, IDENT:max PUNCT:) PUNCT:;
IDENT:return PP-NUMBER:0 PUNCT:;
PUNCT:}

该指令在此阶段仍然存在.请注意， #include 和 #define 分别是两个标记:#和指令名称是分开的.有些人喜欢编写复杂的 #if 嵌套，所有嵌套标记都在第1列中，但是指令名称缩进了.

The directives are still present at this stage. Please notice that #include and #define are each two tokens: the # and the directive name are separate. Some people like to write complex #if nests with the hashmarks all in column 1 but the directive names indented.

在转换5之后，这些指令不见了，我们有了这一系列的16 + n个令牌:

After transformation 5, though, the directives are gone and we have this series of 16+n tokens:

[ ... some large volume of tokens produced from the contents of stdio.h ... ]
INT IDENT:main LPAREN RPAREN
LBRACE
IDENT:printf LPAREN STRING:"max is %d" COMMA DECIMAL-INTEGER:100 RPAREN SEMICOLON
RETURN DECIMAL-INTEGER:0 SEMICOLON
RBRACE

其中"n"是许多令牌，都来自stdio.h.

where 'n' is however many tokens came from stdio.h.

预处理指令( #include ， #define ， #if 等) 始终从令牌流并可能已被其他令牌替换，因此在转换6之后您将永远不会有直接来自指令行文本的令牌.但是，通常会有每个指令的效果产生的标记，例如 stdio.h 和 DECIMAL-INTEGER:100 替换 IDENT:max .

Preprocessing directives (#include, #define, #if, etc.) are always removed from the token stream and perhaps replaced with something else, so you will never have tokens after transformation 6 that directly result from the text of a directive line. But you will usually have tokens that result from the effects of each directive, such as the contents of stdio.h, and DECIMAL-INTEGER:100 replacing IDENT:max.

最后，C和C ++几乎几乎完成了这一系列操作，但并不完全相同，并且这些规范在形式上是独立的.只要您仅使用预处理器执行简单的操作，通常就可以依靠预处理操作来在两种语言中表现相同，这是如今的最佳做法.

Finally, C and C++ do this series of operations almost, but not quite, the same, and the specifications are formally independent. You can usually rely on preprocessing operations to behave the same in both languages, as long as you're only doing simple things with the preprocessor, which is best practice nowadays anyway.

¹ 有时您会看到人们在谈论翻译阶段，这是C和C ++标准正式描述这一系列操作的方式.我的名单不是翻译阶段的名单.它包含一些单独的项目要点，这些要点按标准归为一个阶段，并省略了与该讨论无关的几个步骤.

¹ You will sometimes see people talking about translation phases, which are the way the C and C++ standards officially describe this series of operations. My list is not the list of translation phases; it includes separate bullet points for some things that are grouped as a single phase by the standards, and leaves out several steps that aren't relevant to this discussion.

这篇关于C中的预处理器命令是否算作令牌?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！