寻找“标记器"、“解析器"的明确定义和“词法分析器"它们是以及如何相互关联和使用的? [英] Looking for a clear definition of what a "tokenizer", "parser" and "lexers" are and how they are related to each other and used?

问题描述

我正在寻找关于分词器"、解析器"和词法分析器"的明确定义以及它们之间的关系(例如，解析器是否使用分词器，反之亦然)?我需要创建一个程序，将通过 c/h 源文件来提取数据声明和定义.

I am looking for a clear definition of what a "tokenizer", "parser" and "lexer" are and how they are related to each other (e.g., does a parser use a tokenizer or vice versa)? I need to create a program will go through c/h source files to extract data declaration and definitions.

我一直在寻找示例并且可以找到一些信息，但我真的很难掌握语法规则、解析树和抽象语法树等基本概念以及它们如何相互关联.最终这些概念需要存储在实际程序中，但 1) 它们是什么样子的，2) 是否有通用的实现.

I have been looking for examples and can find some info, but I really struggling to grasp the underlying concepts like grammar rules, parse trees and abstract syntax tree and how they interrelate to each other. Eventually these concepts need to be stored in an actual program, but 1) what do they look like, 2) are there common implementations.

我一直在查看有关这些主题和程序(如 Lex 和 Yacc)的维基百科，但从未上过编译器课程(EE 专业)，我发现很难完全理解正在发生的事情.

I have been looking at Wikipedia on these topics and programs like Lex and Yacc, but having never gone through a compiler class (EE major) I am finding it difficult to fully understand what is going on.

推荐答案

分词器通常通过查找空格(制表符、空格、新行)将文本流分解为标记.

A tokenizer breaks a stream of text into tokens, usually by looking for whitespace (tabs, spaces, new lines).

词法分析器基本上是一个分词器，但它通常为标记附加额外的上下文——这个标记是一个数字，那个标记是一个字符串文字，另一个标记是一个相等运算符.

A lexer is basically a tokenizer, but it usually attaches extra context to the tokens -- this token is a number, that token is a string literal, this other token is an equality operator.

解析器从词法分析器中获取标记流，并将其转换为抽象语法树，表示(通常)由原始文本表示的程序.

A parser takes the stream of tokens from the lexer and turns it into an abstract syntax tree representing the (usually) program represented by the original text.

最后我查了一下，关于这个主题的最好的书是"编译器:原则，技术和工具"通常简称为龙之书".

Last I checked, the best book on the subject was "Compilers: Principles, Techniques, and Tools" usually just known as "The Dragon Book".

这篇关于寻找“标记器"、“解析器"的明确定义和“词法分析器"它们是以及如何相互关联和使用的?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！