寻找关于什么是“令牌化器"，“解析器"的明确定义.和“词法分析器"是什么以及它们如何相互关联和使用? [英] Looking for a clear definition of what a "tokenizer", "parser" and "lexers" are and how they are related to each other and used?

问题描述

我正在寻找"tokenizer"，"parser"和"lexer"分别是什么以及它们之间如何关联的明确定义(例如，解析器是否使用标记器，反之亦然)?我需要创建一个程序，该程序将通过c/h源文件提取数据声明和定义.

我一直在寻找示例并可以找到一些信息，但是我真的很难掌握语法规则，语法分析树和抽象语法树等基本概念以及它们之间的相互关系.最终，这些概念需要存储在实际的程序中，但是1)它们的外观如何，2)有常见的实现方式.

我一直在浏览有关诸如Lex和Yacc之类的主题和程序的Wikipedia，但从未经历过编译器类(EE专业)，我发现很难完全了解正在发生的事情.

解决方案

令牌生成器通常通过查找空格(制表符，空格，换行符)将文本流拆分为令牌.

词法分析器基本上是一个标记器，但通常会在标记上附加额外的上下文-该标记是数字，该标记是字符串文字，另一个标记是一个等于运算符.

解析器从词法分析器中获取令牌流，并将其转换为代表原始文本表示的(通常)程序的抽象语法树.

上次我检查过，关于该主题的最好的书是"Compilers:Principles，技术和工具" 通常简称为龙书".

I am looking for a clear definition of what a "tokenizer", "parser" and "lexer" are and how they are related to each other (e.g., does a parser use a tokenizer or vice versa)? I need to create a program will go through c/h source files to extract data declaration and definitions.

I have been looking for examples and can find some info, but I really struggling to grasp the underlying concepts like grammar rules, parse trees and abstract syntax tree and how they interrelate to each other. Eventually these concepts need to be stored in an actual program, but 1) what do they look like, 2) are there common implementations.

I have been looking at Wikipedia on these topics and programs like Lex and Yacc, but having never gone through a compiler class (EE major) I am finding it difficult to fully understand what is going on.

解决方案

A tokenizer breaks a stream of text into tokens, usually by looking for whitespace (tabs, spaces, new lines).

A lexer is basically a tokenizer, but it usually attaches extra context to the tokens -- this token is a number, that token is a string literal, this other token is an equality operator.

A parser takes the stream of tokens from the lexer and turns it into an abstract syntax tree representing the (usually) program represented by the original text.

Last I checked, the best book on the subject was "Compilers: Principles, Techniques, and Tools" usually just known as "The Dragon Book".

这篇关于寻找关于什么是“令牌化器"，“解析器"的明确定义.和“词法分析器"是什么以及它们如何相互关联和使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！