在python中进行词法分析，标记化和解析的资源 [英] Resources for lexing, tokenising and parsing in python

问题描述

人们可以指出我关于使用Python进行词法分析，解析和标记化的资源吗?

Can people point me to resources on lexing, parsing and tokenising with Python?

我正在对一个开源项目( hotwire )进行一些黑客攻击，对 lexes ，解析和标记化输入的命令.由于它是真正的工作代码，因此相当复杂，并且很难解决.

I'm doing a little hacking on an open source project (hotwire) and wanted to do a few changes to the code that lexes, parses and tokenises the commands entered into it. As it is real working code it is fairly complex and a bit hard to work out.

我以前没有研究过lex/parse/tokenise的代码，所以我想一种方法是在这方面完成一两个教程.我希望学习足够的知识来浏览实际上要更改的代码.有什么合适的地方吗? (理想情况下，它可以在下午完成，而无需先购买和阅读龙书...)

I haven't worked on code to lex/parse/tokenise before, so I was thinking one approach would be to work through a tutorial or two on this aspect. I would hope to learn enough to navigate around the code I actually want to alter. Is there anything suitable out there? (Ideally it could be done in an afternoon without having to buy and read the dragon book first ...)

(2008年10月7日)以下答案都不能完全满足我的要求.有了它们，我可以从头开始生成解析器，但是我想学习如何从头开始编写我自己的基本解析器，而不是使用lex和yacc或类似工具.完成后，我可以更好地了解现有代码.

(7 Oct 2008) None of the below answers quite give what I want. With them I could generate parsers from scratch, but I want to learn how to write my own basic parser from scratch, not using lex and yacc or similar tools. Having done that I can then understand the existing code better.

所以有人可以将我指向一个教程，在其中可以仅使用python从头构建基本解析器吗?

So could someone point me to a tutorial where I can build a basic parser from scratch, using just python?

推荐答案

我是 PLY .它是Lex&的纯Python实现. Yacc，有很多小细节，使其相当Pythonic且易于使用.自Lex& Yacc是最流行的词汇和词汇.作为PLY的分析工具，并且用于大多数项目，PLY的优势在于可以站在巨人的肩膀上.有关Lex& amp;的在线知识很多. Yacc，您可以将其自由地应用于PLY.

I'm a happy user of PLY. It is a pure-Python implementation of Lex & Yacc, with lots of small niceties that make it quite Pythonic and easy to use. Since Lex & Yacc are the most popular lexing & parsing tools and are used for the most projects, PLY has the advantage of standing on giants' shoulders. A lot of knowledge exists online on Lex & Yacc, and you can freely apply it to PLY.

PLY也有一个很好的文档页面，其中包含一些简单的示例，可以帮助您入门.

PLY also has a good documentation page with some simple examples to get you started.

有关许多Python解析工具的列表，请参见此.

For a listing of lots of Python parsing tools, see this.

这篇关于在python中进行词法分析，标记化和解析的资源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！