我如何用alex / haskell做python风格的缩进/缩进标记？ [英] How do I do python-style indent/dedent tokens with alex/haskell?

问题描述

该语言被指定为具有pythonesque重要缩进，INDENT标记或DEDENT在一个像C这样的传统命令式语言中，你会在词法分析器中保持一个全局，并用每一行的缩进级别来更新它。

。

这在Alex / Haskell中不起作用，因为我不能用Haskell在任何地方存储任何全局数据，并且我不能把所有的lexing规则放在任何内部monad或其他东西。

那么，我该怎么做呢？它甚至有可能吗？或者我将不得不编写自己的词法分析器并避免使用alex？注意，在其他对空白敏感的语言中 - 比如Haskell - 布局处理确实是在词法分析器中完成的。 GHC实际上在Alex中实现了布局处理。这里是源代码：

https://github.com/ghc/ghc/blob/master/compiler/parser/Lexer.x

有一些正如jrockway指出的那样，在你的问题中存在严重的错误，导致你误入歧途。 我无法在Haskell的任何地方存储任何全球数据处于错误的轨道上。首先，你可以具有全局状态，其次，当Alex以安全的方式完全支持规则中的状态转换时，你不应该在这里使用全局状态。

查看Alex提供的AlexState结构，让您通过词法分析器对状态进行线程化。然后，看看在GHC的布局实现中如何使用状态来实现缩进/取消缩进布局规则。 （在GHC的词法分析器中搜索 - 布局处理以查看状态是如何被推送和弹出的）。

I'm writing a lexer for a small language in Alex with Haskell.

The language is specified to have pythonesque significant indentation, with an INDENT token or a DEDENT token emitted whenever the indentation level changes.

In a traditional imperative language like C, you'd keep a global in the lexer and update it with the indentation level at each line.

This doesn't work in Alex/Haskell because I can't store any global data anywhere with Haskell, and I can't put all my lexing rules inside any monad or anything.

So, how can I do this? Is it even possible? Or will i have to write my own lexer and avoid using alex?

解决方案

Note that in other whitespace-sensitive languages -- like Haskell -- the layout handling is indeed done in the lexer. GHC in fact implements layout handling in Alex. Here's the source:

https://github.com/ghc/ghc/blob/master/compiler/parser/Lexer.x

There are some serious errors in your question that lead you astray, as jrockway points out. "I can't store any global data anywhere with Haskell" is on the wrong track. Firstly, you can have global state, secondly, you should not be using global state here, when Alex fully supports state transitions in rules in a safe manner.

Look at the AlexState structure that Alex provides, letting you thread state through your lexer. Then, look at how the state is used in GHC's layout implementation to implement indent/unindent of the layout rules. (Search for "-- Layout processing" in GHC's lexer to see how the state is pushed and popped).

这篇关于我如何用alex / haskell做python风格的缩进/缩进标记？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！