与生成器一起使用超前 [英] Using lookahead with generators

问题描述

我已经在Python中实现了基于生成器的扫描程序，该扫描程序将字符串标记为(令牌类型，令牌值)形式的元组:

I have implemented a generator-based scanner in Python that tokenizes a string into tuples of the form (token type, token value):

for token in scan("a(b)"):
    print token

将打印

("literal", "a")
("l_paren", "(")
...

下一个任务意味着解析令牌流，为此，我需要能够将当前项从当前项向前看，而无需将指针也向前移动.迭代器和生成器不能一次提供完整的项目序列，而是根据需要提供每个项目，这使得与列表相比，前瞻变得有些棘手，因为除非调用__next__()，否则下一个项目是未知的.

The next task implies parsing the token stream and for that, I need be able to look one item ahead from the current one without moving the pointer ahead as well. The fact that iterators and generators do not provide the complete sequence of items at once but each item as needed makes lookaheads a bit trickier compared to lists, since the next item is not known unless __next__() is called.

基于生成器的前瞻的简单实现会是什么样子?目前，我正在使用一种解决方法，这意味着从生成器中列出一个列表:

What could a straightforward implementation of a generator-based lookahead look like? Currently I'm using a workaround which implies making a list out of the generator:

token_list = [token for token in scan(string)]

通过以下类似的方法可以很容易地实现前瞻:

The lookahead then is easily implemented by something like that:

try:
    next_token = token_list[index + 1]
except: IndexError:
    next_token = None

当然，这很好.但是想一想，我的第二个问题浮出水面:真的有必要首先将scan()生成为发电机吗?

Of course this just works fine. But thinking that over, my second question arises: Is there really a point of making scan() a generator in the first place?

推荐答案

您可以编写一个包装器，该包装器缓冲生成器中的某些项目，并提供lookahead()函数来查看这些缓冲的项目:

You can write a wrapper that buffers some number of items from the generator, and provides a lookahead() function to peek at those buffered items:

class Lookahead:
    def __init__(self, iter):
        self.iter = iter
        self.buffer = []

    def __iter__(self):
        return self

    def next(self):
        if self.buffer:
            return self.buffer.pop(0)
        else:
            return self.iter.next()

    def lookahead(self, n):
        """Return an item n entries ahead in the iteration."""
        while n >= len(self.buffer):
            try:
                self.buffer.append(self.iter.next())
            except StopIteration:
                return None
        return self.buffer[n]

这篇关于与生成器一起使用超前的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！