如何使用升压::精神::莱克斯莱克斯文件而不读取整个文件到内存中第一次？ [英] How to use Boost::Spirit::Lex to lex a file without reading the whole file into memory first?

问题描述

我期待在编写使用boost ::精神::法词法分析器，但所有我能找到的例子似乎假设你已经先阅读整个文件到内存中。我想编写不需要整个字符串是在RAM中的词法分析，这可能吗？或者我需要用别的东西吗？

I'm looking at writing a lexer using boost::spirit::lex, but all the examples I can find seem to assume that you've read the entire file into RAM first. I'd like to write a lexer that doesn't require the whole string to be in RAM, is that possible? Or do I need to use something else?

我试着使用istream_iterator，但除非我用为const char提升给了我一个编译错误，*作为迭代器类型。

I tried using istream_iterator, but boost gives me a compile error unless I use const char* as the iterator types.

例如。我能找到的所有例子基本上做到这一点：

e.g. All the examples I can find basically do this:

lex_functor_type< lex::lexertl::lexer<> > lex_functor;

// assumes entire file is in memory
char const* first = str.c_str();
char const* last = &first[str.size()];

bool r = lex::tokenize(first, last, lex_functor, 
    boost::bind(lex_callback_functor(), _1, ... ));

此外，是有可能确定从某种程度上标记法行/列数？

Also, is it possible to determine line/column numbers from lex tokens somehow?

谢谢！

推荐答案

只要它符合标准前向迭代器的要求精神莱克斯适用于任何迭代器。这意味着你可以与任何符合迭代器喂词法分析器（调用 ::法记号化（））。举例来说，如果你想使用的std :: istream的，你可以把它包装成一个的boost ::精神:: istream_iterator ：

Spirit Lex works with any iterator as long as it conforms to the requirements of standard forward iterators. That means you can feed the lexer (invoke lex::tokenize()) with any conforming iterator. For instance, if you want to use a std::istream, you could wrap it into a boost::spirit::istream_iterator:

bool tokenize(std::istream& is, ...)
{
    lex_functor_type< lex::lexertl::lexer<> > lex_functor;

    boost::spirit::istream_iterator first(is);
    boost::spirit::istream_iterator last;

    return lex::tokenize(first, last, lex_functor,
        boost::bind (lex_callback_functor(), _1, ... ));   
}

和它的工作。

有关您的问题（与输入的行/列数）的第二部分：是的，它可以跟踪使用词法分析器输入位置。这不是小事，虽然。您需要创建存储行/列信息你自己的令牌类型和使用，而不是predefined令牌类型。很多人一直在问这个，所以我可能只是继续前进，创造一个例子。

For the second part of your question (related to the line/column number of the input): yes it is possible to track the input position using the lexer. It's not trivial, though. You need to create your own token type which stores the line/column information and use this instead of the predefined token type. Many people have been asking for this, so I might just go ahead and create an example.

这篇关于如何使用升压::精神::莱克斯莱克斯文件而不读取整个文件到内存中第一次？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！