有没有一种方法可以将spirit :: lex字符串标记的内容匹配为Spirit :: qi语法中的文字 [英] Is there a way to match the content of a spirit::lex string token as a literal in a spirit::qi grammar
问题描述
我正在编写DSL，并使用Boost Spirit词法分析器标记我的输入.在我的语法中，我想要一个与此类似的规则(其中tok是词法分析器):
I'm writing a DSL and using a Boost Spirit lexer to tokenize my input. In my grammar, I want a rule similar to this (where tok is the lexer):
header_block =
tok.name >> ':' >> tok.stringval > ';' >>
tok.description >> ':' >> tok.stringval > ';'
;
我不想为语言指定保留字(例如名称"，描述")并处理它们在词法分析器和语法之间的同步，我只是将与[a-zA-Z_]\w*匹配的所有内容标记为单个标记类型(例如tok.symbol)，然后让语法将其分类.如果我不使用词法分析器，则可能会执行以下操作:
Rather than specifying reserved words for the language (e.g. "name", "description") and deal with synchronizing these between the lexer and grammar, I want to just tokenize everything that matches [a-zA-Z_]\w* as a single token type (e.g. tok.symbol), and let the grammar sort it out. If I weren't using a lexer, I might do something like this:
stringval = lexeme['"' >> *(char_ - '"') >> '"'];
header_block =
lit("name") >> ':' >> stringval > ';' >>
lit("description") >> ':' >> stringval > ';'
;
在混合词法分析器的情况下，我可以编译以下规则，但是当然，它比我想要的更匹配—它不在乎特定的符号值名称"和描述":
With a lexer in the mix, I can compile the following rule, but of course it matches more than I want — it doesn't care about the particular symbol values "name" and "description":
header_block =
tok.symbol >> ':' >> tok.stringval > ';' >>
tok.symbol >> ':' >> tok.stringval > ';'
;
我正在寻找的东西是这样的:
What I'm looking for is something like this:
header_block =
specific_symbol_matcher("name") >> ':' >> tok.stringval > ';' >>
specific_symbol_matcher("description") >> ':' >> tok.stringval > ';'
;
Qi可以提供什么可用的东西吗?如果我可以使用所提供的内容紧密联系，我宁愿不写自己的匹配器.如果我必须编写自己的匹配器，谁能建议该怎么做?
Does Qi provide anything I can use instead of my specific_symbol_matcher hand-waving, there? I'd rather not write my own matcher if I can get close using stuff that's provided. If I must write my own matcher, can anyone suggest how to do that?
推荐答案
如果令牌公开了std :: string，则应该可以:
If the token exposes a std::string, you should just be able to do:
statement =
( tok.keyword [ qi::_pass = (_1 == "if") ] >> if_stmt )
| ( tok.keyword [ qi::_pass = (_1 == "while) ] >> while_stmt );
如果我没看错，这或多或少就是您要问的内容.
If I understood you right, this is, more or less, what you were asking.
在查看时，请查看如果您只是想使用词法分析器来处理现有的语法，这就是我对
In case you're just struggling to make an existing grammar work with a lexer, here's what I just did with the calc_utree_ast.cpp example to make it work with a lexer.
它显示
- 如何直接使用公开的属性
- 只要这些char文字被注册为(匿名)标记，您仍然可以基于char-literal进行解析
- (简单的)表达游戏如何被最小化了
- 如何将跳过行为转移到词法分析器中
///////////////////////////////////////////////////////////////////////////////
//
// Plain calculator example demonstrating the grammar. The parser is a
// syntax checker only and does not do any semantic evaluation.
//
// [ JDG May 10, 2002 ] spirit1
// [ JDG March 4, 2007 ] spirit2
// [ HK November 30, 2010 ] spirit2/utree
// [ SH July 17, 2012 ] use a lexer
//
///////////////////////////////////////////////////////////////////////////////
#define BOOST_SPIRIT_DEBUG
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/support_utree.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_function.hpp>
#include <iostream>
#include <string>
namespace lex = boost::spirit::lex;
namespace qi = boost::spirit::qi;
namespace spirit = boost::spirit;
namespace phx = boost::phoenix;
// base iterator type
typedef std::string::const_iterator BaseIteratorT;
// token type
typedef lex::lexertl::token<BaseIteratorT, boost::mpl::vector<char, uint32_t> > TokenT;
// lexer type
typedef lex::lexertl::actor_lexer<TokenT> LexerT;
template <typename LexerT_>
struct Tokens: public lex::lexer<LexerT_> {
Tokens() {
// literals
uint_ = "[0-9]+";
space = " \t\r\n";
// literal rules
this->self += uint_;
this->self += '+';
this->self += '-';
this->self += '*';
this->self += '/';
this->self += '(';
this->self += ')';
using lex::_pass;
using lex::pass_flags;
this->self += space [ _pass = pass_flags::pass_ignore ];
}
lex::token_def<uint32_t> uint_;
lex::token_def<lex::omit> space;
};
namespace client
{
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
namespace spirit = boost::spirit;
struct expr
{
template <typename T1, typename T2 = void>
struct result { typedef void type; };
expr(char op) : op(op) {}
void operator()(spirit::utree& expr, spirit::utree const& rhs) const
{
spirit::utree lhs;
lhs.swap(expr);
expr.push_back(spirit::utf8_symbol_range_type(&op, &op+1));
expr.push_back(lhs);
expr.push_back(rhs);
}
char const op;
};
boost::phoenix::function<expr> const plus = expr('+');
boost::phoenix::function<expr> const minus = expr('-');
boost::phoenix::function<expr> const times = expr('*');
boost::phoenix::function<expr> const divide = expr('/');
struct negate_expr
{
template <typename T1, typename T2 = void>
struct result { typedef void type; };
void operator()(spirit::utree& expr, spirit::utree const& rhs) const
{
char const op = '-';
expr.clear();
expr.push_back(spirit::utf8_symbol_range_type(&op, &op+1));
expr.push_back(rhs);
}
};
boost::phoenix::function<negate_expr> neg;
///////////////////////////////////////////////////////////////////////////////
// Our calculator grammar
///////////////////////////////////////////////////////////////////////////////
template <typename Iterator>
struct calculator : qi::grammar<Iterator, spirit::utree()>
{
template <typename Tokens>
calculator(Tokens const& toks) : calculator::base_type(expression)
{
using qi::_val;
using qi::_1;
expression =
term [_val = _1]
>> *( ('+' >> term [plus(_val, _1)])
| ('-' >> term [minus(_val, _1)])
)
;
term =
factor [_val = _1]
>> *( ('*' >> factor [times(_val, _1)])
| ('/' >> factor [divide(_val, _1)])
)
;
factor =
toks.uint_ [_val = _1]
| '(' >> expression [_val = _1] >> ')'
| ('-' >> factor [neg(_val, _1)])
| ('+' >> factor [_val = _1])
;
BOOST_SPIRIT_DEBUG_NODE(expression);
BOOST_SPIRIT_DEBUG_NODE(term);
BOOST_SPIRIT_DEBUG_NODE(factor);
}
qi::rule<Iterator, spirit::utree()> expression, term, factor;
};
}
///////////////////////////////////////////////////////////////////////////////
// Main program
///////////////////////////////////////////////////////////////////////////////
int main()
{
std::cout << "/////////////////////////////////////////////////////////\n\n";
std::cout << "Expression parser...\n\n";
std::cout << "/////////////////////////////////////////////////////////\n\n";
std::cout << "Type an expression...or [q or Q] to quit\n\n";
using boost::spirit::utree;
typedef std::string::const_iterator iterator_type;
typedef Tokens<LexerT>::iterator_type IteratorT;
typedef client::calculator<IteratorT> calculator;
Tokens<LexerT> l;
calculator calc(l); // Our grammar
std::string str;
while (std::getline(std::cin, str))
{
if (str.empty() || str[0] == 'q' || str[0] == 'Q')
break;
std::string::const_iterator iter = str.begin();
std::string::const_iterator end = str.end();
utree ut;
bool r = lex::tokenize_and_parse(iter, end, l, calc, ut);
if (r && iter == end)
{
std::cout << "-------------------------\n";
std::cout << "Parsing succeeded: " << ut << "\n";
std::cout << "-------------------------\n";
}
else
{
std::string rest(iter, end);
std::cout << "-------------------------\n";
std::cout << "Parsing failed\n";
std::cout << "stopped at: \"" << rest << "\"\n";
std::cout << "-------------------------\n";
}
}
std::cout << "Bye... :-) \n\n";
return 0;
}
输入
8*12312*(4+5)
它打印(没有调试信息)
It prints (without debug info)
Parsing succeeded: ( * ( * 8 12312 ) ( + 4 5 ) )
这篇关于有没有一种方法可以将spirit :: lex字符串标记的内容匹配为Spirit :: qi语法中的文字的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
