用libclang解析;无法解析某些令牌（Windows中的Python） [英] Parsing with libclang; unable to parse certain tokens (Python in Windows)

问题描述

我有一些代码（从此处和此处），它使用 libclang解析C ++源文件 Python（Widnows）并获取其所有的声明语句，如下所示：

I have some code (taken and adapted from here and here), which uses libclang to parse C++ sourcefiles in Python (Widnows) and get all of its declaration statements, as seen here:

import clang.cindex

def parse_decl(node):
    reference_node = node.get_definition()
    if node.kind.is_declaration():
        print(node.kind, node.kind.name, 
              node.location.line, ',', node.location.column, 
              reference_node.displayname)

    for ch in node.get_children():
        parse_decl(ch)

# configure path
clang.cindex.Config.set_library_file('C:/Program Files (x86)/LLVM/bin/libclang.dll')

index = clang.cindex.Index.create()
trans_unit = index.parse(r'C:\path\to\sourcefile\test.cpp', args=['-std=c++11'])

parse_decl(trans_unit.cursor)

b $ b

对于以下C ++源文件（ test_ok.cpp ）：

/* test_ok.cpp
 */

#include <iostream>
#include <fstream>
#include <string>
#include <algorithm>
#include <cmath>
#include <iomanip>
using namespace std;

int main (int argc, char *argv[]) {
  int linecount = 0;
  double array[1000], sum=0, median=0, add=0;
  string filename;
  if (argc <= 1)
      {
          cout << "Error: no filename specified" << endl;
          return 0;
      }
//program checks if a filename is specified
  filename = argv[1];
  ifstream myfile (filename.c_str());
  if (myfile.is_open())
  {
    myfile >> array[linecount];
    while ( myfile.good() )
    {
        linecount++;
        myfile >> array[linecount];
    }
    myfile.close();
  }

parse 解析并输出：

CursorKind.USING_DIRECTIVE USING_DIRECTIVE 10 , 17 std
CursorKind.FUNCTION_DECL FUNCTION_DECL 12 , 5 main(int, char **)
CursorKind.PARM_DECL PARM_DECL 12 , 15 argc
CursorKind.PARM_DECL PARM_DECL 12 , 27 argv
CursorKind.VAR_DECL VAR_DECL 13 , 7 linecount
CursorKind.VAR_DECL VAR_DECL 14 , 10 array
CursorKind.VAR_DECL VAR_DECL 14 , 23 sum
CursorKind.VAR_DECL VAR_DECL 14 , 30 median
CursorKind.VAR_DECL VAR_DECL 14 , 40 add
CursorKind.VAR_DECL VAR_DECL 15 , 10 filename
CursorKind.VAR_DECL VAR_DECL 23 , 12 myfile

Process finished with exit code 0

$ b $ c

HOWEVER ，

/ code>）：

/* test.cpp
*/
#include <iostream>
#include <vector>
#include <fstream>
#include <cmath>
#include <algorithm>
#include <iomanip>

using namespace std;

void readfunction(vector<double>& numbers, ifstream& myfile)
{

  double number;
  while (myfile >> number) {
  numbers.push_back(number);}

}

double meanfunction(vector<double>& numbers)
{

  double total=0;
  vector<double>::const_iterator i;
  for (i=numbers.begin(); i!=numbers.end(); ++i) {
  total +=*i; }
  return total/numbers.size();

}

解析不完整：

CursorKind.USING_DIRECTIVE USING_DIRECTIVE 8 , 17 std
CursorKind.VAR_DECL VAR_DECL 10 , 6 readfunction

Process finished with exit code 0

解析不能处理 vector< double>&数字等，并停止解析该部分代码。

the parsing cannot handle the lines such as vector<double>& numbers etc and stops parsing that part of the code.

我相信这个问题类似于另一个 SO问题。我试图明确使用 std = c ++ 11 解析参数没有成功。在该问题的回答中（即使它没有解决问题）使用 - 还建议使用x c ++ ，但我不知道如何在上面的代码中添加。

I believe the issue is similar as the one described in another SO question. I have tried to explicitly use the std=c++11 parse argument with no success. In an answer of that question (even though it did not solve the problem) the use of -x c++ is also suggested but I have no idea how to add that in my code above.

任何人都可以指向libclang解析C ++语句（如 test.cpp 中的语句）的解决方案？

Anyone can point to a solution for libclang to parse C++ statements like the ones in test.cpp?

此外，我可以让它继续解析，即使它得到一个令牌，它不能解析吗？

Also, can I make it so it will continue parsing even though if it gets to a token it cannot parse?

推荐答案

默认情况下，libclang不会添加编译器系统的include路径。

By default, libclang doesn't add the compiler system include path.

始终确保您已检查诊断 - 如编译器错误消息，它们倾向于指示如何解决任何问题。在这种情况下，显然存在包括问题：

Always make sure you've checked the diagnostics - like compiler error messages, they tend to indicate how to resolve any issues. In this case, it would have been a reasonably obvious there was an include issue:

<Diagnostic severity 4, location <SourceLocation file 'test.cpp', line 3, column 10>, spelling "'iostream' file not found">

如果您确保libclang添加了这些路径，它应该开始工作。

If you make sure libclang adds those paths, it should start working.

这个问题包括解决这个问题的方法。这似乎是Stackoverflow上重复出现的主题，因此我写了 ccsyspath 以帮助在OSX上找到这些路径，Linux和Windows。稍微简化您的代码：

This question includes an approach to solving this problem. This seems to be a recurring theme on Stackoverflow, so I wrote ccsyspath to help find those paths on OSX, Linux and Windows. Simplifying your code slightly:

import clang.cindex
clang.cindex.Config.set_library_file('C:/Program Files (x86)/LLVM/bin/libclang.dll')
import ccsyspath

index = clang.cindex.Index.create()

args    = '-x c++ --std=c++11'.split()
syspath = ccsyspath.system_include_paths('clang++')
incargs = [ b'-I' + inc for inc in syspath ]
args    = args + incargs

trans_unit = index.parse('test.cpp', args=args)
for node in trans_unit.cursor.walk_preorder():
    if node.location.file is None:
        continue
    if node.location.file.name != 'test.cpp':
        continue
    if node.kind.is_declaration():
        print(node.kind, node.location)

c $ c> args 结束于：

Where my args end up being:

['-x',
 'c++',
 '--std=c++11',
 '-IC:\\Program Files (x86)\\LLVM\\bin\\..\\lib\\clang\\3.8.0\\include',
 '-IC:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\include',
 '-IC:\\Program Files (x86)\\Windows Kits\\8.1\\include\\shared',
 '-IC:\\Program Files (x86)\\Windows Kits\\8.1\\include\\um',
 '-IC:\\Program Files (x86)\\Windows Kits\\8.1\\include\\winrt']

，输出为：

(CursorKind.USING_DIRECTIVE, <SourceLocation file 'test.cpp', line 10, column 17>)
(CursorKind.FUNCTION_DECL, <SourceLocation file 'test.cpp', line 12, column 6>)
(CursorKind.PARM_DECL, <SourceLocation file 'test.cpp', line 12, column 35>)
(CursorKind.PARM_DECL, <SourceLocation file 'test.cpp', line 12, column 54>)
(CursorKind.VAR_DECL, <SourceLocation file 'test.cpp', line 15, column 14>)
(CursorKind.FUNCTION_DECL, <SourceLocation file 'test.cpp', line 21, column 8>)
(CursorKind.PARM_DECL, <SourceLocation file 'test.cpp', line 21, column 37>)
(CursorKind.VAR_DECL, <SourceLocation file 'test.cpp', line 24, column 14>)
(CursorKind.VAR_DECL, <SourceLocation file 'test.cpp', line 25, column 40>)

这篇关于用libclang解析;无法解析某些令牌（Windows中的Python）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！