Windows上的python libclang绑定无法从崇高的文本初始化翻译单元 [英] python libclang bindings on Windows fail to initialize a translation unit from sublime text

问题描述

简短说明:使用libclang自动完成代码不适用于Sublime Text 3随附的python.

Short description: using libclang to autocomplete code does not work with python that comes bundled with Sublime Text 3.

详细信息: Github

本质上，存在一个脚本，该脚本使用稍作更改的cindex.py(与python 3和clang 3.8兼容)，并从测试源文件构建转换单元.然后重新解析并尝试完成.

In essence, there is a script that uses a slightly changed cindex.py (compatible with python 3 and clang 3.8) and builds a Translation Unit from a test source file. It then reparses it and tries to complete.

该脚本在使用Powershell中的Python 3.3.5时可以按预期工作.

The script works as expected on using Python 3.3.5 from Powershell.

当放入Sublime Text 3的Packages文件夹中时，会产生错误. Sublime Text 3报告的Python版本是3.3.6.错误:

When put into Packages folder on Sublime Text 3 it produces an error. Python version as reported by Sublime Text 3 is 3.3.6. The error:

Traceback (most recent call last):
  File "C:\Program Files\Sublime Text 3\sublime_plugin.py", line 78, in reload_plugin
    m = importlib.import_module(modulename)
  File "./python3.3/importlib/__init__.py", line 90, in import_module
  File "<frozen importlib._bootstrap>", line 1584, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "C:\Users\igor\AppData\Roaming\Sublime Text 3\Packages\test_clang\script.py", line 21, in <module>
    tu = TU.from_source(filename=filename)
  File "C:\Users\igor\AppData\Roaming\Sublime Text 3\Packages\test_clang\clang\cindex38.py", line 2372, in from_source
    raise TranslationUnitLoadError("Error parsing translation unit.")
clang.cindex38.TranslationUnitLoadError: Error parsing translation unit.

之所以发生这种情况，是因为cindex.py中libclang返回的转换单元的ptr是None.唯一让我感到奇怪的是，它仅在捆绑了Sublime Text 3的python中发生.

This is happening because the ptr to tranlation unit returned by libclang inside cindex.py is None. The only thing that is strange for me is that it happens only with python bundled with sublime text 3.

其他人也会发生吗?有谁知道这可能是什么原因或如何对其进行调试?

Does it happen also to other people? Does anyone have any idea what could be the cause or how to debug it?

如果您无法运行此处提供的示例，请随时对我执行ping操作.

Also feel free to ping me if you cannot run the example provided here.

UPD::在测试项目的问题中，我们发现，不是ctypes捆绑在了崇高的文本中.将这些内容从崇高的文本替换为系统中安装的文本会产生相同的错误.

UPD: in the issues of the test project we have found out that it is not ctypes that is bundled in sublime text. Replacing the ones from sublime text to the ones installed in the system produces the same error.

UPD2::我已经剥离了测试库中的cindex.py文件，以仅包含与问题描述相同的问题所需的最少代码.也许这将有助于就什么是错误的问题产生新的想法?另外，我想明确指出，相同的代码在Linux和OSX上都能按预期工作.

UPD2: I have stripped down the cindex.py file in the test repository to only contain a bare minimum of code that is needed to run into the same issue as described in the question. Maybe this will help generating new ideas on what can be wrong? Also, I want to explicitly point out that the same code works exactly as expected on both Linux and OSX.

推荐答案

有关此故障的可调试性缺乏很多.我追踪了一堆libclang绑定源，希望找到一种解决方法，以解决由于python引发的TranslationUnitLoadError缺乏可调试性的问题.

The debugability about this failure is lacking a great deal it turns out. I traced a bunch of the libclang binding source hoping to find a workaround for the absence of debugability surrounding TranslationUnitLoadError's thrown from python.

即使您使用ctypes errcheck 回调，如下所示...

There seems to be some fundamental limitations here even if you use an ctypes errcheck callback like the following...

# MODIFIED cindex.py

def errcheck_callable(result, func, arguments):
    print(f"ERROR---result={result}, func={func}, arguments={arguments}")
    import pdb
    pdb.set_trace()

functionList = [
    ...
    ("clang_parseTranslationUnit",
    [Index, c_interop_string, c_void_p, c_int, c_void_p, c_int, c_int],
    c_object_p,
    errcheck_callable, # <--- makes this dll function a bit more debugable during call to `register_function`
    ),
    ...
]

但是在翻译单元故障期间触发的错误回调中没有太多内容

There's not much content in the error callback triggered during a translation unit failure however:

> /Users/USERX/python3.7/site-packages/clang/cindex.py(155)errcheck_callable()
-> print(f"ERROR---result={result}, func={func}, arguments={arguments}")
ERROR---result=<clang.cindex.LP_c_void_p object at 0x10b1aa9d8>, func=<_FuncPtr object at 0x10b1bf5c0>, arguments=(<clang.cindex.Index object at 0x10aea1e48>, None, <clang.cindex.c_char_p_Array_62 object at 0x10b1aa620>, 62, None, 0, 0)
(Pdb) result.contents
*** ValueError: NULL pointer access

有一个待处理的FIXME

There's a pending FIXME comment added several years back regarding the debugability gaps inherent to libclang's clang_parseTranslationUnit.

# FIXME: Make libclang expose additional error information in this scenario.

此处对此帖子进行了一些讨论

A bit of discussion takes place here on this post Dealing with parse errors with Python bindings of libclang . The best suggestion there appears to come from the idea to attach a debugger to libclang:

...您也许可以让调试器在clang_parseTranslationUnit上中断并检查那里的错误状态.

...you might be able to get a debugger to break on clang_parseTranslationUnit and inspect the error state there.

为了深入了解内部原理，基本上libclang是通过ctypes加载到python中的

To dive into a bit of the internals basically libclang is loaded into python via the ctypes call to cdll.LoadLibrary to create a CDLL instance. Then a hardcoded set of functions defined in functionList as a set of tuples are all registered via register_functions to give them a deeper python presence. The actual TranslationUnitLoadError gets raised within the classmethod TranslationUnit.from_source which makes a direct call to libclang function in the line

ptr = conf.lib.clang_createTranslationUnit(index, fspath(filename))

我认为可调试性在这里被截断了，因为底层的 SEHException 适用于.net.使用该运行时，您可以调试非托管代码.但是，这里没有等效项.

I believe it's here where the debugability gets truncated, because the underlying source for the python bindings is C, not C++, so there's no exception handling to bubble up like an SEHException for .net would. With that runtime you could debug unmanaged code. No equivalent here however.

您可以从其TU. cpp#L3587-L3603"rel =" nofollow noreferrer>源 ...

You can trace the translation unit variable TU down the call stack from its source...

CXTranslationUnit
clang_parseTranslationUnit(CXIndex CIdx,
                        const char *source_filename,
                        const char *const *command_line_args,
                        int num_command_line_args,
                        struct CXUnsavedFile *unsaved_files,
                        unsigned num_unsaved_files,
                        unsigned options) {
CXTranslationUnit TU;
enum CXErrorCode Result = clang_parseTranslationUnit2(
    CIdx, source_filename, command_line_args, num_command_line_args,
    unsaved_files, num_unsaved_files, options, &TU);

如果我发现更多实质性内容，我将更新此答案.鉴于存在可调试性的差距，像这样 fella一样，直接从C ++进行libclang分析可能会更加富有成效. 或通过使用命令行工具clang-query，如详细的

I'll update this answer if I discover anything more substantive. Given this debugability gap it may be more fruitful to do libclang analysis straight from C++ like this fella does, or by using the command line tool clang-query like detailed here

这篇关于Windows上的python libclang绑定无法从崇高的文本初始化翻译单元的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！