从LLVM JIT调用Python代码 [英] Call Python code from LLVM JIT

问题描述

我用python语言编写了lexer/parser/compiler，稍后应在LLVM JIT-VM(使用llvm-py)中运行.到目前为止，前两个步骤非常简单，但是(即使我还没有启动编译任务)，当我的代码想要调用Python代码(通常)或与Python词法分析器交互时，我看到了一个问题/parser/compiler(特殊).我主要担心的是，代码应该能够在运行时动态地将附加代码加载到VM中，因此它必须从VM内部触发Python中的整个lexer/parser/compiler-chain.

I write a language lexer/parser/compiler in python, that should run in the LLVM JIT-VM (using llvm-py) later. The first two steps are quite straightforward for now, but (even if I didn't start the compile-task yet) I see a problem, when my code wants to call Python-Code (in general), or interact with the Python lexer/parser/compiler (in special) respectively. My main concern is, that the code should be able to dynamically load additional code into the VM at runtime and thus it must trigger the whole lexer/parser/compiler-chain in Python from within the VM.

首先:这是否有可能，或者一旦启动VM，VM是否不可更改"?

First of all: Is this even possible, or is the VM "unmutable" once it is started?

如果是这样，我目前会看到3种可能的解决方案(我愿意接受其他建议)

If it is I currently see 3 possible solutions (I am open for other suggestions)

• 脱离" VM，并可以直接调用主进程的Python函数(也许通过将其注册为LLVM函数，以某种方式重定向到主进程).我对此一无所获，无论如何，我不确定(如果是安全性之类)这是个好主意.
• 将运行时(静态或动态地在运行时)编译为LLVM-Assembly/-IR.这就要求IR代码能够修改其运行的VM
• 将运行时(静态)编译到库中，然后直接将其加载到VM中.同样，它必须能够向其运行的VM添加功能(等).

推荐答案

就像Eli所说，并没有阻止您调出Python C-API.当您从LLVM JIT内部调用外部函数时，它实际上仅在进程空间上使用dlopen()，因此，如果您从llvmpy内部运行，则您已经可以访问所有Python解释器符号，甚至可以与活动窗口进行交互.调用了ExecutionEngine的解释器，或者您可以根据需要旋转一个新的Python解释器.

Like Eli said, there's not stopping you from calling out to the Python C-API. When you call an external function from inside of the LLVM JIT it effectively just uses dlopen() on the process space so if you're running from inside of llvmpy you already have all the Python interpreter symbols accessible, you can even interact with the active interpreter that invoked the ExecutionEngine or you can spin a new Python interpreter if needed.

为使您入门，请使用我们的评估器创建一个新的C文件.

To get you started, create a new C file with our evaluator.

#include <Python.h>

void python_eval(const char* s)
{
    PyCodeObject* code = (PyCodeObject*) Py_CompileString(s, "example", Py_file_input);

    PyObject* main_module = PyImport_AddModule("__main__");
    PyObject* global_dict = PyModule_GetDict(main_module);
    PyObject* local_dict = PyDict_New();
    PyObject* obj = PyEval_EvalCode(code, global_dict, local_dict);

    PyObject* result = PyObject_Str(obj);

    // Print the result if you want.
    // PyObject_Print(result, stdout, 0);
}

下面是一个用于编译的Makefile:

Here's a little Makefile to compile that:

CC = gcc
LPYTHON = $(shell python-config --includes)
CFLAGS = -shared -fPIC -lpthread $(LPYTHON)

.PHONY: all clean

all:
    $(CC) $(CFLAGS) cbits.c -o cbits.so

clean:
    -rm cbits.c

然后，我们从LLVM的常用样板开始，但使用ctypes将cbits.so共享库的共享对象加载到全局进程空间中，从而获得python_eval符号.然后只需创建一个带有函数的简单LLVM模块，为带有ctypes的Python源分配一个字符串，然后将指针传递给运行来自我们模块的JIT函数的ExecutionEngine，后者将Python源传递给C函数，调用Python C-API，然后返回到LLVM JIT.

Then we start with the usual boilerplate for LLVM but use ctypes to load the shared object of our cbits.so shared library into the global process space so that we have the python_eval symbol. Then just create a simple LLVM module with a function, allocate a string with some Python source with ctypes and pass the pointer to the ExecutionEngine running the JIT'd function from our module, which in turns passes the Python source to the C-function which invokes the Python C-API and then yields back to the LLVM JIT.

import llvm.core as lc
import llvm.ee as le

import ctypes
import inspect

ctypes._dlopen('./cbits.so', ctypes.RTLD_GLOBAL)

pointer = lc.Type.pointer

i32 = lc.Type.int(32)
i64 = lc.Type.int(64)

char_type  = lc.Type.int(8)
string_type = pointer(char_type)

zero = lc.Constant.int(i64, 0)

def build():
    mod = lc.Module.new('call python')
    evalfn = lc.Function.new(mod,
        lc.Type.function(lc.Type.void(),
        [string_type], False), "python_eval")

    funty = lc.Type.function(lc.Type.void(), [string_type])

    fn = lc.Function.new(mod, funty, "call")
    fn_arg0 = fn.args[0]
    fn_arg0.name = "input"

    block = fn.append_basic_block("entry")
    builder = lc.Builder.new(block)

    builder.call(evalfn, [fn_arg0])
    builder.ret_void()

    return fn, mod

def run(fn, mod, buf):

    tm = le.TargetMachine.new(features='', cm=le.CM_JITDEFAULT)
    eb = le.EngineBuilder.new(mod)
    engine = eb.create(tm)

    ptr = ctypes.cast(buf, ctypes.c_voidp)
    ax = le.GenericValue.pointer(ptr.value)

    print 'IR'.center(80, '=')
    print mod

    mod.verify()
    print 'Assembly'.center(80, '=')
    print mod.to_native_assembly()

    print 'Result'.center(80, '=')
    engine.run_function(fn, [ax])

if __name__ == '__main__':
    # If you want to evaluate the source of an existing function
    # source_str = inspect.getsource(mypyfn)

    # If you want to pass a source string
    source_str = "print 'Hello from Python C-API inside of LLVM!'"

    buf = ctypes.create_string_buffer(source_str)
    fn, mod = build()
    run(fn, mod, buf)

您应该显示以下输出:

=======================================IR=======================================
; ModuleID = 'call python'

declare void @python_eval(i8*)

define void @call(i8* %input) {
entry:
  call void @python_eval(i8* %input)
  ret void
}
=====================================Result=====================================
Hello from Python C-API inside of LLVM!

这篇关于从LLVM JIT调用Python代码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！