如何重新加载 Python3 C 扩展模块? [英] How to Reload a Python3 C extension module?

问题描述

我为 Python 3.2 编写了一个 C 扩展 (mycext.c).该扩展依赖于存储在 C 头文件 (myconst.h) 中的常量数据.头文件由 Python 脚本生成.在同一个脚本中，我使用了最近编译的模块.Python3 myscript(未完整展示)中的工作流程如下:

configure_C_header_constants()write_constants_to_C_header() # 写 myconst.hos.system('python3 setup.py install --user') # 编译 mycext导入 mycextmycext.do_stuff()

这是第一次在 Python 会话中完美运行.如果我在同一会话中重复该过程(例如，在单元测试的两个不同测试用例中)，则始终(重新)加载 mycext 的第一个编译版本.

如何使用最新的编译版本有效地重新加载扩展模块?

解决方案

您可以使用 imp.reload() 函数.(这个函数曾经是 Python 2.x 的内置函数.请务必阅读文档——有一些注意事项！)

Python 的导入机制永远不会dlclose() 成为共享库.加载后，库将一直保留到进程终止.

您的选项(按有用性递减排序):

1. 将模块导入移动到子进程中，重新编译后再次调用子进程，即你有一个 Python 脚本 do_stuff.py 就可以了

   import mycextmycext.do_stuff()

   你使用

   调用这个脚本

   subprocess.call([sys.executable, "do_stuff.py"])

2. 将标头中的编译时常量转换为可从 Python 更改的变量，无需重新加载模块.

3. 在删除对模块的所有引用后手动dlclose()库(有点脆弱，因为您自己没有保存所有引用).

4. 推出自己的导入机制.

   这是如何做到这一点的示例.我写了一个最小的 Python C 扩展 mini.so，只导出一个名为 version 的整数.

   <预><代码>>>>导入 ctypes>>>libdl = ctypes.CDLL("libdl.so")>>>libdl.dlclose.argtypes = [ctypes.c_void_p]>>>so = ctypes.PyDLL("./mini.so")>>>so.PyInit_mini.argtypes = []>>>so.PyInit_mini.restype = ctypes.py_object>>>mini = so.PyInit_mini()>>>迷你版1>>>德尔迷你>>>libdl.dlclose(so._handle)0>>>德尔索

   此时，我增加了mini.c中的版本号并重新编译.

   <预><代码>>>>so = ctypes.PyDLL("./mini.so")>>>so.PyInit_mini.argtypes = []>>>so.PyInit_mini.restype = ctypes.py_object>>>mini = so.PyInit_mini()>>>迷你版2

   可以看到使用了新版本的模块.

   作为参考和实验，这里是 mini.c:

   #include 静态结构 PyModuleDef 微型模块 = {PyModuleDef_HEAD_INIT，迷你"，NULL，-1，NULL};PyMODINIT_FUNCPyInit_mini(){PyObject *m = PyModule_Create(&minimodule);PyModule_AddObject(m, "version", PyLong_FromLong(1));返回 m;}

I wrote a C extension (mycext.c) for Python 3.2. The extension relies on constant data stored in a C header (myconst.h). The header file is generated by a Python script. In the same script, I make use of the recently compiled module. The workflow in the Python3 myscript (not shown completely) is as follows:

configure_C_header_constants() 
write_constants_to_C_header() # write myconst.h
os.system('python3 setup.py install --user') # compile mycext
import mycext
mycext.do_stuff()

This works perfectly fine the in a Python session for the first time. If I repeat the procedure in the same session (for example, in two different testcases of a unittest), the first compiled version of mycext is always (re)loaded.

How do I effectively reload a extension module with the latest compiled version?

解决方案

You can reload modules in Python 3.x by using the imp.reload() function. (This function used to be a built-in in Python 2.x. Be sure to read the documentation -- there are a few caveats!)

Python's import mechanism will never dlclose() a shared library. Once loaded, the library will stay until the process terminates.

Your options (sorted by decreasing usefulness):

1. Move the module import to a subprocess, and call the subprocess again after recompiling, i.e. you have a Python script do_stuff.py that simply does

   import mycext
   mycext.do_stuff()
   

   and you call this script using

   subprocess.call([sys.executable, "do_stuff.py"])
   

2. Turn the compile-time constants in your header into variables that can be changed from Python, eliminating the need to reload the module.

3. Manually dlclose() the library after deleting all references to the module (a bit fragile since you don't hold all the references yourself).

4. Roll your own import mechanism.

   Here is an example how this can be done. I wrote a minimal Python C extension mini.so, only exporting an integer called version.

   >>> import ctypes
   >>> libdl = ctypes.CDLL("libdl.so")
   >>> libdl.dlclose.argtypes = [ctypes.c_void_p]
   >>> so = ctypes.PyDLL("./mini.so")
   >>> so.PyInit_mini.argtypes = []
   >>> so.PyInit_mini.restype = ctypes.py_object 
   >>> mini = so.PyInit_mini()
   >>> mini.version
   1
   >>> del mini
   >>> libdl.dlclose(so._handle)
   0
   >>> del so
   

   At this point, I incremented the version number in mini.c and recompiled.

   >>> so = ctypes.PyDLL("./mini.so")
   >>> so.PyInit_mini.argtypes = []
   >>> so.PyInit_mini.restype = ctypes.py_object 
   >>> mini = so.PyInit_mini()
   >>> mini.version
   2
   

   You can see that the new version of the module is used.

   For reference and experimenting, here's mini.c:

   #include <Python.h>
   
   static struct PyModuleDef minimodule = {
      PyModuleDef_HEAD_INIT, "mini", NULL, -1, NULL
   };
   
   PyMODINIT_FUNC
   PyInit_mini()
   {
       PyObject *m = PyModule_Create(&minimodule);
       PyModule_AddObject(m, "version", PyLong_FromLong(1));
       return m;
   }
   

这篇关于如何重新加载 Python3 C 扩展模块?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！