macOS上的RTLD_GLOBAL和两个级别的命名空间 [英] RTLD_GLOBAL and Two Level Namespaces on macOS
问题描述
阅读Apple文档中的执行Mach-O文件说:
After reading the Apple documentation for Executing Mach-O files it says:
OS X v10.1和更高版本的两级命名空间功能添加了 模块名称，作为其中定义的符号的符号名称的一部分 它. 这种方法可确保模块的符号名称不会与 其他模块中使用的名称.
The two-level namespace feature of OS X v10.1 and later adds the module name as part of the symbol name of the symbols defined within it. This approach ensures a module’s symbol names don’t conflict with the names used in other modules.
因此，在我的示例中，我将 python2 和 python3 加载到同一进程中.这两个Python库(默认情况下)都使用两级名称空间选项进行编译.两个库也都通过dlopen(..)加载了RTLD_GLOBAL标志，因此，具有相同名称的符号应该不会互相干扰，因为两个模块的名称不同(python27和python36).
So in my example I am loading python2 and python3 into the same process. Both Python libs are (by default) compiled with the two-level namespace option. Both libs are also loaded with the RTLD_GLOBAL flag via dlopen(..), so the symbols with the same name are supposed not to interfere with each other, since the two modules have different names (python27 and python36).
示例:
#include <{...}/include/python2.7/Python.h>
int main(int argc, const char * argv[])
{
auto* py3 = dlopen(".../python36", RTLD_GLOBAL | RTLD_NOW);
if (py3 == nullptr)
return 0;
auto* py2 = dlopen(".../python27", RTLD_GLOBAL | RTLD_NOW);
if (py2 == nullptr)
return 0;
auto* init = ((decltype(Py_Initialize)*)dlsym(py2, "Py_Initialize"));
if (init)
{
init();
}
return 0;
}
问题是，在python2导入/path/to/python2/lib/lib-dynload/_locale.so之后，会调用python3中的函数PyModule_GetDict.这是为什么?怎么会这样二级名称空间不应该阻止这种情况吗?
The problem is, after python2 imports /path/to/python2/lib/lib-dynload/_locale.so, the function PyModule_GetDict from python3 gets called. Why is that? How can that happen? Shouldn't the two-level namespace prevent that?
P.S. lib-dynload是一个目录，带有用于macOS上Python的其他C模块.我验证了python2环境中正确的_local.so lib已加载.
P.S. lib-dynload is a directory with additional C-modules for Python on macOS. I verified that the correct _local.so lib from the python2 environment gets loaded.
经过一些实验，我发现第一个加载的python lib的符号始终获得较高的优先级，不确定这是否用于第一个加载的lib或仍然是未定义的行为".
After doing some experiments, I saw that the symbols of the first loaded python lib always get the higher precedence, not sure though if this is intended for first loaded libs or still 'undefined behaviour land'.
调用python27的Py_Initialize()-成功:
Calling Py_Initialize() of python27 - Success:
1. Loading python27 first
2. Loading python36 second
3. PYTHONHOME to python27
4. cal Py_Initialize() of python27
调用python27的Py_Initialize()-崩溃:
Calling Py_Initialize() of python27 - Crash:
1. Loading python36 first
2. Loading python27 second
3. PYTHONHOME to python27
4. cal Py_Initialize() of python27
相反，我得到相同的结果.
I get the same results the other way around.
调用python36的Py_Initialize()-成功:
Calling Py_Initialize() of python36 - Success:
1. Loading python36 first
2. Loading python27 second
3. PYTHONHOME to python36
4. cal Py_Initialize() of python36
调用python36的Py_Initialize()-崩溃:
Calling Py_Initialize() of python36 - Crash:
1. Loading python27 first
2. Loading python36 second
3. PYTHONHOME to python36
4. cal Py_Initialize() of python36
推荐答案
libpython中的符号(例如libpython2.7.dylib)已正确解析.例如，在上述情况下，我看到PyModule_GetDict()在错误解决的调用之前被调用了155次.
The symbols within libpython (e.g. libpython2.7.dylib) are being resolved correctly. For example, under the above described scenario, I see PyModule_GetDict() get called 155 times before the incorrectly resolved call.
问题在于python本身正在编译共享库，并且它正在使用dlopen()加载它们.您可以通过在运行时设置环境变量PYTHONVERBOSE来看到dlopen()发生的情况:
The problem is that python itself is compiling shared libraries, and it's using dlopen() to load them. You can see the dlopen() happening by setting the environment variable PYTHONVERBOSE when running:
$ PYTHONVERBOSE=1 ./main 2>&1 | grep dlopen
产生:
dlopen(".../lib/python2.7/lib-dynload/_locale.so", 2);
2参数对应于RTL_NOW，但这没什么大不了的.问题是，这个单独的库无法指示应根据libpython2.7.dylib库解析其符号.但是，它确实有几个python符号.尤其是最终导致问题的那一个:
The 2 argument corresponds to RTL_NOW, but that doesn't matter too much. The issue is that this separate library isn't able to indicate that it's symbols should be resolved against the libpython2.7.dylib library. Yet, it does have several python symbols; in particular, this one that ends up causing the problem:
$ nm prefix/lib/python2.7/lib-dynload/_locale.so | grep GetDict
U _PyModule_GetDict
因此，当python dlopen()作为库时，它所能做的就是在没有限定的情况下解析符号.显然，dl功能的语义是根据库的加载顺序来解析此类符号，如您所述.
So, when python dlopen()s the library, all it can do is resolve the symbol without the qualification. Apparently, the semantic of the dl functionality is to resolve such symbols based on the order the libraries are loaded, as you noted.
因此，在加载_locale.so之前，一切工作正常，如以下回溯所示:
So, things work fine until we load _locale.so, as you can see from the following backtrace:
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x50)
* frame #0: 0x00000001003f3fc1 libpython3.6m.dylib`PyErr_FormatV [inlined] PyErr_Restore at errors.c:42 [opt]
frame #1: 0x00000001003f3fb7 libpython3.6m.dylib`PyErr_FormatV [inlined] PyErr_Clear at errors.c:355 [opt]
frame #2: 0x00000001003f3fb7 libpython3.6m.dylib`PyErr_FormatV(exception=0x00000001004cba18, format="%s:%d: bad argument to internal function", vargs=0x00007fff5fbfdcb0) at errors.c:841 [opt]
frame #3: 0x00000001003f2c39 libpython3.6m.dylib`PyErr_Format(exception=<unavailable>, format=<unavailable>) at errors.c:860 [opt]
frame #4: 0x0000000100358220 libpython3.6m.dylib`PyModule_GetDict(m=0x0000000101a5a868) at moduleobject.c:450 [opt]
frame #5: 0x00000001000f491c _locale.so`init_locale at _localemodule.c:703 [opt]
frame #6: 0x00000001018d1176 libpython2.7.dylib`_PyImport_LoadDynamicModule(name="_locale", pathname=".../lib/python2.7/lib-dynload/_locale.so", fp=<unavailable>) at importdl.c:53 [opt]
同样值得注意的是，_locale.so只是第一个失败的库.如果您以某种方式克服了它，那么还有许多其他库可能在.../lib/python2.7/lib-dynload中有类似的问题.
Also worth noting, _locale.so is just the first library to fail. If you got past it somehow, there are quite a few other libraries that potentially will have similar problems in .../lib/python2.7/lib-dynload.
这篇关于macOS上的RTLD_GLOBAL和两个级别的命名空间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
