通过共享库连接Python和Torch7(Lua) [英] interfacing Python and Torch7(Lua) via shared library

问题描述

我试图在python和lua之间传递数据(数组)，并且我想使用Torch7框架在lua中操纵数据. 我认为这可以最好地通过C来完成，因为python和lua与C进行了接口.另外一些优点是不需要这种方式进行数据复制(仅传递指针)并且速度很快.

I am trying to pass data (arrays) between python and lua and I want to manipulate the data in lua using the Torch7 framework. I figured this can best be done through C, since python and lua interface with C. Also some advantages are that no data copying is needed this way (passing only pointers) and is fast.

我实现了两个程序，一个程序将lua嵌入到c中，另一个程序将python将数据传递给c. 当编译为可执行二进制文件时，它们都可以工作. 但是，当将c to lua程序改为共享库时，该功能将无效.

I implemented two programs, one where lua is embedded in c and one where python passes data to c. They both work when compiled to executable binaries. However when the c to lua program is instead made to be a shared library things don’t work.

详细信息: 我正在使用64位Ubuntu 14.04和12.04. 我正在使用luajit 2.0.2和/usr/local/中安装的lua 5.1 依赖库位于/usr/local/lib中，标头位于/usr/local/include中 我正在使用python 2.7

The details: I’m using 64-bit ubuntu 14.04 and 12.04. I’m using luajit 2.0.2 with lua 5.1 installed in /usr/local/ Dependency libs are in /usr/local/lib and headers are in /usr/local/include I’m using python 2.7

c到lua程序的代码是:

The code for the c to lua program is:

tensor.lua

tensor.lua

require 'torch'

function hi_tensor(t)
   print(‘Hi from lua')
   torch.setdefaulttensortype('torch.FloatTensor')
   print(t)
return t*2
end

cluaf.h

void multiply (float* array, int m, int n, float *result, int m1, int n1);

cluaf.c

#include <stdio.h>
#include <string.h>
#include "lua.h"
#include "lauxlib.h"
#include "lualib.h"
#include "luaT.h"
#include "TH/TH.h"

void multiply (float* array, int m, int n, float *result, int m1, int n1)
{
    lua_State *L = luaL_newstate();
    luaL_openlibs( L );

    // loading the lua file
    if (luaL_loadfile(L, "tensor.lua") || lua_pcall(L, 0, 0, 0))
    {
        printf("error: %s \n", lua_tostring(L, -1));
    }

    // convert the c array to Torch7 specific structure representing a tensor
    THFloatStorage* storage =  THFloatStorage_newWithData(array, m*n);
    THFloatTensor* tensor = THFloatTensor_newWithStorage2d(storage, 0, m, n, n, 1);
    luaT_newmetatable(L, "torch.FloatTensor", NULL, NULL, NULL, NULL);

    // load the lua function hi_tensor
    lua_getglobal(L, "hi_tensor");
    if(!lua_isfunction(L,-1))
    {
        lua_pop(L,1);
    }

    //this pushes data to the stack to be used as a parameter
    //to the hi_tensor function call
    luaT_pushudata(L, (void *)tensor, "torch.FloatTensor");

    // call the lua function hi_tensor
    if (lua_pcall(L, 1, 1, 0) != 0)
    {
        printf("error running function `hi_tensor': %s \n", lua_tostring(L, -1));
    }

    // get results returned from the lua function hi_tensor
    THFloatTensor* z = luaT_toudata(L, -1, "torch.FloatTensor");
    lua_pop(L, 1);
    THFloatStorage *storage_res =  z->storage;
    result = storage_res->data;

    return ;
}

然后进行测试:

luajit -b tensor.lua tensor.o

gcc -w -c -Wall -Wl,-E -fpic cluaf.c -lluajit -lluaT -lTH -lm -ldl -L /usr/local/lib

gcc -shared cluaf.o tensor.o -L/usr/local/lib -lluajit -lluaT -lTH -lm -ldl -Wl,-E -o libcluaf.so

gcc -L. -Wall -o test main.c -lcluaf

./test

输出:

Hi from lua
 1.0000  0.2000
 0.2000  5.3000
[torch.FloatTensor of dimension 2x2]

c result 2.000000 
c result 0.400000 
c result 0.400000 
c result 10.60000

到目前为止，一切都很好. 但是，当我尝试在python中使用共享库时，它就会中断.

So far so good. But when I try to use the shared library in python it breaks.

test.py

from ctypes import byref, cdll, c_int
import ctypes
import numpy as np
import cython

l = cdll.LoadLibrary(‘absolute_path_to_so/libcluaf.so')

a = np.arange(4, dtype=np.float64).reshape((2,2))
b = np.arange(4, dtype=np.float64).reshape((2,2))

l.multiply.argtypes = [ctypes.POINTER(ctypes.c_float), ctypes.c_int, ctypes.c_int,     ctypes.POINTER(ctypes.c_float), ctypes.c_int, ctypes.c_int]
a_list = []
b_list = []

for i in range(a.shape[0]):
    for j in range(a.shape[1]):
            a_list.append(a[i][j])

for i in range(b.shape[0]):
     for j in range(b.shape[1]):
        b_list.append(b[i][j])

arr_a = (ctypes.c_float * len(a_list))()
arr_b = (ctypes.c_float * len(b_list))()

l.multiply(arr_a, ctypes.c_int(2), ctypes.c_int(2), arr_b, ctypes.c_int(2), ctypes.c_int(2))

我跑步:

python test.py

，输出为:

error: error loading module 'libpaths' from file '/usr/local/lib/lua/5.1/libpaths.so':
    /usr/local/lib/lua/5.1/libpaths.so: undefined symbol: lua_gettop

我在网络上的任何地方都搜索了此错误，但是他们建议(1)包括-Wl，-E以导出符号，或者(2)添加对链接所做的依赖关系. (1)我有-Wl，-E，但似乎什么也没做. (2)我已经包含了依赖项(-L/usr/local/lib -lluajit -lluaT -lTH -lm -ldl)

I searched for this error here and everywhere on the web but they either suggest (1) to include -Wl,-E to export symbols or (2) to add dependencies on linking which I did. (1) I have -Wl,-E but it seems to not be doing anything. (2) I have included the dependencies (-L/usr/local/lib -lluajit -lluaT -lTH -lm -ldl)

不是在导入共享库时，而是在调用lua中的"require torch"时，python测试失败.在这种情况下，这与我发现的其他情况也有所不同.

The python test fails not when the shared library is imported but when the ‘require torch’ inside lua is called. That is also the different thing in this case from the other cases I found.

luajit.so定义了符号lua_gettop(通过nm/usr/local/lib/luajit.so可以看到) lua.h定义LUA_API int(lua_gettop)(lua_State * L);

luajit.so defines the symbol lua_gettop (nm /usr/local/lib/luajit.so to see that) lua.h defines LUA_API int (lua_gettop) (lua_State *L);

我猜将c编译为二进制文件时所有工作都是正确的，因为它会在lua.h中找到所有符号 但是使用共享库不会从luajit.so中选择lua_gettop(我不知道为什么).

I guess when compiling c to binary all works because it finds all symbols in lua.h but using the shared library it doesn’t pick lua_gettop from luajit.so (I don’t know why).

www.luajit.org/running.html说: ``在大多数基于ELF的系统(例如Linux)上，您需要在链接应用程序时显式导出全局符号，例如带有:-Wl，-E require()尝试从导出的​​符号(在Windows上的* .exe或lua51.dll中)和package.cpath的共享库中加载嵌入式字节码数据.'

www.luajit.org/running.html says: 'On most ELF-based systems (e.g. Linux) you need to explicitly export the global symbols when linking your application, e.g. with: -Wl,-E require() tries to load embedded bytecode data from exported symbols (in *.exe or lua51.dll on Windows) and from shared libraries in package.cpath.'

package.cpath和package.path是:

package.cpath and package.path are:

./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so

./?.lua;/usr/local/share/luajit-2.0.2/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua

这是nm libcluaf.so返回的内容:

Here is what nm libcluaf.so returns:

00000000002020a0 B __bss_start
00000000002020a0 b completed.6972
                 w __cxa_finalize@@GLIBC_2.2.5
0000000000000a50 t deregister_tm_clones
0000000000000ac0 t __do_global_dtors_aux
0000000000201dd8 t __do_global_dtors_aux_fini_array_entry
0000000000202098 d __dso_handle
0000000000201de8 d _DYNAMIC
00000000002020a0 D _edata
00000000002020a8 B _end
0000000000000d28 T _fini
0000000000000b00 t frame_dummy
0000000000201dd0 t __frame_dummy_init_array_entry
0000000000000ed0 r __FRAME_END__
0000000000202000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
0000000000000918 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000201de0 d __JCR_END__
0000000000201de0 d __JCR_LIST__
                 w _Jv_RegisterClasses
                 U lua_getfield
0000000000000d99 R luaJIT_BC_tensor
                 U luaL_loadfile
                 U luaL_newstate
                 U luaL_openlibs
                 U lua_pcall
                 U lua_settop
                 U luaT_newmetatable
                 U lua_tolstring
                 U luaT_pushudata
                 U luaT_toudata
                 U lua_type
0000000000000b35 T multiply
                 U printf@@GLIBC_2.2.5
0000000000000a80 t register_tm_clones
                 U THFloatStorage_newWithData
                 U THFloatTensor_newWithStorage2d
00000000002020a0 d __TMC_END__

预先感谢

推荐答案

在Linux上，Lua模块不直接链接到Lua库，而是期望找到已经加载的Lua API函数.通常，这是通过使用-Wl,-E链接器标志从解释器中导出它们来完成的.此标志仅适用于可执行文件中的符号，不适用于共享库.对于共享库，存在类似的内容:dlopen函数的RTLD_GLOBAL标志.默认情况下，编译器命令行上列出的所有共享库都使用RTLD_LOCAL加载，但是幸运的是Linux可以重用已经打开的库句柄.因此，您可以:

On Linux Lua modules don't link to the Lua library directly but instead expect to find the Lua API functions already loaded. This is usually done by exporting them from the interpreter using the -Wl,-E linker flag. This flag only works for symbols in executables, not shared libraries. For shared libraries there exists something similar: the RTLD_GLOBAL flag for the dlopen function. By default all shared libraries listed on the compiler command line are loaded using RTLD_LOCAL instead, but fortunately Linux reuses already opened library handles. So you can either:

使用RTLD_GLOBAL 之前预先加载Lua(JIT)库(它会在您加载libcluaf.so时自动发生):

Preload the Lua(JIT) library using RTLD_GLOBAL before it gets loaded automatically (which happens when you load libcluaf.so):

from ctypes import byref, cdll, c_int
import ctypes

lualib = ctypes.CDLL("libluajit-5.1.so", mode=ctypes.RTLD_GLOBAL)
l = cdll.LoadLibrary('absolute_path_to_so/libcluaf.so')
# ...

或随后使用dlopen的RTLD_NOLOAD标志更改Lua(JIT)库句柄的标志.但是，此标志不在POSIX中，您可能必须使用C来这样做.参见例如此处.

Or change the flags of the Lua(JIT) library handle afterwards using the RTLD_NOLOAD flag for dlopen. This flag is not in POSIX though, and you probably have to use C to do so. See e.g. here.

这篇关于通过共享库连接Python和Torch7(Lua)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！