如何在Linux上超过64位LuaJIT的1gb内存限制? [英] How to get past 1gb memory limit of 64 bit LuaJIT on Linux?

问题描述

概述是我正在制作原型代码以了解我的问题空间，并且遇到了"PANIC:调用Lua API时出现未保护的错误(内存不足)"错误.我正在寻找解决这个问题的方法.

The overview is I am prototyping code to understand my problem space, and I am running into 'PANIC: unprotected error in call to Lua API (not enough memory)' errors. I am looking for ways to get around this limit.

环境的底线是Torch，这是一个在LuaJIT上运行的科学计算框架，而LuaJIT在Lua上运行.我需要Torch，因为我最终想通过GPU上的神经网络解决我的问题，但是要到达那里，我需要对问题有一个很好的表示才能馈入网络.我(被困)在Centos Linux上，我怀疑试图以32位模式从源代码重建所有片段(据报道，这会将LuaJIT内存限制扩展到4gb)，如果它对于所有库.

The environment bottom line is Torch, a scientific computing framework that runs on LuaJIT, and LuaJIT runs on Lua. I need Torch because I eventually want to hammer on my problem with neural nets on a GPU, but to get there I need a good representation of the problem to feed to the nets. I am (stuck) on Centos Linux, and I suspect that trying to rebuild all the pieces from source in 32bit mode (this is reported to extend the LuaJIT memory limit to 4gb) will be a nightmare if it works at all for all of the libraries.

问题空间本身可能并不是特别重要，但是总的来说，我有一些点的数据文件，我先计算它们之间的距离，然后对这些距离进行分箱(即对这些距离进行直方图分析)，以找出最有用的范围.方便地，我可以使用各种垃圾箱和torch.save()创建复杂的Lua表，数出一团糟，然后稍后将其拾起并用不同的归一化进行检查等.因此，在玩了一个月后，我发现这是真正简单而强大.

The problem space itself is probably not particularly relevant, but in overview I have datafiles of points that I calculate distances between and then bin (i.e. make histograms of) these distances to try and work out the most useful ranges. Conveniently I can create complicated Lua tables with various sets of bins and torch.save() the mess of counts out, then pick it up later and inspect with different normalisations etc. -- so after one month of playing I am finding this to be really easy and powerful.

我可以使它在多达3个距离的情况下工作，每个距离有15个垃圾箱(15x15x15加上开销)，但这只能通过添加显式的垃圾收集()调用并为每个数据文件使用fork()/wait()来实现，如果一个(数千个)数据文件仍然超出内存限制并使子进程崩溃，则循环将继续运行.由于每个成功的子进程现在都必须读取，修改和写入当前的箱数计数，因此这会更加痛苦.为此，我最大的文件大小为36mb.我想做得更大(更多的垃圾箱)，并且真的更喜欢将计数保存在我似乎无法访问的15个RAM中.

I can make it work looking at up to 3 distances with 15 bins each (15x15x15 plus overhead), but this only by adding explicit garbagecollection() calls and using fork()/wait() for each datafile so that the outer loop will keep running if one datafile (of several thousand) still blows the memory limit and crashes the child. This gets extra painful as each successful child process now has to read, modify and write the current set of bin counts -- and my largest files for this are currently 36mb. I would like to go larger (more bins), and would really prefer to just hold the counts in the 15 gigs of RAM I can't seem to access.

因此，这是我想到的一些方法；如果您可以确认/否认他们中的任何一个/不会让我超出1GB边界，或者只是提高我在其中的效率，请发表评论.如果您可以提出我未曾想到的另一种方法，请发表评论.

So, here are some paths I have thought of; please do comment if you can confirm/deny that any of them will/won't get me outside of the 1gb boundary, or will just improve my efficiency within it. Please do comment if you can suggest another approach that I have not thought of.

• am我缺少一种启动Lua进程的方法，我可以从中读取任意表吗?毫无疑问，我可以将问题分解成较小的部分，但是从stdio解析返回表(例如从系统调用到另一个Lua脚本)似乎容易出错，并且写入/读取较小的中间文件将占用大量磁盘I/O.

• am I missing a way to fire off a Lua process that I can read an arbitrary table back in from? No doubt I can break my problem into smaller pieces, but parsing a return table from stdio (as from a system call to another Lua script) seems error prone, and writing/reading small intermediate files will be a lot of disk i/o.

我错过了一个存储和访问表高内存模块吗?这似乎是我真正想要的，但尚未找到

am I missing a stash-and-access-table-in-high-memory module ? This seems like what I really want, but not found it yet

可以将FFI C数据结构放在1GB之外吗?似乎不是这种情况，但当然我首先对造成限制的原因缺乏全面的了解.我怀疑这将使我对通用Lua表的效率得到提高，而这已经超出了原型设计的范围了? (除非我为每次更改做一堆编码)

can FFI C data structures be put outside the 1gb? Doesn't seem like that would be the case but certainly I lack a full understanding of what is causing the limit in the first place. I suspect that this will just get me an efficiency improvement over generic Lua tables for the few pieces that have moved beyond prototyping? (unless I do a bunch of coding for each change)

可以肯定的是，我可以使用C语言编写一个扩展程序(Torch似乎支持应该超出限制的网络)，但是我在那里的简短调查却发现了对"lightuserdata"指针的引用-这是否意味着一个更普通的扩展名也不会超出1GB?这似乎也需要进行原型设计，却付出了沉重的开发成本.

Surely I can get out by writing an extension in C (Torch appears to support nets that should go outside of the limit), but my brief investigation there turns up references to 'lightuserdata' pointers -- does this mean that a more normal extension won't get outside 1gb either? This also seems like it has the heavy development cost for what should be a prototyping exercise.

我非常了解C，所以走FFI或扩展路线不会打扰我-但我从经验中知道，以这种方式封装算法既很优雅又很痛苦，有两个地方可以隐藏bug.遍历堆栈表中包含表的数据结构似乎也不是一件好事.在做出这一努力之前，我想确定最终结果确实可以解决我的问题.

I know C well so going the FFI or extension route doesn't bother me - but I know from experience that encapsulating algorithms in this way can be both really elegant and really painful with two places to hide bugs. Working through data structures containing tables within tables on the stack doesn't seem great either. Before I make this effort I would like to be certain that the end result really will solve my problem.

感谢您阅读冗长的帖子.

Thanks for reading the long post.

推荐答案

仅LuaJIT本身分配的对象被限制为前2GB内存.这意味着分配给ffi.new的表，字符串，完整的用户数据(即非lightuserdata)和FFI对象将计入该限制，但分配给malloc，mmap等的对象则不受此限制(不管是由C模块还是FFI调用.

Only object allocated by LuaJIT itself are limited to the first 2GB of memory. This means that tables, strings, full userdata (i.e. not lightuserdata), and FFI objects allocated with ffi.new will count towards the limit, but objects allocated with malloc, mmap, etc. are not subjected to this limit (regardless if called by a C module or the FFI).

使用malloc分配结构的示例:

ffi.cdef[[
    typedef struct { int bar; } foo;
    void* malloc(size_t);
    void free(void*);
]]

local foo_t = ffi.typeof("foo")
local foo_p = ffi.typeof("foo*")

function alloc_foo()
    local obj = ffi.C.malloc(ffi.sizeof(foo_t))
    return ffi.cast(foo_p, obj)
end

function free_foo(obj)
    ffi.C.free(obj)
end

将要在LuaJIT 3.0 IIRC中实现的新GC没有此限制，但是我最近没有听到有关其开发的任何消息.

The new GC to be implemented in LuaJIT 3.0 IIRC will not have this limit, but I haven't heard any news on it's development recently.

来源: http://lua-users.org/lists/lua-l /2012-04/msg00729.html

这篇关于如何在Linux上超过64位LuaJIT的1gb内存限制?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！