我无法在内存中加载我的训练描述符 [英] I cannot load my training descriptors in memory

问题描述

我正在研究Bag of Visual words模块。我从984个训练图像中提取SIFT描述符，并将它们作为.yaml文件存储在磁盘上。我想知道为什么我的描述符文件.yaml太大了，它的大小为923MB，尽管我整个训练图像的大小只有146MB。我的问题是我无法加载我的描述符文件，这是一个矩阵在内存中有140万行和128列，看起来在opencv alloc.cpp中使用的阈值阻止我分配超过730MB。根据训练图像大小是146MB，描述符文件.yaml太大是不正常的？

解决方案

请看我对这个问题的评论 - 还不够信息。



第二个想法，我可以给你一个非常一般的想法。是的，您的数据结构表示训练数据远远超过数据，太大而无法保存在内存中。一种常见且通用的方法是为您现在拥有的数据实现相同的接口，但通过访问文件来模仿对内存的访问。



创建一个包含一些文件的文件适当的结构（可能与您输入的内容不同，您需要将其消化到合适的结构中），打开它以进行只读非共享访问，并在应用程序的生命周期内保持打开状态。您需要根据请求在此文件中查找记录，因此它可能是二进制文件;并且您可能需要在此文件中包含明确定义的记录。是的，我知道YAML数据是分层的，而不是顺序的，但你可以以某种合理的粒度人为地将它细分为记录。



然后，你需要读取此文件一次并记住此文件中记录的文件位置。最有可能的是，它应该是这个文件位置的一些哈希表，能够通过几个键之一执行快速搜索（然后可能是几个哈希表）。请参阅： http://en.wikipedia.org/wiki/Hash_table 。



对于像YAML这样的分层数据，搜索索引可以是YAML树中某些项的位置，例如1-3-203313-9。我希望你明白这个索引应该是二进制结构，而不是文本。此外，您可以在此结构中存储一些元数据信息，例如每个节点中的子项数。



您可以将此哈希表永久存储在另一个要使用的文件中在下一次运行。然后，应保留此文件，直到您更新训练数据。如果您的索引可以是固定大小的二进制结构，则不需要二级索引索引文件的索引。您可以根据文件中此结构的表示大小轻松计算位置。



好​​。现在，下一步。显然，您已经拥有一些软件界面，可以快速访问您的YAML结构，或者其他任何占用太多空间的东西。它可以是访问树节点或任何东西的常用接口。执行以下操作：将此接口的实现替换为包含对表示为文件的相同结构的访问权的另一个实现。它将帮助您保持其余代码的完整。







如果有问题的数据结构是 Matrix ，索引的这个问题变得更简单，更简单。请在对问题的评论中看到我对此的评论。



基本上，任何项目都只用两个索引编制索引：Y和X（行，列）。即使每个矩阵元素在文件中呈现时采用不同的大小，这也不是问题：它可以是索引表，它是文件位置的等级3的数组。因此，您可以将此位置表（索引表）作为一个附加结构，用于通过Y-X索引快速搜索较大文件中的位置。这非常非常简单。

-SA

您的描述符文件也是如此因为SIFT描述符具有双重作为变量，并且在将它们存储在光盘上时不进行压缩，而图像主要被压缩并逐个读入内存。每个图像平均生成2K特征，因此特征数据库必须很大。如果你想创建一个可扩展的系统，内存问题通常是一个非常难的问题，因此你必须以某种方式将矩阵拆分成更小的矩阵并将它们分别存储在磁盘上。然后，您将需要一个搜索算法，该算法将在运行时从场景中自动加载正确的矩阵。这种算法应该是速度近似的，例如基于修改的最佳仓优先搜索算法的算法。您还可以针对此类问题探索局部敏感哈希。请参见 http://en.wikipedia.org/wiki/Locality-sensitive_hashing [ ^ ]



希望这个帮助

I'm working on Bag of Visual words module .I extract SIFT descriptors from 984 Training images,and stored them on disk as .yaml file . I wonder why my descriptors file .yaml is too huge ,it's size 923MB ,although the size of my whole training images is just 146MB . My problem that i cannot load my descriptors file which is a matrix has 1.4 million rows and 128 cols in memory it's look there's a threshold used in the opencv alloc.cpp prevent me from allocate more than 730MB . is it normal that i the descriptors file .yaml is too huge according to the training image size which is 146MB ?

解决方案

Please see my comment to the question — not enough information.

On second thought, I can give you a very general idea. Yes, your data structure represented training data is by far over-populated with data, too big to hold in memory. One common and universal approach is to implement the same interface to this data you have right now, but mimic access to memory through access to file.

Create a file with some appropriate structure (which can be different from what you have in input, you will need to digest it onto suitable structure), open it for read-only non-shared access and keep open during the lifetime of your application. You will need to seek records in this file on request, so it might be binary; and you may need to have well-defined records in this file. Yes, I understand that YAML data is hierarchical, not sequential, but you can somehow artificially subdivide it into records at some reasonable level of granularity.

Then, you need to read this file once and remember file positions of records in this file. Most likely, it should be some hash table of this file positions with the ability to perform quick search by one of several keys (then if could be several hash tables). Please see: http://en.wikipedia.org/wiki/Hash_table.

In case of hierarchical data like YAML, the search index could be the place of some item in the YAML tree, such as 1-3-203313-9. I hope you understand that this index should be binary structure, not text. Also, you can store some metadata information in this structure, such as number of children in each node.

You can permanently store this hash table in another file to be used on next run. Then this file should be kept until you update your training data. If your index can be binary structure of fixed size, you would not need the secondary index, "the index of the index file". You can easily calculate the position based on the size of the representation of this structure in the file.

Good. Now, a next step. Apparently, you already have some software interface providing you fast access to your YAML structure, or anything else taking too much space. It could be a usual interface for access to a tree node or anything. Do the following: replace the implementation of this interface with another implementation wrapping your access to the same structure represented as a file. It will help you to keep the rest of the code intact.

[EDIT]

If the data structure in question is Matrix, this problem with indexing becomes even simpler, much simpler. Please see my comment on this in the comments to the question.

Basically, any item is indexed by just two indices: Y and X (row, column). Even if each matrix element takes different size when presented in a file, this is not a problem: it could be an index table, which is the array of rank 3 of the file positions. So, you can have this position table (index table) as an additional structure used for fast search of the position in the bigger file by Y-X the index. This is very, very simple to do.

—SA

Your descriptor file is too large because SIFT descriptors have double as the variable and no compression is done when storing them on disc while images are mostly compressed and read one by one into memory. Each image on average generates 2K features, thus the feature database must be large. The memory issue is usually a very hard problem if you wish to create a scalable system, and thus you must somehow split the matrix into smaller matrices and store those separately on disk. You will then need a search algorithm that will automatically load the right matrix given a query descriptor from a scene at runtime. Such an algorithm should be approximate for speed such as one based on a modified best-bin first search algorithm. You can also explore locality sensitive hashing for such a problem. See http://en.wikipedia.org/wiki/Locality-sensitive_hashing[^]

Hope this helps.

这篇关于我无法在内存中加载我的训练描述符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！