引用对象在数组中的位置? [英] Location of a refernced object in an array?

问题描述

嘿，所有

背景:::::
我正在使用大量自定义神经元类型的(二维)数组用于神经网络，该网络似乎运行良好.当使用较小的Neuron阵列尺寸时，例如
Neuron[,] NodeList = new Neuron[800,600];
它将很好地序列化.

但是，它陷入了某些对象树展平循环中(根据我历史上的某个论坛)，并用尽了内存
Neuron[,] NodeList = new Neuron[10000,9000]
在对Google进行了一些研究之后，我发现我将不得不使用另一个序列化程序或实现我自己的保存数据的方式.
我选择了一个简单的数据编写器，因为这样会更容易





问题(如果要:p，请在此处跳过):::
对于我的自定义数据编写器，我正在将所有需要的数据写到文件中.
在我的Neuron类型中，它存储对另一个Neuron的引用.
如上所述，所有存储的神经元都位于NodeList数组中(包括链接的神经元)

Hey there all

Background:::::
I am working with a massive (2-dimensional) array of a custom neuron type for a Neural Network that seems to work well. When working with the smaller Neuron array sizes e.g.
Neuron[,] NodeList = new Neuron[800,600];
it will serialize fine.

However it gets stuck in some object tree flattening loop(According to some forum in my history) here and runs out of memory
Neuron[,] NodeList = new Neuron[10000,9000]
After a little research on the Google I found that I will have to use another serializer or implement my own way of saving the data.
I opted for a simple data writer as it would be easier





Problem(Skip here if you want :p):::
For my custom data writer i''m writing all the data needed to a file.
In my Neuron type it stores a reference to another Neuron.
All Neurons stored are all in the NodeList array as mentioned (including linked ones)

class Neuron
{
    //Some Old Neuron Info
    float IThreshold = 1;
    float IFiringValue = 1;

    //Link
    float IWeight = 1;
    Neuron NextNeuron;   //This Line HERE
    ...
}

对于我的数据编写者，我要遍历数组中的每个神经元并存储其信息...例如数组中的位置，权重等
我还想存储其链接的神经元在数组中的位置

我想做类似的事情

For my data writer I am going through each neuron in the array and storing its info... like location in the array, weighting etc
I also want to store the location where its linked neuron is in the array

I would like to do something like

int NewX = NodeList[x,y].NextNeuron.ParentArrayIndex(0);
int NewX = NodeList[x,y].NextNeuron.ParentArrayIndex(1);

其中0& 1仅代表数组维数
是否有任何可能或有另一种获取方式，而无需搜索并检查它们是否相等?

任何帮助表示赞赏，...并且，谢谢您:)

Where the 0 & 1 merely represent the array dimension
Is any of this possible or have a different way of getting it without searching through and checking they equal?

Any help appreciated,... and thank you :)

推荐答案

您正在谈论的文件大小为432MB(每个浮点数为4个字节，最小为4个字节)两个索引中的每一个的字节数).那是一个巨大的怪胎''文件.如果是我，我将使用SQLLight并将数据存储在数据库的记录中.它提供了快速检索和对单个神经元记录的索引访问的功能.

You''re talking about a MINIMUM file size of 432MB (4 bytes for each float, and 4 bytes for each of the two indexes). That''s a huge freakin'' file. If it were me, I''d use SQLLight and store the data in records in a database. It affords fast retrieval, and index access to individual neuron records.

哦……这不是那么简单.我记得你以前的问题.让我们看看.

据我了解，您将完整的神经元阵列存储在磁盘上.您没有显示搜索算法，因此很难说出它的优缺点.通常的想法是创建一个数据现金，您可以在运行时将其加载并保存在内存中.在这种情况下，每个神经元可以存储的信息少于每个神经元上的完整数据.问题是:您是否有足够的内存来将现金数据全部存储在内存中?如果没有，则可能是二级缓存…

问题是:每个神经元的持久表示是否在磁盘中占据完全相同的空间?是的，您只需通过每个神经元的索引来计算其位置.如果不是，则缓存数据应索引磁盘上的数组，因此代表神经元的每个缓存元素应包含每个神经元的流位置. (这不是很慢:前一阵子，我基于这个想法开发了一个字典系统:无论字典的大小比字典大多少个数量级，它都比最佳的商业系统快或快.可用内存).

现在，神经元之间的关系(它们如何连接)是真正的数学关系，也就是说，如果您有一组神经元S，则根据定义，该关系是笛卡尔平方S×S的子集. (如果您认为它比这复杂，则仅意味着神经元参与的关系不止一个.)以这种方式引入的关系图始终是有向图的，但方向可以忽略也可以不忽略.您需要确定如何将其存储在磁盘和(缓存)内存中.首先，这取决于这个笛卡尔正方形的稀疏程度.最终，您需要开发一种通过关系访问所有中子邻居的方法.

这只是一些基本思想.他们都需要思考.你觉得呢?



请提供我的其他解决方案，以获取有关如何将存储层与语义层隔离以及如何以透明方式访问神经元和神经元层的详细信息.我还展示了存储层应始终组织成类似于秩1数组的接口，而不必与语义层相同.

—SA

Oh… it can be not so simple. I remember your previous question. Let''s see.

As I understand, you store the full neuron array on disk. You did not show your search algorithm, so it''s hard to say what how good or bad is that. The general idea is to create a data cash which you could load and keep in memory during run time. In this case you can store less information per neuron than the full data on each neuron. The problems is: would you have enough memory to store even the cash data all in memory? If not, it could be two level of cache…

The question is: is the persistent representation of each neuron takes exact same space in disk? It yes, you simply calculate a position of each neuron by its index. If not, you cache data should index the array on disk, so each cache element representing a neuron should contain a stream position for each neuron. (Don''t this it is slow: a while ago I developed a dictionary system based on this idea: it''s as fast or faster then then best commercial systems regardless of the dictionary size which can be orders of magnitude bigger than the available memory).

Now the relationship between neuron (how they are connected) is a real mathematical relationship, that is, if you have a set of neurons S, the relationship is by definition is a subset of the Cartesian Square S×S. (If you think it''s more complex than that, is simply means that a neurons participate in more then one relationship.) The graph of a relationship introduced in this way is always directed, but the direction can be ignored or not. You need to decide how to store it on disk and in (cache) memory. First of all, it depends on how sparse is this Cartesian Square. Ultimately, you need to develop the way to access all neutron neighbors by a relationship.

This is just some basic ideas. They all need some thinking. What do you think?



Please my my other solution for detail on how storage layer should be isolated from semantic layer and how neuron and neuron layers can be accessed in transparent way. I also shows that the storage layers should always be organized a rank-1 array-like interface which does not have to be the same interface as the semantic layer.

—SA

正如我所承诺的，我将在讨论存储和访问神经元数据之后发布我的草图.

这只是您可能不熟悉的文化技巧.
这些技术说明了以下内容:

As I promised, I''m posting my sketch following the discussion of storage and access to the neuron data.

This is just cultural techniques which you may or may not be well familiar with.
These techniques illustrate the following:

1. 将存储层与语义层分离.
2. 您不需要使用数组，而是需要使用索引属性this与网络类似的数组接口.
3. 您不需要rank-2数组进行存储；它实际上是rank-1数组.
4. 如何同时使用rank-1和rank-2(rank-3或其他任何数组)的接口使用相同的数据.请参见GetStorageIndex.
5. 如何开发一个类，以支持多个具有不同签名的索引属性this.做到这一点的唯一方法是使用索引属性this实现接口.
6. 如何限制存储层的可见性.
7. 如何对不同维度的存储空间使用相同的存储模式神经网络.请参阅NeuralNetworkMetadata用于支撑，存储和加载尺寸.

1. Separation of storage layer from semantic layer.
2. You don''t need to work with arrays, instead, you need array-like interface to the network with the use of indexed property this.
3. You do not need rank-2 array for storage; it is essentially the rank-1 array.
4. How to use the same data using the interface of rank-1 and rank-2 (rank-3, or whatever else) array at the same time. See GetStorageIndex.
5. How to develop a class supporting more than one indexed property this with different signature. The only way to do this is implementing interface with the indexed property this.
6. How to limit visibility of the storage layer.
7. How to use identical storage schema for different dimensions of the neural network. See NeuralNetworkMetadata used to support, store and load dimensions.

我希望您的磁盘存储是二进制的吗?出于性能考虑应该如此.另外，应缓冲流.

I hope your disk storage is binary? It should be, for the sake of performance. Also, the stream should be buffered.

using System.IO;
using StreamPosition = System.Int64;
using StorageIndex = System.UInt64;
using NeuronIndex = System.UInt32;
using NeuronLayerIndex = System.UInt32;

class NeuronStorage {
    internal static Neuron Load(Stream stream) { return null; }
    internal void Store(Stream stream) { /*...*/ }
    //...
} //class NeuronStorage

class Neuron : NeuronStorage { /*...*/ }

interface INeuralNetworkStorage {
    Neuron this[StorageIndex index] { get; set; }
} //interface INeuralNetworkStorage

interface INeuralNetwork : INeuralNetworkStorage {
    Neuron this[NeuronLayerIndex layer, NeuronIndex index] { get; set; }
} //interface INeuralNetwork

class NeuralNetworkMetadata {
    internal NeuronIndex NeuronsPerLayer { get { /*...*/ } }
    internal NeuronLayerIndex LayerCount { get { /*...*/ } }
    //...
    internal static NeuralNetworkMetadata Load(Stream stream) { /*...*/ }
    internal void Store(Stream stream) { /*...*/ }
    //...
} //class NeuralNetworkMetadata

class NeuralNetworkStorage : INeuralNetworkStorage {
    public NeuralNetworkStorage(string fileName) { /*...*/ }
    public Neuron this[StorageIndex index] {
        get {
            Stream.Seek(GetNeuronStreamPosition(index), SeekOrigin.Begin);
            return Neuron.Load(Stream);
        }
        set {
            Stream.Seek(GetNeuronStreamPosition(index), SeekOrigin.Begin);
            value.Store(Stream);
        }
    } //this
    //...
    internal protected NeuralNetworkMetadata Metadata;
    StreamPosition GetNeuronStreamPosition(StorageIndex index) { /*...*/ }
    Stream Stream;
} //class NeuralNetworkStorage

class NeuralNetwork : NeuralNetworkStorage, INeuralNetwork {
    public NeuralNetwork(string fileName) : base(fileName) {  /*...*/ }
    public NeuralNetwork(string fileName, NeuralNetworkMetadata metadata) : base(fileName) { /*store meta-data*/ }
    public Neuron this[NeuronIndex index, NeuronLayerIndex layer] {
        get { return this[GetStorageIndex(index, layer)]; }
        set { this[GetStorageIndex(index, layer)] = value; }
    } //this
    StorageIndex GetStorageIndex(NeuronIndex neutron, NeuronLayerIndex layer) {
        return layer * Metadata.NeuronsPerLayer + neutron;
    } //GetStorageIndex
    //...
} //class NeuralNetwork

应该为中子之间的关系指定单独的存储部分.该技术可能取决于几个因素，首先是笛卡尔正方形的稀疏程度，关联的总数有多少，应如何使用.

我希望我提供的所有技术和想法都可以有用.

—SA

The separate section of storage should be designated for relationships between neutron. The technique may depend on several factors, first of an how sparse is the Cartesian Square, how big is the total number of associations, how the should be used.

I hope all the techniques and ideas I provide can be useful.

—SA

这篇关于引用对象在数组中的位置?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！