在计算过程中如何存储数百万的Double？ [英] How to store millions of Double during a calculation?

问题描述

我的引擎在 X 交易中执行了1,000,000次模拟。在每个模拟期间，对于每个交易，可以验证特定条件。在这种情况下，我将值（它是一个 double ）存储到数组中。每个交易都有自己的价值清单（即这些价值是从一个交易到另一个交易的顺序）。

My engine is executing 1,000,000 of simulations on X deals. During each simulation, for each deal, a specific condition may be verified. In this case, I store the value (which is a double) into an array. Each deal will have its own list of values (i.e. these values are indenpendant from one deal to another deal).

在所有模拟结束时，对于每笔交易，我在他的列表< Double> 上运行一个算法来获得一些输出。不幸的是，该算法需要这些值的完整列表，因此，我无法修改我的算法来计算输出即时，即在模拟期间。

At the end of all the simulations, for each deal, I run an algorithm on his List<Double> to get some outputs. Unfortunately, this algorithm requires the complete list of these values, and thus, I am not able to modify my algorithm to calculate the outputs "on the fly", i.e. during the simulations.

在正常条件（即 X 为低，条件证明小于10％的时间），计算结果正确，即使这可能加强。

In "normal" conditions (i.e. X is low, and the condition is verified less than 10% of the time), the calculation ends correctly, even if this may be enhanced.

当我有很多交易（例如 X = 30 ）和几乎所有我的模拟验证我的具体情况（让我们说90％的模拟）。所以只要存储这些值，我需要大约$ code> 900,000 * 30 * 64bits 的内存（约216Mb）。我的未来要求之一是能够运行500万次模拟...

My problem occurs when I have many deals (for example X = 30) and almost all of my simulations verify my specific condition (let say 90% of simulations). So just to store the values, I need about 900,000 * 30 * 64bits of memory (about 216Mb). One of my future requirements is to be able to run 5,000,000 of simulations...

所以我无法继续使用我目前的存储方式。目前，我使用 Map< String，List< Double>> 的简单结构，其中键是元素的ID，而$ code>列表< Double> 值列表。

So I can't continue with my current way of storing the values. For the moment, I used a "simple" structure of Map<String, List<Double>>, where the key is the ID of the element, and List<Double> the list of values.

所以我的问题是如何按顺序增强我的应用程序的这一部分在模拟中减少内存使用量？

So my question is how can I enhance this specific part of my application in order to reduce the memory usage during the simulations?

另外一个重要的注意事项是，对于最终计算，我的列表< Double> （或者我将要使用的任何结构）必须被订购。所以如果我以前的问题的解决方案也提供了一个结构来排序新的插入元素（比如一个 SortedMap ），那真是太棒了！

Also another important note is that for the final calculation, my List<Double> (or whatever structure I will be using) must be ordered. So if the solution to my previous question also provide a structure that order the new inserted element (such as a SortedMap), it will be really great!

我正在使用Java 1.6。

I am using Java 1.6.

编辑1

我的引擎确实执行了一些财务计算，在我的情况下，所有交易都是相关的。这意味着我不能在第一笔交易中运行我的计算，得到输出，清理列表< Double> ，然后移动到第二个交易等等。

My engine is executing some financial calculations indeed, and in my case, all deals are related. This means that I cannot run my calculations on the first deal, get the output, clean the List<Double>, and then move to the second deal, and so on.

当然，作为临时解决方案，我们将增加分配给引擎的内存，但这不是我期待的解决方案;）

Of course, as a temporary solution, we will increase the memory allocated to the engine, but it's not the solution I am expecting ;)

编辑2

关于算法本身。我不能在这里给出确切的算法，但这里有一些提示：

Regarding the algorithm itself. I can't give the exact algorithm here, but here are some hints:

我们必须处理一个排序的列表< Double> 。然后，我将计算一个索引（根据给定的参数计算一个索引，并计算一个索引（ List 本身的大小）。然后，我终于返回此列表的 index-th 值。

We must work on a sorted List<Double>. I will then calculate an index (which is calculated against a given parameter and the size of the List itself). Then, I finally return the index-th value of this List.

public static double algo(double input, List<Double> sortedList) {
    if (someSpecificCases) {
        return 0;
    }
    // Calculate the index value, using input and also size of the sortedList...
    double index = ...;
    // Specific case where I return the first item of my list.
    if (index == 1) {
        return sortedList.get(0);
    }
    // Specific case where I return the last item of my list.
    if (index == sortedList.size()) {
        return sortedList.get(sortedList.size() - 1);
    }
    // Here, I need the index-th value of my list...
    double val = sortedList.get((int) index);
    double finalValue = someBasicCalculations(val);
    return finalValue;
}

我希望它有助于现在有这样的信息...

I hope it will help to have such information now...

编辑3

目前，我不会考虑任何硬件修改（在这里太长而复杂了）（）增加内存的解决方案将会完成，但这只是一个快速的修复。

Currently, I will not consider any hardware modification (too long and complicated here :( ). The solution of increasing the memory will be done, but it's just a quick fix.

I正在考虑使用临时文件的解决方案：直到某个阈值（例如100,000），我的列表< Double> 将新值存储在内存中，当列表< Double> 达到此阈值，我将该列表附加到临时文件（每个交易一个文件）。

I was thinking of a solution that use a temporary file: Until a certain threshold (for example 100,000), my List<Double> stores new values in memory. When the size of List<Double> reaches this threshold, I append this list in the temporary file (one file per deal).

这样的：

public void addNewValue(double v) {
    if (list.size() == 100000) {
        appendListInFile();
        list.clear();
    }
    list.add(v);
}

在整个计算结束时，对于每笔交易，我将从我在内存中以及临时文件中重建完整的列表< Double> 。然后，我运行我的算法。我清理这笔交易的价值，并转到第二个交易（我现在可以做所有的模拟现在完成）。

At the end of the whole calculation, for each deal, I will reconstruct the complete List<Double> from what I have in memory and also in the temporary file. Then, I run my algorithm. I clean the values for this deal, and move to the second deal (I can do that now, as all the simulations are now finished).

你觉得这样的解决方案？你认为这是可以接受的吗？

What do you think of such solution? Do you think it is acceptable?

当然，我会失去一些时间来读写我的值在外部文件中，但我认为这是可以接受的，不是吗？

Of course I will lose some time to read and write my values in an external file, but I think this can be acceptable, no?

推荐答案

你可以使用浮标而不是双打吗？这将节省您100Mb。

Can you get away with using floats instead of doubles? That would save you 100Mb.

这篇关于在计算过程中如何存储数百万的Double？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！