如何将非常大的CSV数据集加载到d3中 [英] How to load very large CSV dataset into d3

问题描述

正如标题所示，我有一个CSV文件（约250mb和700k行），我无法加载到d3中。我尝试按照我通常为csv文件的方式加载它，但没有运气。目前，它没有出错，我在控制台中得到一个空的数据数组。不确定文件是否太大或我加载错误。将不胜感激任何帮助。谢谢。

As the title suggests, I have a CSV file (~250mb and 700k rows) which I am not able to load into d3. I tried to load it the way I usually do for csv files but no luck. Currently, it does not error out and I get an empty data array in the console. Not sure if the file is too big or I'm loading it incorrectly. Will appreciate any help on this. Thanks.

            var dataset;
            d3.csv("Consumer_Complaints.csv", function (error, data) {
                if (error) {  
                    console.log(error);  
                } else {
                    console.log("file load successful?");
                    console.log(data);   

                    dataset = data;
                }
            });

推荐答案

这与D3无关，但是与JavaScript无关一般来说。 D3对可以加载和解析的文件大小没有限制。

This has nothing to do with D3, but with JavaScript in general. D3 imposes no limit on the size of the file it can load and parse.

在客户端端运行Javascript（有一些例外）。这意味着您的代码必须下载（如果在不同的服务器中）所有巨大的 CSV文件，并且不仅如此，它还必须解析大量对象中的数十万行。这太过分了。

Javascript runs (with some exceptions) at the client side. That means your code has to download (if in a different server) all that huge CSV file and, not only that, it has to parse those hundreds of thousands of rows in a huge array of objects. It's simply too much.

因此，常识告诉我们要考虑：

So, common sense tells us to think about:

• 用户的连接速度


• 用户的处理能力


• 用户耐心等待空白屏幕几分钟，等待数据下载/ parsed。

这是一个加载巨大CSV文件的演示（来自 data.gov 网站），你可以在控制台中看到金额数据加载。我还放了一个 console.time 来显示下载和解析文件所需的总时间（如果你有耐心等到最后，我没有）：

This is a demo that loads a huge CSV file (from the data.gov site), you can see in the console the amount of data loaded. I also put a console.time to show the total time needed to download and parse the file (if you have patience to wait until the end, I hadn't):

console.time("totalTime:");
d3.csv("https://data.consumerfinance.gov/api/views/s6ew-h6mp/rows.csv")
    .on("progress", function(evt) {
        console.log("Amount loaded: " + evt.loaded)
    })
    .get(function(data) {
        console.timeEnd("totalTime:");
    });

<script src="https://d3js.org/d3.v4.min.js"></script>

这篇关于如何将非常大的CSV数据集加载到d3中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！