d3js v5 + Topojson v3关于加入csv& json [英] d3js v5 + Topojson v3 Optimization about joining csv & json

问题描述

为了制作地图，我需要直接在代码中将一些值从csv导入到 中. 为了加载json和csv文件，我对Promise对象使用了异步操作，并且使用了两个循环和一个公用键在json文件上添加新属性.

In order to make maps, I need to import some values from csv to json directly in the code. For loading json and csv files, I use an asynchronous operation with Promise object and I use two loops and a common key to add new properties on json file.

for (var i=0; i< fr[1].length;i++){
        var csvId = fr[1][i].codgeo;
        var csvValue1 = parseFloat(fr[1][i].value1);
        var csvValue0 = parseFloat(fr[1][i].value0);
        for (var j=0; j<topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features.length;j++){
          var jsonId = topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features[j].properties.codgeo;
          if (csvId === jsonId) {
            topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features[j].properties.value1 = csvValue1;
            topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features[j].properties.value0 = csvValue0;
            break;

一切正常，但是在网络上显示地图需要时间. 有没有一种方法可以优化地图的加载时间?

Everything is working but show up the map on the web takes time. Is there a way to optimize the loading time of the map ?

以下是我的代码示例: https://plnkr.co/edit/ccwIQzlefAbd53qnjCX9 ?p =预览

Here is a sample of my code : https://plnkr.co/edit/ccwIQzlefAbd53qnjCX9?p=preview

推荐答案

我拿了你的plunkr并为其添加了一些计时点，运行了很多次并获得了一些有关脚本花费时间的数据:

I took your plunkr and added some timing points to it, ran it a bunch of times and got some data on where your script takes its time:

这是带有日志记录的阻止.

我非常确定我所居住的带宽低于平均水平，并且存在大量可变性；文件加载时间对我来说显示了很大的可变性，最小为500毫秒，最大为1800毫秒，其他所有内容都是一致的

让我们仔细看看数据处理阶段，该阶段包括在您的问题中:

Let's take a closer look a the data manipulation stage, which you include in your question:

for (var i=0; i< fr[1].length;i++){
        var csvId = fr[1][i].codgeo;
        var csvValue1 = parseFloat(fr[1][i].value1);
        var csvValue0 = parseFloat(fr[1][i].value0);
        for (var j=0; j<topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features.length;j++){
          var jsonId = topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features[j].properties.codgeo;
          if (csvId === jsonId) {
            topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features[j].properties.value1 = csvValue1;
            topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8).features[j].properties.value0 = csvValue0;
            break;

按我的计算，嵌套的for语句运行大约5,151次.语句的父级运行101.在您的数据固定后，这些值不应更改.为什么这些周期要花这么长时间?因为您每次都调用topojson.feature()进行迭代:

The nested for statement runs approximately 5,151 times by my count. The parent for statement runs 101. These shouldn't change as your data is fixed. Why do these cycles take so long? Because you are calling topojson.feature() every for iteration:

如果我隔离这行:

topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8)

我们可以看到，这实际上只需要几毫秒.

We can see that this actually takes a few milliseconds alone.

Topojson.feature

Topojson.feature

返回指定的GeoJSON Feature或FeatureCollection 给定拓扑中的对象.如果指定的对象是 GeometryCollection，一个FeatureCollection返回，并且每个几何 集合中的映射到功能.否则，功能是 回来.返回的功能是源对象的浅表副本: 它们可能会共享标识符，边界框，属性和 坐标. (来自文档).

Returns the GeoJSON Feature or FeatureCollection for the specified object in the given topology. If the specified object is a GeometryCollection, a FeatureCollection is returned, and each geometry in the collection is mapped to a Feature. Otherwise, a Feature is returned. The returned feature is a shallow copy of the source object: they may share identifiers, bounding boxes, properties and coordinates. (from the docs).

因此，每次使用topojson.feature时，我们实际上都是将topojson转换为geojson.我们不需要在for循环中执行此操作.让我们做一次:

So, everytime we use topojson.feature we are essentially converting the topojson to geojson. We don't need to do this in the for loop. Let's do that once:

  var featureCollection = topojson.feature(fr[0],fr[0].objects.dep_GEN_WGS84_UTF8);

  //Merge csv & json
  //Add properties from csv to json)
 for (var i=0; i< fr[1].length;i++){
    var csvId = fr[1][i].codgeo;
    var csvValue1 = parseFloat(fr[1][i].value1);
    var csvValue0 = parseFloat(fr[1][i].value0);
    for (var j=0; j<featureCollection.features.length;j++){
      var jsonId = featureCollection.features[j].properties.codgeo;
      if (csvId === jsonId) {
        featureCollection.features[j].properties.value1 = csvValue1;
        featureCollection.features[j].properties.value0 = csvValue0;
        break;
      }
    }
  }

当然，我们也必须更新渲染的代码部分以使用featureCollection变量，而不是topojson

Of course, we have to update the portion of code that renders to use the featureCollection variable too, rather than the topojson

现在让我们看一下时间:

Let's take a look at timing now:

这是基于上述内容的更新的 bl.ock ，还带有计时点.

Here's an updated bl.ock based on the one above, also with timing points.

不，我没有忘记包括一次操纵时间，对我来说平均只有1.5毫秒.是的，我的带宽有所变化-但无论外界因素如何，花费在其他操作上的时间都应该明显减少

进一步的增强功能

几何的预投影，请参见此问题/答案.

Preprojection of geometry, see this question/answer.

简化几何形状，请参见 mapshaper.org (尽管我相信您已经做到了).

Simplification of geometry, see mapshaper.org (though I believe you have already done this).

从csv或topojson中删除不必要的属性-您是否真的在使用topojson中的填充字段，是否需要在topojson中同时使用libgeo和libgeo_m(例如:"libgeo":"Puy-de-Dôme","libgeo_m":"PUY-DE-DÔME")?

Removal of non-necessary attributes from csv or topojson - are you really using the population field in the topojson, do you need both libgeo and libgeo_m in the topojson (eg: "libgeo":"Puy-de-Dôme","libgeo_m":"PUY-DE-DÔME")?

这篇关于d3js v5 + Topojson v3关于加入csv& json的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！