提高访问地图中元素和写入文件的性能 [英] Improve performance of accessing elements in a map and writing to a file

问题描述

我在C ++中有一个映射结构，可容纳大约6000万对。填充地图后，我想将其写入一个普通文件。

I have a map structure in C++ which holds about 60 million pairs. After I populate the map, I would like to write it to a plain file.

目前这段代码很慢，我想知道您对我如何加快写作速度有什么建议吗？

At the moment this code is quite slow and I'm wondering if you have any suggestions on how I can speed up the writing?

我可以1）改进在地图中访问元素的方式吗？还是2）通过加快文件写入速度？

Can I 1) Improve the way in which elements are accessed in the map? OR 2) by speeding up file writing itself?

map<int,std::pair<int, int>> myMap;
//populate map
typedef map<int,std::pair<int, int>>::const_iterator MapIterator;
int cnt=0;

for (MapIterator iter = myMap.begin(); iter != myMap.end(); ++iter)
{     
    ofile<<iter->second.first<<","<<iter->second.second<<"\n";
    //just printing every 1 million edges written.
    cnt++;
    if(cnt%1000000==0)
        cout<<cnt<<endl;
}

请注意，我必须使用保留键顺序的数据结构（即地图）

Note that I have to use a data structure that preserves order of the keys (i.e. a map)

推荐答案

在我的计算机（i7 / 16gb / win7 / vs2013）上，以下示例在about中运行：

On my machine (i7/16gb/win7/vs2013), the example below runs in about:

• 6s（c流/二进制）；


• 22s（c流/文本）；
  
• 9s（c ++流/二进制）和


• 56s（c ++流/文本）。

• 6s (c stream/binary);
• 22s (c stream/text);
• 9s (c++ stream/binary) and
• 56s (c++ stream/text).

在另一台机器上（PentiumD / 4gb / win10 / vs2015），最长时间约为4分钟。

On a different machine (PentiumD/4gb/win10/vs2015), the longest time was about 4 minutes.

#include <iostream>
#include <cstdint>
#include <map>
#include <chrono>

int main()
{
  typedef std::pair<uint32_t, uint32_t> C_Value;
  typedef std::map< uint32_t, C_Value > C_Map;

  //
  C_Map m;
  const uint32_t maxsize = 50000000;
  try
  {
    for ( int i = 0; i < maxsize; ++i )
      m[i] = C_Value( i, i );
  }
  catch ( ... )
  {
    std::cout << "too many elements\n";
    return -1;
  }

  //
  std::cout << "writing " << m.size() << " elements... ";
  auto t_start = std::chrono::high_resolution_clock::now();

#if 1
  //
  FILE* f = fopen( "test.bin", "wb" );
  if ( ! f )
  {
    std::cout << "could not open file\n";
    return -2;
  }

  //
  for ( auto e : m )
  {
    fwrite( &e.second.first, sizeof e.second.first, 1, f );
    fwrite( &e.second.second, sizeof e.second.second, 1, f );
  }

  //
  fclose( f );
#endif
#if 0
  //
  FILE* f = fopen( "test.bin", "w" );
  if ( ! f )
  {
    std::cout << "could not open file\n";
    return -2;
  }

  //
  for ( auto e : m )
  {
    fprintf( f, "%d, %d\n", e.second.first, e.second.second );
  }

  //
  fclose( f );
#endif
#if 0
  std::ofstream os( "test.bin", std::ios::binary );
  if ( ! os )
  {
    std::cout << "could not open file\n";
    return -2;
  }

  //
  for ( auto e : m )
  {
    os.write( (const char*)&e.second.first, sizeof e.second.first );
    os.write( (const char*)&e.second.second, sizeof e.second.second );
  }

  //
  os.close();
#endif
#if 0
  std::ofstream os( "test.bin" );
  if ( ! os )
  {
    std::cout << "could not open file\n";
    return -2;
  }

  //
  for ( auto e : m )
  {
    os << e.second.first << ',' << e.second.second << '\n';
  }

  //
  os.close();
#endif

  //
  auto t_end = std::chrono::high_resolution_clock::now();
  std::cout << "done in [ms]: " << std::chrono::duration<double, std::milli>( t_end-t_start ).count() << std::endl;

  return 0;
}

这篇关于提高访问地图中元素和写入文件的性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！