从文件反序列化的数据。性能问题 [英] Deserializing data from file. Performance issue

问题描述

我有100多万条记录从数据库中读取表。它需要3分钟，直到我在内存中的填充物。我希望优化这个过程中，并使用二进制BinaryFormatter的序列化该对象到一个文件。它创建了1/2 GB大小的文件。当我反序列化这个文件回内存对象。这个耗时11分钟！

问题：为什么它的速度更快，以从数据库中读取这些数据而不是从一个文件？是否有可能以某种方式优化反序列化进程？

数据库是在同一台机器我做了这个测试上。无其他进程了CPU时间在这个时候。 CPU有4个内核，并有40 GB的内存。

编辑：code反序列化：

 使用（的FileStream FS =新的FileStream（文件名，FileMode.Open））
    {
        变种BF =新System.Runtime.Serialization.Formatters.Binary.BinaryFormatter（）;
        VAR数据=（MyType的）bf.Deserialize（FS）;
        ...
    }
 

由于二进制序列的工作方式的解决方案

，这是痛苦的缓慢。它注入了很多基于反射的元数据到二进制文件。我撞上了一些相当大的结构，一些测试在几年前，发​​现XMLSerializer的既比二进制序列更小，更快。去图！

在这两种情况下，该序列是通过反射，这是缓慢进行。你可能会考虑自己的序列化机制。

有一次，我创造了我自己的二进制序列化机制（使用文件读/写），并进行快20倍，比XML序列化，它的性能比二进制序列更快。它也是显著较小

您可能要考虑做这样的事情。

I read a table with more than 1 million records from a database. It takes 3 minutes until I have a populated object in memory. I wanted to optimize this process, and serialized this object to a file using binary BinaryFormatter. It created a file with 1/2 GB size. After I deserialized this file back to memory object. This took 11 minutes!

The question: why it's much faster to read all these data from a database than from a file? Is it possible to optimize deserializing process somehow?

Database is on the same machine I did this test. None of other processes took CPU time at this time. CPU has 4 cores and there are 40 GB memory.

Edit: Code for deserializing:

    using (FileStream fs = new FileStream(filename, FileMode.Open))
    {
        var bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
        var data = (MyType)bf.Deserialize(fs);
        ...
    }

解决方案

Because of the way the Binary Serializer works, it is painfully slow. It injects a lot of reflection-based metadata into the binary file. I ran some tests against some rather large structures a few years back and found that the XMLSerializer is both smaller and faster than the binary serializer. Go figure!

In either case, the serialization is done via reflection, which is slow. You might consider your own serialization mechanism.

I once created my own binary serialization mechanism (using file write/read), and it performed 20 times faster than the XML serializer, which performed faster than the binary serializer. It was also significantly smaller.

You might want to consider doing something like that.

这篇关于从文件反序列化的数据。性能问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！