从驱动器读取大文件的部分 [英] Reading parts of large files from drive

问题描述

我与大型文件在C＃中（可高达20％的可用内存-40％），而我只需要将文件的一小部分以每次被加载到内存中（如1-2％文件）。我的想法是使用FileStream将是最好的选择，但IDK的。我将需要给一个起点（字节）和一长度（以字节为单位）和该区域复制到一个字节[]。访问文件可能需要线程之间共享，并且将在文件（非线性存取）随机斑点。我还需要它要快。

I'm working with large files in C# (can be up to 20%-40% of available memory) and I will only need small parts of the files to be loaded into memory at a time (like 1-2% of the file). I was thinking that using a FileStream would be the best option, but idk. I will need to give a starting point (in bytes) and a length (in bytes) and copy that region into a byte[]. Access to the file might need to be shared between threads and will be at random spots in the file (non-linear access). I also need it to be fast.

该项目已经有了不安全方法，可以随意从建议的东西C＃的更危险的副作用。

The project already has unsafe methods, so feel free to suggest things from the more dangerous side of C#

推荐答案

A 的FileStream 将允许您寻求你想要的文件，没有问题的一部分。 。线程之间这是推荐的方法做，在C＃和它的快速

A FileStream will allow you to seek to the portion of the file you want, no problem. It's the recommended way to do it in C#, and it's fast.

分享：你需要创建一个锁来防止其他线程改变而文件流位置你想从中读取。最简单的方式做到这一点：

Sharing between threads: You will need to create a lock to prevent other threads from changing the FileStream position while you're trying to read from it. The simplest way to do this:

//  This really needs to be a member-level variable;
private static readonly object fsLock = new object();

//  Instantiate this in a static constructor or initialize() method
private static FileStream fs = new FileStream("myFile.txt", FileMode.Open);


public string ReadFile(int fileOffset) {

    byte[] buffer = new byte[bufferSize];

    int arrayOffset = 0;

    lock (fsLock) {
        fs.Seek(fileOffset, SeekOrigin.Begin);

        int numBytesRead = fs.Read(bytes, arrayOffset , bufferSize);

        //  Typically used if you're in a loop, reading blocks at a time
        arrayOffset += numBytesRead;
    }

    // Do what you want to the byte array and return it

}

添加在try..catch 必要的陈述和其他代码。不论你访问该的FileStream，把一个锁成员级变量fsLock ......这将继续读取/操纵文件指针，当你想读其他的方法。

Add try..catch statements and other code as necessary. Everywhere you access this FileStream, put a lock on the member-level variable fsLock... this will keep other methods from reading/manipulating the file pointer while you're trying to read.

速度明智的，我想你会发现你被磁盘访问速度，而不是代码的限制。

Speed-wise, I think you'll find you're limited by disk access speeds, not code.

您将不得不考虑所有的关于多线程文件访问...谁intializes /打开文件，谁关闭它，等有很多的地面覆盖的问题。

You'll have to think through all the issues about multi-threaded file access... who intializes/opens the file, who closes it, etc. There's a lot of ground to cover.

这篇关于从驱动器读取大文件的部分的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！