异步读取/写入同一个流？ [英] Asynchronous Reading/Writing to same stream?

问题描述

有时，我们需要从一个桶复制大文件到另一个AWS S3。只要有可能，我们用CopyRequest来处理此操作全部在AWS（因为需要返回到客户端没有往返）。但有时我们没有做到这一点的选择，因为我们需要2个完全独立的账户之间进行复制，需要一个<一个href="http://docs.amazonwebservices.com/sdkfornet/latest/apidocs/html/T_Amazon_S3_Model_GetObjectRequest.htm"相对=nofollow> GET ，然后一个<一个href="http://docs.amazonwebservices.com/sdkfornet/latest/apidocs/html/T_Amazon_S3_Model_PutObjectRequest.htm"相对=nofollow> PUT 。

Occasionally we need to copy huge files from one bucket to another in AWS S3. Whenever possible we use the CopyRequest to handle this operation all on AWS (since no round trip required back to the client). But sometimes we do not have the option to do this because we need to copy between 2 completely separate accounts which requires a GET and then a PUT.

问题：

1. 响应流从GET返回的是不可搜索，因此它不能被传递到PUT请求，并让无缝地从一个流式传输到其他
2. 响应流复制到中介流（MemoryStream的）使用CopyTo从（）并传送到PUT操作效果很好，但没有规模（大文件会抛出OutOfMemory异常）

因此​​，基本上，我需要一个中介流，我可以读/写在同一时间，基本上我会从响应流中读取一个数据块，并把它写入我的中介流，同时PUT请求被读出的内容和它只是一个无缝直通那种场景。

So basically I need an intermediary stream that I can read/write to at the same time, basically I would read a chunk from the response stream and write it to my intermediary stream, meanwhile the PUT request is reading out the content and its just a seamless pass-thru sort of scenario.

我发现这个职位上的计算器，它似乎有希望在第一，但它仍然抛出内存溢出异常大文件。

I found this post on stackoverflow and it seemed promising at first but it still throws an OutOfMemory exception with large files.

.NET异步数据流的读/写

有谁不得不做一些类似的？你会如何​​应对呢？谢谢advcance

Anyone ever had to do something similar to this? How would you tackle it? Thanks in advcance

推荐答案

目前还不清楚你为什么会想用的MemoryStream 。该 Stream.CopyTo 在.NET 4的方法并不需要使用一个中间流 - 它只是读入一个固定大小的本地缓冲区，然后写该缓冲区输出流，然后读取更多的数据（覆盖缓冲液）等。

It's not clear why you would want to use MemoryStream. The Stream.CopyTo method in .NET 4 doesn't need to use an intermediate stream - it will just read into a local buffer of a fixed size, then write that buffer to the output stream, then read more data (overwriting the buffer) etc.

如果你不使用.NET 4中，可以很容易地实现类似的东西，如：

If you're not using .NET 4, it's easy to implement something similar, e.g.

public static void CopyTo(this Stream input, Stream output)
{
    byte[] buffer = new byte[64 * 1024]; // 64K buffer
    int bytesRead;
    while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write(buffer, 0, bytesRead);
    }
}

这篇关于异步读取/写入同一个流？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！