提高流式传输大(1-10 GB)文件的速度 .Net Core [英] Increase Speed for Streaming Large(1-10 gb) files .Net Core

问题描述

我正在尝试使用 multipartform-data 通过我的 API 上传 *.iso 文件并将它们流式传输到本地文件夹中.我使用了 Stream.CopyAsync(destinationStream) 并且它运行缓慢，但还不错.但现在我需要报告进展.所以我使用了自定义 CopyTOAsync 并为其添加了进度报告.但是该方法非常慢(根本无法接受)，即使与 Stream::CopyToASync 相比也是如此.

I'm trying to upload *.iso files via my API using multipartform-data and stream them into local folder. I used Stream.CopyAsync(destinationStream) and it worked slow, but not too bad. But now I need to report progress. So I used custom CopyTOAsync and added a progress report to it. But the method is very slow(not acceptable at all), even compared to Stream::CopyToASync.

 public async Task CopyToAsync(Stream source, Stream destination, long? contentLength, ICommandContext context, int bufferSize = 81920 )
    {
        var buffer = new byte[bufferSize];
        int bytesRead;
        long totalRead = 0;
        while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length)) > 0)
        {
            await destination.WriteAsync(buffer, 0, bytesRead);
            totalRead += bytesRead;
            context.Report(CommandResources.RID_IMAGE_LOADIND, Math.Clamp((uint)((totalRead * 100) / contentLength), 3, 99));
        }
        _logger.Info($"Total read during upload : {totalRead}");
    }

我尝试过的:Stream::CopyToAsync 的默认缓冲区大小为 81920 字节，我首先使用相同的值，然后我尝试将缓冲区大小增加到 104857600 字节 - 没有区别.

What I tried: default buffer size for Stream::CopyToAsync is 81920 bytes, I used the same value first, then I tried to increase buffer size up to 104857600 bytes- no difference.

您对如何提高自定义 CopyToAsync 的性能还有其他想法吗?

Do you have any other ideas on how to improve the performance of custom CopyToAsync?

推荐答案

• 始终使用 ConfigureAwait 和 await 来指定异步延续的线程同步.
  • 根据平台，省略 ConfigureAwait 可能默认与 UI 线程(WPF、WinForms)或任何线程(ASP.NET Core)同步.如果它与您的 Stream 复制操作中的 UI 线程同步，那么性能会急剧下降也就不足为奇了.
  • 如果您在线程同步上下文中运行代码，那么您的 await 语句将被不必要地延迟，因为程序将继续安排到一个可能很忙的线程.
  • Always use ConfigureAwait with await to specify thread synchronization for the async continuation.
    • Depending on the platform, omitting ConfigureAwait may default to synchronizing with the UI thread (WPF, WinForms) or to any thread (ASP.NET Core). If it's synchronizing with the UI thread inside your Stream copy operation then it's no wonder performance takes a nose-dive.
    • If you're running code in a thread-synchronized context, then your await statements will be unnecessarily delayed because the program schedules the continuation to a thread that might be otherwise busy.
    • 这个QA 显示的基准测试表明，异步 IO 需要明显更大的缓冲区才能获得比同步 IO 更好的性能.

    关于您的实际代码 - 只需使用 Stream::CopyToAsync 而不是自己重新实现它.如果您需要进度报告，请考虑子类化 Stream(作为代理包装器).

    With respect to your actual code - just use Stream::CopyToAsync instead of reimplementing it yourself. If you want progress reporting then consider subclassing Stream (as a proxy wrapper) instead.

    以下是我将如何编写您的代码:

    Here's how I would write your code:

    1. 首先，从这个 GitHub Gist 添加我的 ProxyStream 类到您的项目.
    2. 然后将ProxyStream子类化以添加对IProgress的支持:
    3. 确保使用 FileOptions.Asynchronous | 创建任何 FileStream 实例FileOptions.SequentialScan.
    4. 使用CopyToAsync.
    1. First, add my ProxyStream class from this GitHub Gist to your project.
    2. Then subclass ProxyStream to add support for IProgress:
    3. Ensure any FileStream instances are created with FileOptions.Asynchronous | FileOptions.SequentialScan.
    4. Use CopyToAsync.

    public class ProgressProxyStream : ProxyStream
    {
        private readonly IProgress<(Int64 soFar, Int64? total)> progress;
        private readonly Int64? total;
    
        public ProgressProxyStream( Stream stream, IProgress<Int64> progress, Boolean leaveOpen )
            : base( stream, leaveOpen ) 
        {
            this.progress = progress ?? throw new ArgumentNullException(nameof(progress));
            this.total = stream.CanSeek ? stream.Length : (Int64?)null;
        }
    
        public override Task<Int32> ReadAsync( Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken )
        {
            this.progress.Report( ( offset, this.total ) );
            return this.Stream.ReadAsync( buffer, offset, count, cancellationToken );
        }
    }
    
    

    如果性能仍然受到上述 ProgressProxyStream 的影响，那么我愿意打赌瓶颈在 IProgress.Report 回调目标内(我假设它与UI 线程) - 在这种情况下，更好的解决方案是使用(System.Threading.Channels.Channel) 用于转储 ProgressProxyStream(甚至您的 IProgress 实现)在不阻止任何其他 IO 活动的情况下向进度报告.

    If performance still suffers with the above ProgressProxyStream then I'm willing to bet the bottleneck is inside the IProgress.Report callback target (which I assume is synchronised to a UI thread) - in which case a better solution is to use a (System.Threading.Channels.Channel) for the ProgressProxyStream (or even your implementation of IProgress<T>) to dump progress reports to without blocking any other IO activity.

    这篇关于提高流式传输大(1-10 GB)文件的速度 .Net Core的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！