如何在 C# 中直接将文件流式传输到 S3 而无需先在本地保存 [英] How to stream a file directly to S3 in C# without saving locally first

问题描述

我是 .Net 编程的新手，并且一直在努力解决如何将文件从 C#(DOTNETCORE 2.+)流式传输到 S3 而无需将其保存在本地.

I'm new to .Net programming and have been wrestling with how to stream a file from C# (DOTNETCORE 2.+) to S3 without saving it locally.

用例如下:

能够向 SQL 服务器提交查询并将 GZipped 分隔文件提取到 S3 存储桶.这将需要在 .NETCORE 中编码，因为最终目标是在 Linux Docker 容器中运行它.

Be able to submit a query to SQL server and extract a GZipped, delimited file to an S3 bucket. This will need to be coded in .NETCORE since the ultimate goal is to run it in a Linux Docker Container.

限制条件 - 磁盘空间和内存有限，因此无法在本地保存大文件然后将其移动到 S3.

Constraints - There will be limited disk space and memory, so saving a large file locally then moving it to S3 is not an option.

以下代码是我的 .NETCORE 控制台应用程序中的一个片段，它将 sqldatareader 的数据写入到本地文件的压缩流中:

The following code is a snippet from my .NETCORE console app that writes a sqldatareader's data to a zipped stream to a local file:

        using (SqlConnection conn = new SqlConnection(cs))
        using (SqlCommand cmd = new SqlCommand(sql, conn))
        using (FileStream outFile = File.Create(filepath + filename + fileextension))
        using (GZipStream compress = new GZipStream(outFile, CompressionMode.Compress))
        using (StreamWriter sw = new StreamWriter(compress))
        {
            conn.Open();
            SqlDataReader dr = cmd.ExecuteReader();
            StringBuilder sb = new StringBuilder();
            Object[] row = new Object[dr.FieldCount];

            while (dr.Read())
            {
                dr.GetValues(row);
                sw.WriteLine(String.Join(delimiter, row));
            }
        }

.NETCORE AWSSDK 用于公开 Amazon.S3.IO 类，我可以将其用于 outFile 文件流，但 AWS 决定不将其发布到 .NETCORE 2.+.AWS 说使用 AmazonS3Client 或 TransferUtility.但是，我找不到有关如何在没有本地文件的情况下上传流的示例.

The .NETCORE AWSSDK used to expose an Amazon.S3.IO class that I could have used for the outFile filestream, but AWS decided not to release it to .NETCORE 2.+. AWS says use the AmazonS3Client or TransferUtility. However, I can't find an example on how to upload a stream without a local file.

任何帮助将不胜感激.

谢谢.

推荐答案

我最终创建了一个类，该类在使用 S3 分段上传 API 时返回默认为 10 MB 的流.在该类中，使用了内存流，因此可以控制磁盘和内存.

I wound up creating a class that returned a stream defaulted to 10 MB while using the S3 multipart upload API. In that class, a memory stream is used, so, disk and memory can be controlled.

这篇关于如何在 C# 中直接将文件流式传输到 S3 而无需先在本地保存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！