在Web API响应中将zip文件添加为Content，下载时文件大小加倍 [英] Adding zip file as Content in Web API response doubling file size on download

问题描述

我正在将zip文件保存到AWS S3存储桶中.我现在正在尝试创建一个C#.NET API，该API将允许我从存储桶中下载指定的密钥，并将其保存到Content密钥中的HttpResponseMessage中.

我已参考以下问题来设置我对zip文件的答复:

我从API返回的响应如下:

相关代码如下:

  [HttpGet，Route("GetFileFromS3Bucket")]公共HttpResponseMessage GetFileFromS3Bucket(字符串keyName){HttpResponseMessage response = new HttpResponseMessage();字符串bucketName ="myBucket";RegionEndpoint bucketRegion = RegionEndpoint.ARegion;IAmazonS3 s3Client；s3Client =新的AmazonS3Client(bucketRegion);尝试{var fileTransferUtility =新的TransferUtility(s3Client);var stream = fileTransferUtility.OpenStream(bucketName，keyName);response.Content =新的StreamContent(stream);response.Content.Headers.ContentDisposition =新的System.Net.Http.Headers.ContentDispositionHeaderValue(附件")；response.Content.Headers.ContentDisposition.FileName = keyName +".zip";response.Content.Headers.ContentType =新的System.Net.Http.Headers.MediaTypeHeaderValue("application/zip")；response.StatusCode = HttpStatusCode.OK;}捕获(异常e){response.Content = new StringContent(出问题了，错误:" + e.Message);response.StatusCode = HttpStatusCode.InternalServerError;}返回响应；} 

故障排除结果:

• 基于S3中的内容，来自Web API的文件的大小几乎是预期大小的两倍.这在不同文件中是一致的
• 将存储桶更改为可公开访问无济于事(将设置恢复为禁止公共访问)
• 将文件类型更改为XML不会显示格式正确的错误(有建议，如果S3提供了错误，您可能会收到XML响应)
• 将S3流保存到文件中，然后直接保存到文件中可以得到正确的文件大小.似乎可以肯定地说来自S3的流不是问题

看来，HTTPResponseMessage处理zip文件的方式存在问题.我不确定它是真正在服务器端，还是在客户机上以解析数据，而Swagger根本无法做到这一点.任何帮助将不胜感激.

更新1 我不认为此字符串是Base64编码的，因为我将流转换为字符串后得到的结果如下:

我用两行更新了代码示例，其中显示了从流到字符串的转换.

更新2 我已经确认问题在于响应如何处理流或响应本身中的某些内容.从S3下载文件流并将其保存到本地计算机上的新文件中，导致有效文件已按预期打开.

更新3 链接到带有测试文件的GDrive文件夹:

文件名两侧的下划线很奇怪.

我正在运行以下相关软件包:

更新4 我在各种文件中找到了以下UTF8参考:

文件:configuration91.svcinfo

我在项目中的任何地方都找不到关于'responseEncoding'的内容.

解决方案

我要提出答案，因为您所发生的事情是非正统的.我在很多事情上都使用了S3，过去所做的一切都没有问题.为了确保我模仿您在做什么，我复制了您的代码:

  [HttpGet，Route(" GetFileFromS3Bucket/{keyName}"))]公共HttpResponseMessage GetFileFromS3Bucket(字符串keyName){字符串bucketName =" testzipfilesagain" ;;字符串awsAccessKey ="AKIAJ ******** A3QHOUA";字符串awsSecretKey ="IYUJ9Gy2wFCQ ************ dCq5suFS"；IAmazonS3客户端=新的AmazonS3Client(awsAccessKey，awsSecretKey，RegionEndpoint.USEast1);var fileTransferUtility =新的TransferUtility(客户端);var stream = fileTransferUtility.OpenStream(bucketName，"md5.zip");var resp = new HttpResponseMessage();resp.Content =新的StreamContent(stream);resp.Content.Headers.ContentDisposition =新的System.Net.Http.Headers.ContentDispositionHeaderValue(附件")；resp.Content.Headers.ContentDisposition.FileName = keyName +".zip";resp.Content.Headers.ContentType =新的System.Net.Http.Headers.MediaTypeHeaderValue("application/zip")；resp.StatusCode = HttpStatusCode.OK;回报} 

这些是我已安装的软件包:

 < ItemGroup>< PackageReference Include ="AWSSDK.S3"；版本="3.3.111.37"；/>< PackageReference Include ="Microsoft.AspNetCore.Mvc.WebApiCompatShim"；版本="2.2.0&"/>< PackageReference Include ="Swashbuckle.AspNetCore"；版本="5.5.1"./></ItemGroup> 

一切运行正常.

尝试对代码进行故障排除将是徒劳的，因为它可以正常工作，但是您的环境存在问题.

因此，这不是您的问题的答案，而是关于如何尝试解决当前问题并超越此问题的答案.

1. 确保您的nuget包是最新的
2. 您是否在管道中注入了任何中间件?如果是这样，什么呢?
3. 发布您的 startup.cs -也许您的 Configure 例程中出现了问题.
4. 您可以启动一个全新的项目并尝试其中的代码吗?
5. 您可以尝试使用5KB的小型zip文件并将原始文件和损坏的文件张贴出来，以便我们查看吗?

我很想深入了解这个问题，因为我真的很想解决这些类型的问题.

---

编辑1

因此，我查看了zip文件，它们已通过UTF8编码过程运行.因此，如果您获取原始的zip文件，并在其上运行以下代码:

  var goodBytes = File.ReadAllBytes(将上传到S3.zip的某些测试")；var badBytes = File.ReadAllBytes("_进行一些测试以上传到S3.zip.zip_")；File.WriteAllText(上传到S3.zip.utf8的一些测试"，Encoding.UTF8.GetString(goodBytes))；var utf8EncodedGoodBytes = File.ReadAllBytes(上传到S3.zip.utf8的一些测试")；var same = badBytes.SequenceEqual(utf8EncodedGoodBytes); 

结果是:

我将做一些研究，找出可能导致您的流变成UTF-8编码的原因.您的配置中是否有看起来像这样的东西?您能在整个解决方案中搜索与"utf"相似的内容吗?或"utf8"或"utf-8"?

I am saving zip files to an AWS S3 bucket. I am now trying to create a C# .NET API that will allow me to download a specified key from the bucket and save it to a HttpResponseMessage in the Content key.

I've referred to the following question to set up my response for zip files: How to send a zip file from Web API 2 HttpGet

I have modified the code in the previous question so that it instead reads from a TransferUtility stream.

Problem is I am coming into an error when trying to extract or view the file that looks like the following:

The response I am getting back from the API looks like:

The relevant code looks like:

[HttpGet, Route("GetFileFromS3Bucket")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
    HttpResponseMessage response = new HttpResponseMessage();
    string bucketName = "myBucket";
    RegionEndpoint bucketRegion = RegionEndpoint.ARegion;
    IAmazonS3 s3Client;
    s3Client = new AmazonS3Client(bucketRegion);

    try
    {
        var fileTransferUtility = new TransferUtility(s3Client);
        var stream = fileTransferUtility.OpenStream(bucketName, keyName);
        response.Content = new StreamContent(stream);
        response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
        response.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
        response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
        response.StatusCode = HttpStatusCode.OK;
    }
    catch (Exception e)
    {
        response.Content = new StringContent("Something went wrong, error: " + e.Message);
        response.StatusCode = HttpStatusCode.InternalServerError;
    }

    return response;
}

Results of troubleshooting:

• The file from the Web API comes out with nearly double the expected size based on what is in S3. This is consistent across different files
• Changing the bucket to be publicly accessible did not help (setting since reverted to not allowing public access)
• Changing the file type to XML did not display a nicely formatted error (there was a suggestion that you may receive an XML response if an error was provided from S3)
• Saving the S3 stream to a file and then saving directly to a file resulted in the correct file size. Seems safe to say the stream from S3 is not the problem

It appears that there ia a problem with the way the HTTPResponseMessage is handling the zip file. I'm unsure of whether it is actually on the server side, or whether it is on the client to parse the data and Swagger is simply incapable of doing that. Any help would be greatly appreciated.

Update 1 I do not believe this string is Base64 encoded as the result I got from converting the stream to a string is the following:

I've updated the code sample with the two lines showing the conversion from a stream to string.

Update 2 I've confirmed the issue is with how the response is handling the stream, or something in the response itself. Downloading the file stream from S3 and saving to a new file on the local computer resulted in a valid file that opened as expected.

Update 3 Link to GDrive folder with testing files: https://drive.google.com/drive/folders/1q_N3NTHz5E_nebtBQJHor3HfqUZWhGgd?usp=sharing I unfortunately can't provide access to the original file as it contains sensitive data. The provided files are still causing the same problem however. Interesting to note that the test file came out looking like:

The underscores on either side of the filename are quite strange.

I am running the following relevant packages:

Update 4 I've found the following UTF8 references in various files:

File: configuration91.svcinfo

I could not find anything that said anything about 'responseEncoding' anywhere in the project.

解决方案

I am going to throw an answer up, because what's happening to you is unorthodox. I use S3 for many things and have done what you are doing with no problems in the past. To ensure that I am mimicking what you are doing, I duplicated your code:

[HttpGet, Route("GetFileFromS3Bucket/{keyName}")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
    string bucketName = "testzipfilesagain";
    string awsAccessKey = "AKIAJ********A3QHOUA";
    string awsSecretKey = "IYUJ9Gy2wFCQ************dCq5suFS";

    IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1);

    var fileTransferUtility = new TransferUtility(client);
    var stream = fileTransferUtility.OpenStream(bucketName, "md5.zip");

    var resp = new HttpResponseMessage();

    resp.Content = new StreamContent(stream);
    resp.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
    resp.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
    resp.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
    resp.StatusCode = HttpStatusCode.OK;

    return resp;
}

These are the packages I have installed:

  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="3.3.111.37" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.WebApiCompatShim" Version="2.2.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="5.5.1" />
  </ItemGroup>

Everything runs perfectly well.

Trying to troubleshoot your code is going to be fruitless because it works perfectly fine, but there is something wrong with your environment.

So this isn't an answer to your question, but a answer to how you can try to solve the issue at hand and get past this.

1. Make sure your nuget packages are up to date
2. Do you have any middleware injected in your pipeline? If so, what?
3. Post your startup.cs -- maybe something is out of order in your Configure routine.
4. Could you start a brand new project and try your code in that?
5. Can you try a small 5KB zip file and post the original and the corrupt so we can look?

I would love to get to the bottom of this as I really like to solve these types of problems.

---

EDIT 1

So I looked at the zip files and they have been run through a UTF8 encoding process. So, if you take your original zip file, and run this code on it:

    var goodBytes = File.ReadAllBytes("Some test to upload to S3.zip");
    var badBytes = File.ReadAllBytes("_Some test to upload to S3.zip.zip_");

    File.WriteAllText("Some test to upload to S3.zip.utf8", Encoding.UTF8.GetString(goodBytes));
    var utf8EncodedGoodBytes = File.ReadAllBytes("Some test to upload to S3.zip.utf8");

    var identical = badBytes.SequenceEqual(utf8EncodedGoodBytes);

It the results are:

I am going to do some research and figure out what could be causing your stream to become UTF-8 encoded. Is there anything in your config that looks like this? Can you search your entire solution for anything that resembles "utf" or "utf8" or "utf-8"?

这篇关于在Web API响应中将zip文件添加为Content，下载时文件大小加倍的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！