从C#中的HTTPWebResponse.GetResponseStream()上传到S3 [英] Upload to S3 from HTTPWebResponse.GetResponseStream() in c#

问题描述

我正在尝试从HTTP流直接上载到S3，而不先存储在内存中或作为文件.我已经使用Rackspace Cloud Files作为HTTP到HTTP来执行此操作，但是AWS身份验证超出了我的权限，因此我尝试使用SDK.

I am trying to upload from an HTTP stream directly to S3, without storing in memory or as a file first. I am already doing this with Rackspace Cloud Files as HTTP to HTTP, however the AWS authentication is beyond me so am trying to use the SDK.

问题是上传流因以下异常而失败:

The problem is the upload stream is failing with this exception:

"This stream does not support seek operations."

我尝试使用PutObject和TransferUtility.Upload，但都因相同的原因而失败.

I've tried with PutObject and TransferUtility.Upload, both fail with the same thing.

有什么方法可以在流进入流中时将其流到S3中，而不是将整个内容缓冲到MemoryStream或FileStream中?

Is there any way to stream into S3 as the stream comes in, rather than buffering the whole thing to a MemoryStream or FileStream?

或是否有使用HTTPWebRequest对S3请求进行身份验证的良好示例，因此我可以复制对Cloud Files所做的工作吗?

or is there any good examples of doing the authentication into S3 request using HTTPWebRequest, so I can duplicate what I do with Cloud Files?

或 AWSSDK中是否有一个辅助函数来生成授权标头?

or is there a helper function in the AWSSDK for generating the authorization header?

代码:

这是失败的S3部分(包括两种方法以确保完整性):

This is the failing S3 part (both methods included for completeness):

string uri = RSConnection.StorageUrl + "/" + container + "/" + file.SelectSingleNode("name").InnerText;
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
req.Method = "GET";

using (var resp = req.GetResponse() as HttpWebResponse)
{
    using (Stream stream = resp.GetResponseStream())
    {
        Amazon.S3.Transfer.TransferUtility trans = new Amazon.S3.Transfer.TransferUtility(S3Client);
        trans.Upload(stream, config.Element("root").Element("S3BackupBucket").Value, container + file.SelectSingleNode("name").InnerText);

        //Use EITHER the above OR the below

        PutObjectRequest putReq = new PutObjectRequest();
        putReq.WithBucketName(config.Element("root").Element("S3BackupBucket").Value);
        putReq.WithKey(container + file.SelectSingleNode("name").InnerText);
        putReq.WithInputStream(Amazon.S3.Util.AmazonS3Util.MakeStreamSeekable(stream));
        putReq.WithMetaData("content-length", file.SelectSingleNode("bytes").InnerText);

        using (S3Response putResp = S3Client.PutObject(putReq))
        {

        }
    }

}

这是我从S3到云文件成功完成的方法:

And this is how I do it successfully from S3 to Cloud Files:

using (GetObjectResponse getResponse = S3Client.GetObject(new GetObjectRequest().WithBucketName(bucket.BucketName).WithKey(file.Key)))
{
    using (Stream s = getResponse.ResponseStream)
    {
        //We can stream right from s3 to CF, no need to store in memory or filesystem.                                            
        var req = (HttpWebRequest)WebRequest.Create(uri);
        req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
        req.Method = "PUT";

        req.AllowWriteStreamBuffering = false;
        if (req.ContentLength == -1L)
            req.SendChunked = true;


        using (Stream stream = req.GetRequestStream())
        {
            byte[] data = new byte[32768];
            int bytesRead = 0;
            while ((bytesRead = s.Read(data, 0, data.Length)) > 0)
            {
                stream.Write(data, 0, bytesRead);
            }
            stream.Flush();
            stream.Close();
        }
        req.GetResponse().Close();
    }
}   

推荐答案

似乎没有人回答过，所以我花了一些时间根据史蒂夫的回答来解决问题:

As no-one answering seems to have done it, I spent the time working it out based on guidance from Steve's answer:

在回答这个问题是否有使用HTTPWebRequest对S3请求进行身份验证的好例子，以便我可以复制对Cloud Files的操作?"，这是如何手动生成auth标头:

In answer to this question "is there any good examples of doing the authentication into S3 request using HTTPWebRequest, so I can duplicate what I do with Cloud Files?", here is how to generate the auth header manually:

string today = String.Format("{0:ddd,' 'dd' 'MMM' 'yyyy' 'HH':'mm':'ss' 'zz00}", DateTime.Now);

string stringToSign = "PUT\n" +
    "\n" +
    file.SelectSingleNode("content_type").InnerText + "\n" +
    "\n" +
    "x-amz-date:" + today + "\n" +
    "/" + strBucketName + "/" + strKey;

Encoding ae = new UTF8Encoding();
HMACSHA1 signature = new HMACSHA1(ae.GetBytes(AWSSecret));
string encodedCanonical = Convert.ToBase64String(signature.ComputeHash(ae.GetBytes(stringToSign)));

string authHeader = "AWS " + AWSKey + ":" + encodedCanonical;

string uriS3 = "https://" + strBucketName + ".s3.amazonaws.com/" + strKey;
var reqS3 = (HttpWebRequest)WebRequest.Create(uriS3);
reqS3.Headers.Add("Authorization", authHeader);
reqS3.Headers.Add("x-amz-date", today);
reqS3.ContentType = file.SelectSingleNode("content_type").InnerText;
reqS3.ContentLength = Convert.ToInt32(file.SelectSingleNode("bytes").InnerText);
reqS3.Method = "PUT";

请注意添加的x-amz-date标头，因为HTTPWebRequest以与AWS期望的格式不同的格式发送日期.

Note the added x-amz-date header as HTTPWebRequest sends the date in a different format to what AWS is expecting.

从那里开始只是重复我已经在做的事情.

From there it was just a case of repeating what I was already doing.

这篇关于从C#中的HTTPWebResponse.GetResponseStream()上传到S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！