使用预先签名的URL将文件放入S3 [英] PUT file to S3 with presigned URL

问题描述

我整夜都在使用Amazon S3预签名URL来尝试放置文件.我用Java代码生成了预签名的URL.

    AWSCredentials credentials = new BasicAWSCredentials( accessKey, secretKey );
    client = new AmazonS3Client( credentials );
    GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest( bucketName, "myfilename", HttpMethod.PUT);
    request.setExpiration( new Date( System.currentTimeMillis() + (120 * 60 * 1000) ));
    return client.generatePresignedUrl( request ).toString();

然后我想使用生成的预先签名的URL使用curl放置文件.

curl -v -H "content-type:image/jpg" -T mypicture.jpg https://mybucket.s3.amazonaws.com/myfilename?Expires=1334126943&AWSAccessKeyId=<accessKey>&Signature=<generatedSignature>

我认为，像GET一样，它可以在不公开的存储桶上工作(这是预先签名的地方，对吗?)嗯，每次尝试都拒绝访问.最终，出于沮丧，我更改了存储桶的权限，以允许所有人进行写操作.当然，然后预签名的URL起作用了.我迅速从存储桶中删除了所有人的权限.现在，我无权删除通过自己的预签名URL上传到存储桶中的项目.我立即查看，我可能应该放一个x-amz-acl我上传的标题.我怀疑我会在正确之前创建更多无法删除的对象.

这会引起一些问题:

• 如何使用PUT和生成的预签名URL进行curl上传?
• 如何删除上传的文件和我创建的用于测试的存储桶?

最终目标是手机将使用该预签名的URL来放置图像.我正在尝试使其卷曲，以作为概念证明.

更新:我在 amazon上问了一个问题论坛.如果提供了答案，我将在此处作为答案.

解决方案

这确实有点令人费解，我认为这是 curl 命令将这样上传您的文件(当然，当然要假设已更新了预签名URL):

curl -v -T mypicture.jpg https://mybucket.s3.amazonaws.com/myfilename?Expires=1334126943&AWSAccessKeyId=<accessKey>&Signature=<generatedSignature>

也就是说，我排除了Content type标头，结果产生了application/octet-stream(或binary/octet-stream)，这显然是不希望的.因此，已经进行了进一步的挖掘.

背景/分析

对 Amazon S3 的PUT(以及DELETE和HEAD)请求的预签名URL是已知可以正常工作，但在此站点上的相关问题中至少有证据证明(请参阅例如，我对的回答，使用pre-签名的URL(获取403)).

记录了便利的查询字符串请求身份验证替代使用以下伪语法说明查询字符串请求身份验证方法

:

StringToSign = HTTP-VERB + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Expires + "\n" +
    CanonicalizedAmzHeaders +
    CanonicalizedResource;    

它确实包含Content-Type标头，并且(如您已经发现的那样)在某些已记录的情况下这是缺少的部分，请参见例如AWS团队对 GetPreSignedURL和PUT请求的响应，一旦添加了预签名的URL.

使用适用于.NET的AWS开发工具包确实很容易实现，它提供了便捷的方法 GetPreSignedUrlRequest.WithContentType 即可做到:

为此请求设置ContentType属性.该属性默认 到二进制/八位字节流"，但是如果您需要其他内容，则可以 设置此属性.

相应地，扩展相应的示例使用预先签名的URL上传对象-AWS如下所示，适用于.NET的SDK 产生具有内容类型的有效的预签名URL，可以按预期(即，完全按照您的尝试)通过 curl 上传:

     // ...
    GetPreSignedUrlRequest request = new GetPreSignedUrlRequest();
    // ...
    request.WithContentType("image/jpg");
    // ...
 

现在，我们想扩展语义相同的示例使用Pre上载对象-签名的URL-适用于Java的AWS开发工具包(SDK)以类似的方式进行，但是(正如您已经发现的那样)，没有专用的方法可以实现此目的.但是，这可能只是一种缺乏便利的方法，可以通过 // ... request.setExpiration( new Date( System.currentTimeMillis() + (120 * 60 * 1000) )); request.addRequestParameter("content-type", "image/jpg"); return client.generatePresignedUrl( request ).toString(); // ...

但是，这两种方法的文档都提出了其他目的，但实际上并没有用，即无论设置哪种内容类型(如果有)，它们总是产生相同的签名.

进一步调试SDK显示，它们都提供了语义相似的核心方法，可以根据上面引用的 pseudo-grammar 计算查询字符串身份验证，请参见 buildSigningString() https://github.com/amazonwebservices/aws-sdk-for-java/blob/master/src/main/java/com/amazonaws/services/s3/internal/RestUtils.java#L57"rel =" noreferrer>适用于Java的makeS3CanonicalString().

但是Java版本中的相应代码将所有有趣的标头添加到列表中，然后对其进行排序，其中有趣"定义为Content-MD5，Content-Type，Date ，实际上从未执行过x-amz-，因为确实没有方法以某种方式提供这些标头，这些标头仅可用于类 aws-sdk-for-java项目提交拉取请求) ，但是以兼容且可维护的方式实现这一目标似乎有些棘手，因此，最好由SDK维护者自己完成.

I've been playing with Amazon S3 presigned URLs all night attempting to PUT a file. I generate the presigned URL in java code.

    AWSCredentials credentials = new BasicAWSCredentials( accessKey, secretKey );
    client = new AmazonS3Client( credentials );
    GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest( bucketName, "myfilename", HttpMethod.PUT);
    request.setExpiration( new Date( System.currentTimeMillis() + (120 * 60 * 1000) ));
    return client.generatePresignedUrl( request ).toString();

I then want to use the generated, presigned URL to PUT a file using curl.

curl -v -H "content-type:image/jpg" -T mypicture.jpg https://mybucket.s3.amazonaws.com/myfilename?Expires=1334126943&AWSAccessKeyId=<accessKey>&Signature=<generatedSignature>

I assumed that, like a GET, this would work on a bucket which is not public (that's the point of presigned, right?) Well, I got access denied on every attempt. Finally out of frustration I changed the permission of the bucket to allow EVERYONE to write. Of course, then the presigned URL worked. I quickly removed the EVERYONE permission from the bucket. Now, I don't have permission to delete the item that was uploaded into my bucket by my own self-pre-signed URL. I see now that I probably should have put a x-amz-acl header on what I uploaded. I suspect I'll create several more undelete-able objects before I get that right.

This leads to a few questions:

• How can I upload with curl using PUT and a generated presigned URL?
• How can I delete the uploaded file and the bucket I created to test it with?

The end goal is that a mobile phone will use this presigned URL to PUT images. I'm trying to get it going in curl as a proof of concept.

Update: I asked a question on the amazon forums. If an answer is provided there I'll put it as an answer here.

解决方案

This is indeed a bit puzzling, I consider it to be a bug in the AWS SDK for Java (see below) - but first and foremost, the following curl command will upload your file as such (assuming an updated pre-signed URL of course):

curl -v -T mypicture.jpg https://mybucket.s3.amazonaws.com/myfilename?Expires=1334126943&AWSAccessKeyId=<accessKey>&Signature=<generatedSignature>

That is, I've excluded the Content type header, which yields application/octet-stream (or binary/octet-stream) as a result, which is obviously not desired; thus, further digging had been order.

Background / Analysis

Pre-signed URLs for PUT (and DELETE as well as HEAD) requests to Amazon S3 are known to work in principle, not the least evidenced in related questions on this site (see e.g. my answer to Upload to s3 with curl using pre-signed URL (getting 403)).

The facilitated Query String Request Authentication Alternative is documented to use the following pseudo-grammar that illustrates the query string request authentication method:

StringToSign = HTTP-VERB + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Expires + "\n" +
    CanonicalizedAmzHeaders +
    CanonicalizedResource;    

It does include the Content-Type header, and (as you already discovered) this has been the missing piece in some documented cases, see e.g. the AWS team response to GetPreSignedURL with PUT request, yielding a working pre-signed URL once added.

This is easy to achieve with the AWS SDK for .NET indeed, which provides the convenience method GetPreSignedUrlRequest.WithContentType to do just that:

Sets the ContentType property for this request. This property defaults to "binary/octet-stream", but if you require something else you can set this property.

Accordingly, extending the respective sample Upload an Object Using Pre-Signed URL - AWS SDK for .NET as follows yields a working pre-signed URL with content type, that can be uploaded via curl as expected (i.e. exactly as you attempted to):

    // ...
    GetPreSignedUrlRequest request = new GetPreSignedUrlRequest();
    // ...
    request.WithContentType("image/jpg");
    // ...

Now, one would like to extend the semantically identical sample Upload an Object Using Pre-Signed URL - AWS SDK for Java in a similar fashion, but (as you've discovered already as well), there is no dedicated method to achieve this. This might just be a lacking convenience method though and could be achievable via addRequestParameter() or setResponseHeaders() eventually, e.g.:

  // ...
  request.setExpiration( new Date( System.currentTimeMillis() + (120 * 60 * 1000) ));
  request.addRequestParameter("content-type", "image/jpg");
  return client.generatePresignedUrl( request ).toString();
  // ...

However, both method's documentation suggests other purposes, and it doesn't work indeed, i.e. they always yield the identical signature, no matter which content type is set like so (if any).

Debugging further into the SDKs reveals, that both provide a semantically similar core method to calculate the query string authentication according to the pseudo-grammar referenced above, see buildSigningString() for .NET and makeS3CanonicalString() for Java.

But the respective code in the Java version to Add all interesting headers to a list, then sort them, where "Interesting" is defined as Content-MD5, Content-Type, Date, and x-amz- is never executed in fact, because there is indeed no method to provide these headers somehow, which are only available for class DefaultRequest and not class GeneratePresignedUrlRequest used to initialize the former, which is used as input for calculating the signature in turn, see protected method createRequest().

Interestingly/Notably, the two methods to calculate the query string authentication in .NET vs. Java compose their input from an almost inverse combination of header vs. parameter sources on the call stack, which could hint on the cause of the Java bug, but obviously that might as well be just difficult to decipher, i.e. the internal architecture could differ significantly of course.

Preliminary Conclusion

There are two angles to this:

• The AWS SDK for Java is definitely lacking the convenience method for setting the content type, which might be a comparatively rare, but nonetheless obvious use case accounted for in other AWS SDKs accordingly - this is surprising, given its widespread use in AWS related backend services.
• Regardless, there seems to be something fishy with the way the Query String Request Authentication is implemented in comparison to the .NET version for example - again this is surprising, given it is a core functionality, however, this is still within the S3 model/namespace and thus might only be required by the respective uses cases above.

In conclusion, the only reasonable way to resolve this would be an updated SDK, so a bug report is in order - obviously one could as well duplicate/extend the SDK functionality to account for this special case separately (ideally in a way allowing to submit a pull request for the aws-sdk-for-java project), but getting this right in a compatible and maintainable way seems to be a bit tricky, thus is likely best done by the SDK maintainers themselves.

这篇关于使用预先签名的URL将文件放入S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！