如何从 Node.js 中的 S3 getObject 获得响应? [英] How to get response from S3 getObject in Node.js?

问题描述

在 Node.js 项目中，我试图从 S3 取回数据.

In a Node.js project I am attempting to get data back from S3.

当我使用 getSignedURL 时，一切正常:

When I use getSignedURL, everything works:

aws.getSignedUrl('getObject', params, function(err, url){
    console.log(url); 
}); 

我的参数是:

var params = {
              Bucket: "test-aws-imagery", 
              Key: "TILES/Level4/A3_B3_C2/A5_B67_C59_Tiles.par"

如果我将 URL 输出带到控制台并将其粘贴到 Web 浏览器中，它会下载我需要的文件.

If I take the URL output to the console and paste it in a web browser, it downloads the file I need.

但是，如果我尝试使用 getObject，我会遇到各种奇怪的行为.我相信我只是错误地使用它.这是我试过的:

However, if I try to use getObject I get all sorts of odd behavior. I believe I am just using it incorrectly. This is what I've tried:

aws.getObject(params, function(err, data){
    console.log(data); 
    console.log(err); 
}); 

输出:

{ 
  AcceptRanges: 'bytes',
  LastModified: 'Wed, 06 Apr 2016 20:04:02 GMT',
  ContentLength: '1602862',
  ETag: '9826l1e5725fbd52l88ge3f5v0c123a4"',
  ContentType: 'application/octet-stream',
  Metadata: {},
  Body: <Buffer 01 00 00 00  ... > }

  null

所以看起来这工作正常.但是，当我在 console.log 之一上放置断点时，我的 IDE (NetBeans) 会抛出错误并拒绝显示数据值.虽然这可能只是 IDE，但我决定尝试其他方式来使用 getObject.

So it appears that this is working properly. However, when I put a breakpoint on one of the console.logs, my IDE (NetBeans) throws an error and refuses to show the value of data. While this could just be the IDE, I decided to try other ways to use getObject.

aws.getObject(params).on('httpData', function(chunk){
    console.log(chunk); 
}).on('httpDone', function(data){
    console.log(data); 
});

这不会输出任何内容.放置断点表明代码永远不会到达 console.log 中的任何一个.我也试过:

This does not output anything. Putting a breakpoint in shows that the code never reaches either of the console.logs. I also tried:

aws.getObject(params).on('success', function(data){
    console.log(data); 
});

然而，这也不会输出任何内容，并且放置断点表明永远不会到达 console.log.

However, this also does not output anything and placing a breakpoint shows that the console.log is never reached.

我做错了什么?

推荐答案

当从 S3 API 执行 getObject() 时，根据 docs 您的文件内容位于 Body 属性中，其中您可以从示例输出中看到.你应该有类似下面的代码

When doing a getObject() from the S3 API, per the docs the contents of your file are located in the Body property, which you can see from your sample output. You should have code that looks something like the following

const aws = require('aws-sdk');
const s3 = new aws.S3(); // Pass in opts to S3 if necessary

var getParams = {
    Bucket: 'abc', // your bucket name,
    Key: 'abc.txt' // path to the object you're looking for
}

s3.getObject(getParams, function(err, data) {
    // Handle any error and exit
    if (err)
        return err;

  // No error happened
  // Convert Body from a Buffer to a String
  let objectData = data.Body.toString('utf-8'); // Use the encoding necessary
});

您可能不需要从 data.Body 对象创建新缓冲区，但如果需要，您可以使用上面的示例来实现.

You may not need to create a new buffer from the data.Body object but if you need you can use the sample above to achieve that.

自从我在 2016 年写下这个答案以来，亚马逊发布了一个新的 JavaScript SDK，@aws-sdk/client-s3.这个新版本通过始终返回承诺而不是通过链接到 getObject() 的 .promise() 来改进原始 getObject()>.除此之外，response.Body 不再是 Buffer 但是，Readable|ReadableStream|Blob 之一.这稍微改变了 response.Data 的处理.这应该会更高效，因为我们可以流式传输返回的数据，而不是将所有内容保存在内存中，但代价是实现起来更加冗长.

Since I wrote this answer in 2016, Amazon has released a new JavaScript SDK, @aws-sdk/client-s3. This new version improves on the original getObject() by returning a promise always instead of opting in via .promise() being chained to getObject(). In addition to that, response.Body is no longer a Buffer but, one of Readable|ReadableStream|Blob. This changes the handling of the response.Data a bit. This should be more performant since we can stream the data returned instead of holding all of the contents in memory, with the trade-off being that it is a bit more verbose to implement.

在下面的示例中，response.Body 数据将被流式传输到一个数组中，然后作为字符串返回.这是我原始答案的等效示例.或者，response.Body 可以使用 stream.Readable.pipe() 到 HTTP 响应、文件或任何其他类型的 stream.Writeable 以供进一步使用，这将是获取大对象时更高效的方式.

In the below example the response.Body data will be streamed into an array and then returned as a string. This is the equivalent example of my original answer. Alternatively, the response.Body could use stream.Readable.pipe() to an HTTP Response, a File or any other type of stream.Writeable for further usage, this would be the more performant way when getting large objects.

如果您想使用 Buffer，就像原始的 getObject() 响应一样，可以通过将 responseDataChunks 包装在 responseDataChunks 中来实现a href="https://nodejs.org/dist/latest-v16.x/docs/api/buffer.html#buffer_static_method_buffer_concat_list_totallength" rel="nofollow noreferrer">Buffer.concat() 而不是使用 Array#join()，这在与二进制数据交互时很有用.需要注意的是，由于 Array#join() 返回一个字符串，responseDataChunks 中的每个 Buffer 实例都会有 Buffer.toString() 隐式调用和默认调用将使用 utf8 的编码.

If you wanted to use a Buffer, like the original getObject() response, this can be done by wrapping responseDataChunks in a Buffer.concat() instead of using Array#join(), this would be useful when interacting with binary data. To note, since Array#join() returns a string, each Buffer instance in responseDataChunks will have Buffer.toString() called implicitly and the default encoding of utf8 will be used.

const { GetObjectCommand, S3Client } = require('@aws-sdk/client-s3')
const client = new S3Client() // Pass in opts to S3 if necessary

function getObject (Bucket, Key) {
  return new Promise(async (resolve, reject) => {
    const getObjectCommand = new GetObjectCommand({ Bucket, Key })

    try {
      const response = await client.send(getObjectCommand)
  
      // Store all of data chunks returned from the response data stream 
      // into an array then use Array#join() to use the returned contents as a String
      let responseDataChunks = []

      // Handle an error while streaming the response body
      response.Body.once('error', err => reject(err))
  
      // Attach a 'data' listener to add the chunks of data to our array
      // Each chunk is a Buffer instance
      response.Body.on('data', chunk => responseDataChunks.push(chunk))
  
      // Once the stream has no more data, join the chunks into a string and return the string
      response.Body.once('end', () => resolve(responseDataChunks.join('')))
    } catch (err) {
      // Handle the error or throw
      return reject(err)
    } 
  })
}

@aws-sdk/client-s3 文档链接

• GetObjectCommand
• GetObjectCommandInput
• GetObjectCommandOutput

这篇关于如何从 Node.js 中的 S3 getObject 获得响应?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！