JavaScript堆内存不足| React/Node应用中的s3.getObject [英] JavaScript heap out of memory | s3.getObject in React/Node app

问题描述

我们在Digital Ocean上托管了一个React/Node应用程序.我们还将利用可与AWS S3互操作的Digital Ocean空间进行对象存储.基本上，该应用程序有点像内部保管箱.我们拥有能够创建文件夹并将内容上传到这些文件夹的管理员.然后，我们的客户便能够登录并下载我们允许他们访问的任何文件.

We have a React/Node app hosted on Digital Ocean. We're also utilizing Digital Ocean spaces which is interoperable with AWS S3 for Object storage. Basically, the app is sort of an in-house dropbox. We have Admins who are able to create folders and upload content to these folders. We then have Clients who are able to login and download any files we allow them access to.

我们成功地将所有文件上传到Digital Ocean Spaces.不管它们有多大/小.

We're successfully able to upload all files to Digital Ocean Spaces. No matter how large/small they are.

问题是当我们尝试以管理员或客户端身份下载任何大小超过100MB的内容时，会遇到JavaScript堆内存不足错误.该错误出现在系统的后端.

The problem is when we try to download (as Admin or Client) any content that is over 100MB in size we experience a JavaScript Heap out of memory error. This error appears on the Backend of the system.

我们尝试管理的一些解决方案是:

Some solutions we've attempted to administer are:

1. 为VM提供更多的内存
2. 增加浏览器的内存'--max-old-space-size'
3. 使Digital Ocean CDN能够通过边缘服务器流式传输内容
4. 手动将文件数据流式传输到我们的后端，然后将其发送到前端

前端代码

downloadFile = (id, name, type) => {
axios
  .get(
    `/test-download/${id}`,
    this.props.handleSnackBar(
      "Your download has been started. Please wait."
    )
  )

  .then(res => {
    download(

      new Blob([new Uint8Array(res.data.data.Body.data)]),
      `${name}.${type}`
    );
    console.log(res);

    console.log(res.data.data.Body),
      this.props.handleSnackBar("Your download is now ready.");
  })
  .catch(err => console.log(err));
};

后端代码

app.get("/test-download/:id", (req, res) => {
var params = {
  Bucket: bucketName,
  Key: req.params.id
};

s3.getObject(params, function(err, data) {
  //
  console.log(data);
  //
  if (!err) {

    res.send({ data, key: params.Key });
  } else {
    console.log({ err }); // an error occurred
  }
});
});

带有流的后端代码

app.get("/test-download/:id", (req, res) => {
var params = {
  Bucket: bucketName,
  Key: req.params.id
};
// TRY

const fileRequest = s3.getObject(params);

let chunks = [];
fileRequest
  .createReadStream()

  .on("data", function(data) {
    console.log(`Received ${data.length} bytes of data`);
    chunks.push(data);
  })
  .on("end", function() {
    console.log("no more data");
    bufferData = Buffer.concat(chunks);
    console.log(bufferData);
    res.send({ bufferData, key: params.Key });
  });

});

所以，基本上我有点卡住了.可以提供的任何帮助都将不胜感激.谢谢.

So, basically I'm sort of stuck. Any assistance that can be offered is greatly appreciated. Thanks.

推荐答案

感谢Marcos，我重新尝试了我们尝试使用的管道代码.但是，现在完全了解了我从createReadStream().pipe()收到的原始数据响应，便能够转换数据.

Thanks to Marcos, I revisited the piping code we had attempted. But now fully understanding the raw data response I was receiving from the createReadStream().pipe() I was able to convert the data.

前端代码

app.get("/test-download/:id", (req, res) => {
var params = {
  Bucket: bucketName,
  Key: req.params.id
};

s3.getObject(params)
  .createReadStream()
  .pipe(res)
  .on("finish", () => {
    console.log("** done");
  });
});

后端代码

downloadFile = (id, name, type) => {
axios
  .get(
    `/test-download/${id}`,
    { responseType: "arraybuffer" },
    this.props.handleSnackBar(
      "Your download has been started. Please wait."
    )
  )
  .then(res => {
    console.log(res);
    download(res.data, `${name}.${type}`);
    this.props.handleSnackBar("Your download is now ready.");
  })
  .catch(err => console.log(err));
};

这篇关于JavaScript堆内存不足| React/Node应用中的s3.getObject的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！