GCS - 上传大小较大的文件时发生堆错误 [英] GCS - Heap Error When File of higher size was uploaded

问题描述

我想将400MB的文件大小上传到Google云端存储。以下是我用于获取上传URL并在angularjs中发布的代码片段

  Java Servlet：
 
 UploadOptions opts = UploadOptions.Builder 
 .withGoogleStorageBucketName（GCS_BUCKET_NAME）; 
 String url = blobstore.createUploadUrl（/ upload，opts）; 
 
 AngularJS：
 
 $ scope.formData.append（'file'，$ scope.filename）; 
 
 $ http.post（uploadUrl，$ scope.formData，{
 withCredentials：true，
 headers：{
'Content-Type'：undefined 
}，
 transformRequest：angular.identity 
}）
  

工作正常，直到文件大小为100-150MB左右，文件大小更高时我在浏览器控制台中出现以下错误。

未能加载资源：服务器响应状态为500（Java堆空间）

我需要做些什么来避免这个错误？正如你在回答我上面的问题时所说的那样，这只发生在开发中，而不是生产中。我的猜测是，谷歌的blobstore代码将整个请求读入其开发服务器的内存中，但很难说，因为我们无法访问该代码。如果你需要在dev中测试更大的上传，你可以尝试配置你的dev服务器以启动更多的内存。看起来我使用 -Xmx512M （在eclipse中启动配置的VM arguments部分），你甚至可以尝试更多。或者，如果它在开发中不重要，则可以测试较小的上传并相信大型的上传可用于生产。如果你决定使用更多的内存来启动开发服务器：在我的例子中，一起使用Eclipse和Maven，我必须创建一个启动配置的副本Maven会自动生成并修改它，否则Maven会覆盖我的更改。这也意味着我必须不时手动更新我的副本，例如更新appengine SDK（以反映新版本号）。

I want to upload file size of 400MB to Google Cloud Storage. Following is the code snippet that I am using to get upload url and for posting in angularjs

Java Servlet:

UploadOptions opts = UploadOptions.Builder
        .withGoogleStorageBucketName(GCS_BUCKET_NAME);
String url = blobstore.createUploadUrl("/upload", opts);

AngularJS:

$scope.formData.append('file', $scope.filename);

$http.post(uploadUrl, $scope.formData, {
    withCredentials: true,
    headers: {
        'Content-Type': undefined
    },
    transformRequest: angular.identity
})

The code is working fine till the file size is around 100-150MB, when the file size is higher I got the following error in browser console.

Failed to load resource: the server responded with a status of 500 (Java heap space)

What I need to do to avoid this error?

解决方案

As you said in your response to my question above, this is only happening in development, not in production. My guess is that google's blobstore code reads the whole request into memory in their development server, but it's hard to say because we don't have access to that code. If you need to test larger uploads in dev, you could try configuring your dev server to launch with more memory. Looks like I use -Xmx512M (in the "VM arguments" section of the launch configuration in eclipse), you could even try more. Or if it's not important in dev, you can just test smaller uploads and trust that large ones will work in production. :)

FYI if you decide to launch the dev server with more memory: In my case, using Eclipse and Maven together, I had to create a copy of the launch configuration that Maven generated automatically and modify that one, otherwise Maven would overwrite my changes. That also means I have to update my copy from time to time by hand, such as when updating the appengine SDK (to reflect the new version number).

这篇关于GCS - 上传大小较大的文件时发生堆错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！