将文件上传到Google云端存储：500后端错误 [英] Uploading files into Google Cloud Storage: 500 Backend Error

问题描述

我们正在尝试将文件上传到Google Cloud Storage，然后再将它们转移到BigQuery中，但在上传某些内容时，我们经常面临'500内部服务器错误'或'410 Gone'（原始邮件）。

我们正在使用官方的SDK，并增加了指数退避的重试，但错误总是在这里。你有任何建议吗？

以下是我们如何上传（scala）：

  val credential = new GoogleCredential（）。setAccessToken（accessToken）
 $ b $ val requestInitializer = new HttpRequestInitializer（）{
 def initialize（request：HttpRequest）：Unit = {
 credential.initialize（请求）
 
 //避免读取超时异常
 request.setConnectTimeout（200000）
 request.setReadTimeout（200000）
 
 request.setIOExceptionHandler（new 
 HttpBackOffIOExceptionHandler（new ExponentialBackOff（）））
 request.setUnsuccessfulResponseHandler（new 
 HttpBackOffUnsuccessfulResponseHandler（new ExponentialBackOff（）））
} 
} 
 
 val storage = new Storage.Builder（
 new NetHttpTransport，
 JacksonFactory.getDefaultInstance，
 requestInitializer 
）.setApplicationName（MyAppHere）。build 
 
 val objectMetadata = new StorageObject（）
 .setBucket（bucketName）
 .setName（distantFileName）
 
 val isc = new InputStreamContent（binary / octet-stream，fis）
 val length = isc.getLength 
 val insertObject = storage.objects（）。insert（bucketName，objectMetadata，isc）
 //对于小文件，您可能希望调用setDirectUploadEnabled（true）到
 //减少向服务器发出的HTTP请求数量。 
 if（length> 0&& length< = 2 * 1000 * 1000 / * 2MB * /）{
 insertObject.getMediaHttpUploader.setDirectUploadEnabled（true）
} 
 insertObject.execute（）
  

我们的scala依赖关系：

 com.google.api-client％google-api-client％1.18.0-rc，
com.google.api-客户端％google-api-client-jackson2％1.18.0-rc，
com.google.apis％google-api-services-bigquery％v2-rev142-1.18。 0-rc，
com.google.apis％google-api-services-storage％v1-rev1-1.18.0-rc，
com.google.http-客户端％google-http-client％1.18.0-rc，
com.google.oauth-client％google-oauth-client％1.18.0-rc
  

原始SDK错误回应：

  500内部服务器错误
 {
code：500，
errors：[{
domain：global，
message：Backend Error，
reason：backendError
 }]，
message：后端错误
} 
 
 410去掉
 {
code：500，
错误：[{
domain：global，
message：Backend Error，
reason：backendError
}]，
message：Backend Error
} 
  

解决方案

每个后端错误都应该使用指数重试来处理，因为可能存在服务问题。

如果错误在我们说10个小时后仍然存在，那么您应该联系支持人员，以便为您提供1：1的帮助。

We are trying to upload files into Google Cloud Storage before moving them into BigQuery, but we are often facing '500 Internal Server Error' or '410 Gone' (raw messages below) during some uploads.

We are using the official SDK and have added retry with exponential backoff but the errors are always here. Do you have any advise please ?

Here is how we upload (scala) :

val credential = new GoogleCredential().setAccessToken(accessToken)

val requestInitializer = new HttpRequestInitializer() {
    def initialize(request: HttpRequest): Unit = {
        credential.initialize(request)

        // to avoid read timed out exception
        request.setConnectTimeout(200000)
        request.setReadTimeout(200000)

        request.setIOExceptionHandler(new 
            HttpBackOffIOExceptionHandler(new ExponentialBackOff()))
        request.setUnsuccessfulResponseHandler(new 
            HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()))
    }
}

val storage = new Storage.Builder(
    new NetHttpTransport,
    JacksonFactory.getDefaultInstance,
    requestInitializer
).setApplicationName("MyAppHere").build

val objectMetadata = new StorageObject()
    .setBucket(bucketName)
    .setName(distantFileName)

val isc = new InputStreamContent("binary/octet-stream", fis)
val length = isc.getLength
val insertObject = storage.objects().insert(bucketName, objectMetadata, isc)
// For small files, you may wish to call setDirectUploadEnabled(true), to
// reduce the number of HTTP requests made to the server.
if (length > 0 && length <= 2 * 1000 * 1000 /* 2MB */ ) {
    insertObject.getMediaHttpUploader.setDirectUploadEnabled(true)
}
insertObject.execute()

Our scala dependancies :

"com.google.api-client" % "google-api-client" % "1.18.0-rc",
"com.google.api-client" % "google-api-client-jackson2" % "1.18.0-rc",
"com.google.apis" % "google-api-services-bigquery" % "v2-rev142-1.18.0-rc",
"com.google.apis" % "google-api-services-storage" % "v1-rev1-1.18.0-rc",
"com.google.http-client" % "google-http-client" % "1.18.0-rc",
"com.google.oauth-client" % "google-oauth-client" % "1.18.0-rc"

Raw SDK error responses :

500 Internal Server Error
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

410 Gone
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

解决方案

Every Backend error should be handled with an exponential retry, as there might be service problems.

If the error still persists after let's say 10 hours then you should contact the support in order to provide you 1:1 help to your problem.

这篇关于将文件上传到Google云端存储：500后端错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！