将文件写入Google云端存储时发生IO异常 [英] IO Exception when writing files to Google Cloud Storage

问题描述

我正在尝试将Google App Engine中的文件写入Google Cloud，并使用后端处理将一堆文件写入Google云端存储的请求。但似乎我在大约5分钟内得到了IOException（我尝试了不同的文件大小，为每个文件睡了30秒等等，只有时间似乎是一致的）。

然后我测试了下面的代码：

  String testString = 1,1,1,1,1,1,1,1\" ; 
 for（int i = 0; i <200; i ++）{
 log.info（Dumping file+ i）; 
 String fileName =test / test+ i +.csv; 
 FileService fs = FileServiceFactory.getFileService（）; 
 GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder（）
 .setBucket（BUCKET_NAME）
 .setKey（fileName）
 .setAcl（public_read）; 
 AppEngineFile writableFile = fs.createNewGSFile（optionsBuilder.build（））; 
 FileWriteChannel writeChannel = fs.openWriteChannel（writableFile，true）; 
 PrintWriter out = new PrintWriter（Channels.newWriter（writeChannel，UTF8））; 
 for（int j = 0; j <10; j ++）{
 out.println（testString）; 
} 
 out.close（）; 
 writeChannel.closeFinally（）; 
 Thread.sleep（30000）; 
} 
  

以下是IOException：

  java.io.IOException 
 at com.google.appengine.api.files.FileServiceImpl.translateException（FileServiceImpl.java:617）
 at com.google.appengine.api.files.FileServiceImpl.makeSyncCall（FileServiceImpl.java:590）
 at com.google.appengine.api.files.FileServiceImpl.create（FileServiceImpl.java:498）
 at com.google.appengine.api.files.FileServiceImpl.createNewGSFile（FileServiceImpl.java:153）
  

在谷歌云存储桶，11个文件被写入并且很好（每个都有160个字节），并且该请求在创建第十二个文件时失败。对于我的真实应用程序，它将在writeChannel.closeFinally（）上失败。

  java.io.IOException 
 at com.google.appengine.api.files.FileServiceImpl.translateException（FileServiceImpl。 java：617）
在com.google.appengine.api.files.FileServiceImpl.makeSyncCall（FileServiceImpl.java:590）
在com.google.appengine.api.files.FileServiceImpl.close（FileServiceImpl。 java：551）
 at com.google.appengine.api.files.FileServiceImpl.close（FileServiceImpl.java:441）
 at com.google.appengine.api.files.FileWriteChannelImpl.closeFinally（FileWriteChannelImpl。 java：81）
  

btw。我确信我可以完全控制GAE应用程序的Google服务帐户，以写入相应的存储分区。

非常感谢！

解决方案

文件API仍然是实验性的 - 我认为你现在只是遇到了有关该服务的一般问题，因为它仍然是实验性的。

我知道在提高此服务的可靠性方面有一些非常活跃的工作，所以现在您应该可以进行防御性编码并在您看到类似此类短暂错误时执行重试。

I am trying to write files from Google App Engine to Google Cloud, and I use backend to process a request which writes a bunch of files to google cloud storage. But it seems that I get IOException in approximately about 5 minutes (I tried different file size, sleep the thread for 30s for each file etc, and only the time seems to be consistent).

Then I tested the following code:

         String testString = "1,1,1,1,1,1,1,1"; 
         for (int i = 0; i < 200; i++) {
            log.info("Dumping file " + i);
            String fileName = "test/test" + i + ".csv";
            FileService fs = FileServiceFactory.getFileService();
            GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder()
                .setBucket(BUCKET_NAME)
                .setKey(fileName)
                .setAcl("public_read");
            AppEngineFile writableFile = fs.createNewGSFile(optionsBuilder.build());
            FileWriteChannel writeChannel = fs.openWriteChannel(writableFile, true);
            PrintWriter out = new PrintWriter(Channels.newWriter(writeChannel, "UTF8"));
            for (int j = 0; j < 10; j++) { 
                out.println(testString);
            }
            out.close();
            writeChannel.closeFinally();
            Thread.sleep(30000);
        }

And I get IOException as below:

java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:590)
    at com.google.appengine.api.files.FileServiceImpl.create(FileServiceImpl.java:498)
    at com.google.appengine.api.files.FileServiceImpl.createNewGSFile(FileServiceImpl.java:153)

In the google cloud storage bucket, 11 files are written and are good (each has 160 bytes), and the request fails on creation of the twelfth file. For my real application, it will fail on writeChannel.closeFinally().

java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:590)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:551)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:441)
    at com.google.appengine.api.files.FileWriteChannelImpl.closeFinally(FileWriteChannelImpl.java:81)

btw. I am sure I give full control access to the google service account of my GAE application in order to write to the corresponding bucket.

Thanks very much!

解决方案

The files API is still experimental - I think you're now just running into general issues with the service because it's still experimental.

I know that there is some very active work happening in improving the reliability of this service so for now you should probably code defensively and implement retries when you see transiently errors like this.

这篇关于将文件写入Google云端存储时发生IO异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！