从Google Storage下载数据时出现SocketTimeoutException [英] SocketTimeoutException while downloading data from Google Storage

问题描述

从Google存储下载数据时，我收到大量SocketTimeoutException。什么是增加超时的最佳方法？
$ b

用例

下载24个文件（总共10GB）服务器使用Map / Reduce。

源代码

  Storage storage = new Storage.Builder（
 new NetHttpTransport（），
 new JacksonFactory（），
 new GoogleCredential（）。setAccessToken（accessToken））
 .setApplicationName （FooBar）
 .build（）; 
 
 Storage.Objects.Get getObject = storage.objects（）。get（bucket，fn）; 
 getObject.getMediaHttpDownloader（）。setDirectDownloadEnabled（true）; 
 getObject.executeMediaAndDownloadTo（outputStream）; 
  

堆栈跟踪

  java.net.SocketTimeoutException：读取超时
在java.net.SocketInputStream.socketRead0（本地方法）
在java.net.SocketInputStream.read SocketInputStream.java:150）
 at java.net.SocketInputStream.read（SocketInputStream.java:121）
 at sun.security.ssl.InputRecord.readFully（InputRecord.java:442）
在sun.security.ssl.InputRecord.read（InputRecord.java:480）
在sun.security.ssl.SSLSocketImpl.readRecord（SSLSocketImpl.java:927）
在sun.security.ssl.SSLSocketImpl .readDataRecord（SSLSocketImpl.java:884）
 at sun.security.ssl.AppInputStream.read（AppInputStream.java:102）
 at java.io.BufferedInputStream.fill（BufferedInputStream.java:235）
在java.io.BufferedInputStream.read1（BufferedInputStream.java:275）
在java.io.BufferedInputStream.read（BufferedInputStream.java:334）
在sun.net.www.MeteredStream。阅读（MeteredS tream.java:134）
在java.io.FilterInputStream.read（FilterInputStream.java:133）
在sun.net.www.protocol.http.HttpURLConnection $ HttpInputStream.read（HttpURLConnection.java： 3052）
在sun.net.www.protocol.http.HttpURLConnection $ HttpInputStream.read（HttpURLConnection.java:3046）
在com.google.api.client.util.ByteStreams.copy（ByteStreams。 java：51）
在com.google.api.client.util.IOUtils.copy（IOUtils.java:94）
在com.google.api.client.util.IOUtils.copy（IOUtils。 java：63）
，位于com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest（MediaHttpDownloader.java:261）
位于com.google.api.client.googleapis.media.MediaHttpDownloader。通过com.google.api.services.storage上的com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo（AbstractGoogleClientRequest.java:553）
下载（MediaHttpDownloader.java:209）
。存储$对象$ Get.executeMediaAndDownloadTo（Storage.java:4494 ）
  

编辑

这只有在我从日本下载数据时才会发生。从美国东部和美国西部一切都很好。

解决方案

试试以下...

1. 在linux框中设置代理。
在创建存储时 - 新的NetHttpTransport（）提供了自己的方式来创建套接字并将代理添加到传输中。

  public HttpClient myHttpClient（）抛出异常{
 SchemeRegistry schemeRegistry = new SchemeRegistry（）; 
 // SetRegisrty用于HTTP和HTTPS  - 检查谷歌。 
 schemeRegistry.register（new Scheme（http，PlainSocketFactory 
 .getSocketFactory（），YOUR_PROXY_PORT））; 
 schemeRegistry.register（new Scheme（https，SSLSocketFactory 
 .getSocketFactory（），443））; 
 
 HttpParams params = new BasicHttpParams（）; 
 HttpConnectionParams.setConnectionTimeout（params，30 * 1000）; //设置超时
 HttpProtocolParams.setVersion（params，HttpVersion.HTTP_1_1）; 
 ClientConnectionManager connManager = new ThreadSafeClientConnManager（
 params，schemeRegistry）; 
 DefaultHttpClient httpClient = new DefaultHttpClient（connManager，
 params）; 
尝试{
 int proxyPort = YOUR_PROXY_PORT; 
字符串proxyHost =YOUR_PROXT_HOST_NAME; （proxyPort> 0&&proxyHost！= null&& proxyHost.length（）> 0）{
 System.setProperty（https.proxyHost，proxyHost）; 
 System.setProperty（https.proxyPort，proxyPort +）; 
 System.setProperty（http.proxyHost，proxyHost）; 
 System.setProperty（http.proxyPort，proxyPort +）; 
 HttpHost proxy = new HttpHost（proxyHost，proxyPort）; 
 httpClient.getParams（）。setParameter（
 ConnRoutePNames.DEFAULT_PROXY，proxy）; 
} 
} catch（NullPointerException e）{
 System.out.println（Proxy error here）; 
} 
返回httpClient; 
} 
 
 public static HttpTransport myNetHttpTransport（）
 throws Exception {
返回新的ApacheHttpTransport（myHttpClient（））; 
 
 code $ 
 
 $ b $ p使用 .setTransport（myNetHttpTransport（））而不是 NetHttpTransport（）。
 
 
 
我们在此花了很长时间。但截至目前，这似乎正在起作用。请让我知道是否有任何帮助...  
I'm getting a large number of SocketTimeoutException while downloading data from Google Storage. What's the best way to increase the timeout ? 

Use case

Download 24 files (10GB total) in parallel from multiple servers using Map/Reduce.

Source Code
  Storage storage = new Storage.Builder(
                new NetHttpTransport(),
                new JacksonFactory(),
                new GoogleCredential().setAccessToken(accessToken))
            .setApplicationName("FooBar")
            .build();

  Storage.Objects.Get getObject = storage.objects().get(bucket, fn);
  getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
  getObject.executeMediaAndDownloadTo(outputStream);
Stack Trace
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.MeteredStream.read(MeteredStream.java:134)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3046)
    at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:261)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:209)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:553)
    at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:4494)
Edit

This only happens when I download the data from Japan. Everything is fine from US-East and US-West.
解决方案
Try the following ...

Set the proxy in the linux box. 
While creating storage - new NetHttpTransport() provide your own way to create the socket and add the proxy in the transport.
public HttpClient myHttpClient() throws Exception {
    SchemeRegistry schemeRegistry = new SchemeRegistry();
    //SetRegisrty for both HTTP and HTTPS - Check google for this.
    schemeRegistry.register(new Scheme("http", PlainSocketFactory
            .getSocketFactory(), YOUR_PROXY_PORT));
    schemeRegistry.register(new Scheme("https", SSLSocketFactory
            .getSocketFactory(), 443));

    HttpParams params = new BasicHttpParams();
    HttpConnectionParams.setConnectionTimeout(params, 30 * 1000);  // SET the timeout
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    ClientConnectionManager connManager = new ThreadSafeClientConnManager(
            params, schemeRegistry);
    DefaultHttpClient httpClient = new DefaultHttpClient(connManager,
                params);
    try {
        int proxyPort = YOUR_PROXY_PORT;
        String proxyHost = "YOUR_PROXT_HOST_NAME";
        if (proxyPort > 0 && proxyHost != null && proxyHost.length() > 0) {
            System.setProperty("https.proxyHost", proxyHost);
            System.setProperty("https.proxyPort", proxyPort + "");
            System.setProperty("http.proxyHost", proxyHost);
            System.setProperty("http.proxyPort", proxyPort + "");
            HttpHost proxy = new HttpHost(proxyHost, proxyPort);
            httpClient.getParams().setParameter(
                    ConnRoutePNames.DEFAULT_PROXY, proxy);
        }
    } catch (NullPointerException e) {
        System.out.println("Proxy error here");
    }
    return httpClient;
}

public static HttpTransport myNetHttpTransport()
        throws Exception {
    return new ApacheHttpTransport(myHttpClient());
}

Use .setTransport(myNetHttpTransport()) instead of NetHttpTransport().

We spent long time on this. but as of now this seems to be working. Please let me know if any help in this...

                        
这篇关于从Google Storage下载数据时出现SocketTimeoutException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！