如何在java中进行批量http调用 [英] How to make batch http call in java

问题描述

我正在尝试通过Http访问另一个服务来使用HttpClient获取数据。 uri应该看起来像端点：80 /.../ itemId。

I am trying to access another service by Http to fetch data using HttpClient. The uri should look like endpoint:80/.../itemId.

我想知道是否有办法进行批量调用以指定一组itemIds？我确实在创建请求时发现有人建议使用.setHeader（HttpHeaders.CONNECTION，keep-alive）。通过这样做，我怎样才能在获取所有数据后释放客户端？

I am wondering if there is a way to make a batch call to specify a set of itemIds? I did find someone suggests to .setHeader(HttpHeaders.CONNECTION, "keep-alive") when creating request. Via doing this, how could I release the client after getting all data?

此外，似乎此方法仍然需要获得一个响应然后发送另一个请求？这可能是async，以及如何做到这一点？ BTW，似乎我出于某种原因无法在这种情况下使用AsyncHttpClient。

Also, seems like this method still need to get one response then send another request? Is this possible to do it as async, and how? BTW, seems like I could not use AsyncHttpClient in this case for some reason.

由于我对HttpClient几乎一无所知，这个问题可能看起来很愚蠢。真的希望有人能帮我解决问题。

Since I am barely know nothing about HttpClient, the question may look dumb. Really hope someone could help me solve the problem.

推荐答案

服务器上的API支持

API支持一次请求多个ID的可能性很小（例如，使用格式为 http：// endpoint：80 /.../ itemId1，itemId2的URL， itemId3 ）。检查API文档以查看它是否可用，因为如果是这样，那将是最佳解决方案。

API support on the server

There is a small chance that the API supports requesting multiple IDs at a time (e.g. using a URL of the form http://endpoint:80/.../itemId1,itemId2,itemId3). Check the API documentation to see if this is available, because if so that would be the best solution.

默认情况下，Apache HttpClient使用持久（保持活动）连接（请参阅连接管理教程链接在 @ kichik的评论）。 日志工具可以帮助验证连接是否被重用几个请求。

It looks like Apache HttpClient uses persistent ("keep alive") connections by default (see the Connection Management tutorial linked in @kichik's comment). The logging facilities could help to verify that connections are reused for several requests.

要释放客户端，请使用 close（）方法。来自 2.3.4。连接管理器关闭：

To release the client, use the close() method. From 2.3.4. Connection manager shutdown:

当不再需要HttpClient实例并且即将超出范围时，重要的是关闭其连接管理器，以确保管理器保持活动状态的所有连接都被关闭，并释放这些连接分配的系统资源。

When an HttpClient instance is no longer needed and is about to go out of scope it is important to shut down its connection manager to ensure that all connections kept alive by the manager get closed and system resources allocated by those connections are released.

CloseableHttpClient httpClient = <...>
httpClient.close();

持久连接消除了建立新连接的开销，但是当你已经注意到客户端仍然会在发送下一个请求之前等待响应。

Persistent connections eliminate the overhead of establishing new connections, but as you've noted the client will still wait for a response before sending the next request.

您可以使您的程序多线程并使用 PoolingHttpClientConnectionManager 以控制对服务器的连接数。以下是基于 2.3.3的示例。池连接管理器和 2.4。多线程请求执行：

You can make your program multithreaded and use a PoolingHttpClientConnectionManager to control the number of connections made to the server. Here is an example based on 2.3.3. Pooling connection manager and 2.4. Multithreaded request execution:

import java.io.*;
import org.apache.http.*;
import org.apache.http.client.*;
import org.apache.http.client.methods.*;
import org.apache.http.client.protocol.*;
import org.apache.http.impl.client.*;
import org.apache.http.impl.conn.*;
import org.apache.http.protocol.*;

// ...
PoolingHttpClientConnectionManager cm =
        new PoolingHttpClientConnectionManager();
cm.setMaxTotal(200); // increase max total connection to 200
cm.setDefaultMaxPerRoute(20); // increase max connection per route to 20
CloseableHttpClient httpClient = HttpClients.custom()
        .setConnectionManager(cm)
        .build();

String[] urisToGet = { ... };
// start a thread for each URI
// (if there are many URIs, a thread pool would be better)
Thread[] threads = new Thread[urisToGet.length];
for (int i = 0; i < threads.length; i++) {
    HttpGet httpget = new HttpGet(urisToGet[i]);
    threads[i] = new Thread(new GetTask(httpClient, httpget));
    threads[i].start();
}
// wait for all the threads to finish
for (int i = 0; i < threads.length; i++) {
    threads[i].join();
}

class GetTask implements Runnable {
    private final CloseableHttpClient httpClient;
    private final HttpContext context;
    private final HttpGet httpget;

    public GetTask(CloseableHttpClient httpClient, HttpGet httpget) {
        this.httpClient = httpClient;
        this.context = HttpClientContext.create();
        this.httpget = httpget;
    }

    @Override
    public void run() {
        try {
            CloseableHttpResponse response = httpClient.execute(
                httpget, context);
            try {
                HttpEntity entity = response.getEntity();
            } finally {
                response.close();
            }
        } catch (ClientProtocolException ex) {
            // handle protocol errors
        } catch (IOException ex) {
            // handle I/O errors
        }
    }
}

多线程将帮助使链接饱和（保持为因为当一个线程正在发送请求时，其他线程可以接收响应并利用下行链路。

Multithreading will help saturate the link (keep as much data flowing as possible) because while one thread is sending a request, other threads can be receiving responses and utilizing the downlink.

HTTP / 1.1支持管道传输，它可以在单个上发送多个请求连接而不等待响应。 基于NIO教程的异步I / O 3.10一节中的示例。流水线请求执行：

HTTP/1.1 supports pipelining, which sends multiple requests on a single connection without waiting for the responses. The Asynchronous I/O based on NIO tutorial has an example in section 3.10. Pipelined request execution:

HttpProcessor httpproc = <...>
HttpAsyncRequester requester = new HttpAsyncRequester(httpproc);
HttpHost target = new HttpHost("www.apache.org");
List<BasicAsyncRequestProducer> requestProducers = Arrays.asList(
    new BasicAsyncRequestProducer(target, new BasicHttpRequest("GET", "/index.html")),
    new BasicAsyncRequestProducer(target, new BasicHttpRequest("GET", "/foundation/index.html")),
    new BasicAsyncRequestProducer(target, new BasicHttpRequest("GET", "/foundation/how-it-works.html"))
);
List<BasicAsyncResponseConsumer> responseConsumers = Arrays.asList(
    new BasicAsyncResponseConsumer(),
    new BasicAsyncResponseConsumer(),
    new BasicAsyncResponseConsumer()
);
HttpCoreContext context = HttpCoreContext.create();
Future<List<HttpResponse>> future = requester.executePipelined(
    target, requestProducers, responseConsumers, pool, context, null);

HttpCore示例（管道化HTTP GET请求）。

There is a full version of this example in the HttpCore Examples ("Pipelined HTTP GET requests").

较旧的Web服务器可能无法正确处理流水线请求。

Older web servers may be unable to handle pipelined requests correctly.

这篇关于如何在java中进行批量http调用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！