如何在java中进行批量http调用 [英] How to make batch http call in java

问题描述

我正在尝试通过 Http 访问另一个服务以使用 HttpClient 获取数据.uri 应该看起来像端点:80/.../itemId.

I am trying to access another service by Http to fetch data using HttpClient. The uri should look like endpoint:80/.../itemId.

我想知道是否有办法进行批量调用以指定一组 itemId?我确实发现有人在创建请求时建议 .setHeader(HttpHeaders.CONNECTION, "keep-alive") .通过这样做，如何在获取所有数据后释放客户端?

I am wondering if there is a way to make a batch call to specify a set of itemIds? I did find someone suggests to .setHeader(HttpHeaders.CONNECTION, "keep-alive") when creating request. Via doing this, how could I release the client after getting all data?

另外，似乎这种方法仍然需要获得一个响应然后发送另一个请求?这是否可以异步执行，以及如何执行?顺便说一句，在这种情况下，由于某种原因，我似乎无法使用 AsyncHttpClient.

Also, seems like this method still need to get one response then send another request? Is this possible to do it as async, and how? BTW, seems like I could not use AsyncHttpClient in this case for some reason.

由于我对 HttpClient 几乎一无所知，所以这个问题可能看起来很愚蠢.真的希望有人能帮我解决问题.

Since I am barely know nothing about HttpClient, the question may look dumb. Really hope someone could help me solve the problem.

推荐答案

服务器上的 API 支持

API 支持一次请求多个 ID 的可能性很小(例如使用 http://endpoint:80/.../itemId1,itemId2,itemId3 形式的 URL)).检查 API 文档以了解这是否可用，因为如果可用，那将是最佳解决方案.

API support on the server

There is a small chance that the API supports requesting multiple IDs at a time (e.g. using a URL of the form http://endpoint:80/.../itemId1,itemId2,itemId3). Check the API documentation to see if this is available, because if so that would be the best solution.

看起来 Apache HttpClient 默认使用持久(保持活动")连接(请参阅 连接管理教程链接在@kichik 的评论).日志记录工具 可以帮助验证连接是否重复用于多个请求.

It looks like Apache HttpClient uses persistent ("keep alive") connections by default (see the Connection Management tutorial linked in @kichik's comment). The logging facilities could help to verify that connections are reused for several requests.

要释放客户端，请使用 close() 方法.来自 2.3.4.连接管理器关闭:

To release the client, use the close() method. From 2.3.4. Connection manager shutdown:

当不再需要 HttpClient 实例并且即将超出范围时，关闭其连接管理器非常重要，以确保管理器保持活动的所有连接都关闭并释放这些连接分配的系统资源.

When an HttpClient instance is no longer needed and is about to go out of scope it is important to shut down its connection manager to ensure that all connections kept alive by the manager get closed and system resources allocated by those connections are released.

CloseableHttpClient httpClient = <...>
httpClient.close();

持久连接消除了建立新连接的开销，但正如您所指出的，客户端在发送下一个请求之前仍将等待响应.

Persistent connections eliminate the overhead of establishing new connections, but as you've noted the client will still wait for a response before sending the next request.

您可以使您的程序多线程并使用 PoolingHttpClientConnectionManager 控制与服务器建立的连接数.这是一个基于 2.3.3 的示例.池连接管理器和2.4.多线程请求执行:

You can make your program multithreaded and use a PoolingHttpClientConnectionManager to control the number of connections made to the server. Here is an example based on 2.3.3. Pooling connection manager and 2.4. Multithreaded request execution:

import java.io.*;
import org.apache.http.*;
import org.apache.http.client.*;
import org.apache.http.client.methods.*;
import org.apache.http.client.protocol.*;
import org.apache.http.impl.client.*;
import org.apache.http.impl.conn.*;
import org.apache.http.protocol.*;

// ...
PoolingHttpClientConnectionManager cm =
        new PoolingHttpClientConnectionManager();
cm.setMaxTotal(200); // increase max total connection to 200
cm.setDefaultMaxPerRoute(20); // increase max connection per route to 20
CloseableHttpClient httpClient = HttpClients.custom()
        .setConnectionManager(cm)
        .build();

String[] urisToGet = { ... };
// start a thread for each URI
// (if there are many URIs, a thread pool would be better)
Thread[] threads = new Thread[urisToGet.length];
for (int i = 0; i < threads.length; i++) {
    HttpGet httpget = new HttpGet(urisToGet[i]);
    threads[i] = new Thread(new GetTask(httpClient, httpget));
    threads[i].start();
}
// wait for all the threads to finish
for (int i = 0; i < threads.length; i++) {
    threads[i].join();
}

class GetTask implements Runnable {
    private final CloseableHttpClient httpClient;
    private final HttpContext context;
    private final HttpGet httpget;

    public GetTask(CloseableHttpClient httpClient, HttpGet httpget) {
        this.httpClient = httpClient;
        this.context = HttpClientContext.create();
        this.httpget = httpget;
    }

    @Override
    public void run() {
        try {
            CloseableHttpResponse response = httpClient.execute(
                httpget, context);
            try {
                HttpEntity entity = response.getEntity();
            } finally {
                response.close();
            }
        } catch (ClientProtocolException ex) {
            // handle protocol errors
        } catch (IOException ex) {
            // handle I/O errors
        }
    }
}

多线程将有助于使链路饱和(保持尽可能多的数据流动)，因为当一个线程发送请求时，其他线程可以接收响应并利用下行链路.

Multithreading will help saturate the link (keep as much data flowing as possible) because while one thread is sending a request, other threads can be receiving responses and utilizing the downlink.

HTTP/1.1 支持 pipelining，它在单个连接上发送多个请求而无需等待响应.基于NIO教程的异步I/O有3.10 部分中的示例.流水线请求执行:

HTTP/1.1 supports pipelining, which sends multiple requests on a single connection without waiting for the responses. The Asynchronous I/O based on NIO tutorial has an example in section 3.10. Pipelined request execution:

HttpProcessor httpproc = <...>
HttpAsyncRequester requester = new HttpAsyncRequester(httpproc);
HttpHost target = new HttpHost("www.apache.org");
List<BasicAsyncRequestProducer> requestProducers = Arrays.asList(
    new BasicAsyncRequestProducer(target, new BasicHttpRequest("GET", "/index.html")),
    new BasicAsyncRequestProducer(target, new BasicHttpRequest("GET", "/foundation/index.html")),
    new BasicAsyncRequestProducer(target, new BasicHttpRequest("GET", "/foundation/how-it-works.html"))
);
List<BasicAsyncResponseConsumer> responseConsumers = Arrays.asList(
    new BasicAsyncResponseConsumer(),
    new BasicAsyncResponseConsumer(),
    new BasicAsyncResponseConsumer()
);
HttpCoreContext context = HttpCoreContext.create();
Future<List<HttpResponse>> future = requester.executePipelined(
    target, requestProducers, responseConsumers, pool, context, null);

HttpCore 示例(流水线式 HTTP GET 请求").

There is a full version of this example in the HttpCore Examples ("Pipelined HTTP GET requests").

较旧的 Web 服务器可能无法正确处理流水线请求.

Older web servers may be unable to handle pipelined requests correctly.

这篇关于如何在java中进行批量http调用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！