反应式WebClient不发出响应 [英] Reactive WebClient not emitting a response

问题描述

我对Spring Reactive WebClient有疑问... 几天前，我决定在Spring Framework中使用新的反应式内容，并做了一个小项目，仅出于个人目的抓取数据. (对一个网页发出多个请求并合并结果).

I have a question about Spring Reactive WebClient... Few days ago I decided to play with the new reactive stuff in Spring Framework and I made one small project for scraping data only for personal purposes. (making multiple requests to one webpage and combining the results).

我开始使用新的反应式WebClient发出请求，但是发现的问题是客户端没有为每个请求发出响应.听起来很奇怪.这是我为获取数据所做的工作:

I started using the new reactive WebClient for making requests but the problem I found is that the client not emitting response for every request. Sounds strange. Here is what I did for fetching data:

private Mono<String> fetchData(String uri) {
    return this.client
            .get()
            .uri(uri)
            .header("X-Fsign","SW9D1eZo")
            .retrieve()
            .bodyToMono(String.class)
            .timeout(Duration.ofSeconds(35))
            .log("category", Level.ALL, SignalType.ON_ERROR, SignalType.ON_COMPLETE, SignalType.CANCEL, SignalType.REQUEST);
}

以及调用fetchData的函数:

public Mono<List<Stat>> fetch() {
    return fetchData(URL)
            .map(this::extractUrls)
            .doOnNext(System.out::println)
            .doOnNext(s-> System.out.println("all ids are "+s.size()))
            .flatMapIterable(q->q)
            .map(s -> s.substring(7, 15))
            .map(s -> "http://d.flashscore.com/x/feed/d_hh_" + s + "_en_1") // list of N-length urls
            .flatMap(this::fetchData)
            .map(this::extractHeadToHead)
            .collectList();
}

和订户:

    FlashScoreService bean = ctx.getBean(FlashScoreService.class);
    bean.fetch().subscribe(s->{
        System.out.println("finished !!! " + s.size()); //expecting same N-length list size
    },Throwable::printStackTrace);

问题是如果我发出的请求多于100个. 我没有收到所有请求的响应，没有抛出错误，也没有返回错误响应代码，并且以不同于请求数量的大小调用了subscribe方法.

The problem is if I made a little bit more requests > 100. I didn't get responses for all of them, no error is thrown or error response code is returned and subscribe method is invoked with size different from the number of requests.

我的请求基于字符串列表(URL)，并且在发出所有响应之后，我应该将所有请求作为列表接收，因为我正在使用collectList().当我执行100个请求时，我希望收到100个响应的列表，但实际上我有时会收到100个，有时会收到96个等等. 这很容易复制，这是我的github项目链接.

The requests I made are based on List of Strings (urls) and after all responses are emitted I should receive all of them as list because I'm using collectList(). When I execute 100 requests, I expect to receive list of 100 responses but actually I'm receiving sometimes 100, sometimes 96 etc ... May be something fails silently. This is easy reproducible here is my github project link.

示例输出:

all ids are 176
finished !!! 171

请给我一些建议，告诉我如何调试或做错了什么.感谢您的帮助.

Please give me suggestions how I can debug or what I'm doing wrong. Help is appreciated.

更新:

日志显示例如我是否传递了126个网址:

The log shows if I pass 126 urls for example:

onNext(ReactorClientHttpResponse{request=[GET/some_url],status=200}) is called 121 times. May be here is the problem.
onComplete() is called 126 times which is the exact same length of the passed list of urls

但是在不调用onNext()或onError()的情况下如何完成某些请求呢? (单声道中的成功和错误)

but how it's possible some of the requests to be completed without calling onNext() or onError( ) ? (success and error in Mono)

我认为问题不在WebClient中，而在其他地方.环境或服务器阻止了该请求，但可能是我应该收到一些错误日志.

I think the problem is not in the WebClient but somewhere else. Environment or server blocking the request, but may be I should receive some error log.

ps.感谢您的帮助！

ps. Thanks for the help !

推荐答案

这是一个棘手的问题.调试收到的实际HTTP帧后，似乎我们确实没有收到某些请求的响应.使用Wireshark进行更多调试后，看起来远程服务器正在请求结束FIN, ACK TCP数据包的连接，并且客户端已确认该连接.问题在于，该连接仍从池中获取，以在第一个FIN, ACK TCP数据包之后发送另一个GET请求.

This is a tricky one. Debugging the actual HTTP frames received, it seems we're really not getting responses for some requests. Debugging a little more with Wireshark, it looks like the remote server is requesting the end of the connection with a FIN, ACK TCP packet and that the client acknowledges it. The problem is this connection is still taken from the pool to send another GET request after the first FIN, ACK TCP packet.

也许远程服务器在处理了许多请求之后就关闭了连接；在任何情况下，这都是完全合法的行为.请注意，我并没有始终如一地重现这一点.

Maybe the remote server is closing connections after they've served a number of requests; in any case it's perfectly legal behavior. Note that I'm not reproducing this consistently.

您可以在客户端上禁用连接池；这会变慢，并且显然不会触发此问题.为此，请使用以下命令:

You can disable connection pooling on the client; this will be slower and apparently doesn't trigger this issue. For that, use the following:

this.client = WebClient.builder()
                .clientConnector(new ReactorClientHttpConnector(new Consumer<HttpClientOptions.Builder>() {
                    @Override
                    public void accept(HttpClientOptions.Builder builder) {
                        builder.disablePool();
                    }
                }))
                .build();

根本问题

根本问题是在关闭TCP连接而不发送响应时，HTTP客户端不应该onComplete.或更妙的是，HTTP客户端在关闭时不应重用连接.当我了解更多信息时，我会在这里报告.

Underlying issue

The root problem is that the HTTP client should not onComplete when the TCP connection is closed without sending a response. Or better, the HTTP client should not reuse a connection while it's being closed. I'll report back here when I'll know more.

这篇关于反应式WebClient不发出响应的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！