从带有 spring 集成的分页 REST 服务中读取和下载 [英] Read and download from a paginated REST-Services with spring integration

问题描述

目前我正在开发具有以下场景的 Spring Integration 应用程序:

Currently I am working on a Spring Integration application which has a following scenario:

1. 一个 int-http:outbound-gateway 从 REST-Services 读取分页元素列表:about in
2. 将每个页面内容拆分并存储在一个文件夹中，以便稍后由 Spring 批处理作业进行处理.

1. An int-http:outbound-gateway read from a REST-Services a list of paginated elements: about in
2. Each page content splitted and stored in a folder to be processed later by a spring batch job.

我对 spring-integration 很陌生，我不知道是否可以创建一种带有 `int-http:outbound-gateway' 的循环来读取所有页面，直到最后一个页面.

I'm quite new with spring-integration and I don't know if it's possibile to create a kind of loop with `int-http:outbound-gateway' to read all pages until the last one.

我们讨论的是分成 2651 页的 66254 个元素.我正在寻找的是在没有任何内存问题的情况下阅读和下载所有页面并收集数据的最佳实践.

We're talking about 66254 elements splitted in 2651 pages. What I'm looking for is a best practice to read and download all pages and collecting data without have any memory issue.

任何建议将不胜感激

谢谢

推荐答案

是的，这是可能的，虽然有点棘手.

Yes, it is possible, although a bit tricky.

假设您的 REST 服务需要 page 作为请求参数，因此，您希望从页面 #1 发出请求并循环(增加 page 参数)直到服务返回空结果.

Assume your REST service require page as request param, so, you would like to make a request from the page #1 and loop (increment page param) until the service returns empty result.

因此，您可能对 REST 服务进行了配置，例如:

So, you may have configuration for the REST service like:

<int-http:outbound-gateway url="http://service/elements?page={page}">
    <int-http:uri-variable name="page" expression="headers['page']"/>
</int-http:outbound-gateway>

注意 定义.从一开始，您就必须将消息发送到此 ，并将 page 标头作为 1.

Pay attention to that <int-http:uri-variable> definition. From the beginning you have to send the message to this <int-http:outbound-gateway> with the page header as a 1.

您应该将此网关的回复发送到诸如 或 之类的内容，其中之一订阅者仍然是您的 splitter 来将项目存储到文件夹中.

The reply from this gateway you should send to something like <recipient-list-router>, or <publish-subscribe-channel>, where one of the subscriber is still your splitter to to store items into the folder.

另一个订阅者有点聪明.它从 <filter> 开始检查 payload(来自 REST 调用的结果)是否为空，这意味着我们已经完成并且没有更多关于服务的页面取回.否则，您使用 来增加和替换 page 标头并将结果发送到我们的第一个 ;.

Another subscriber is a bit smart. It starts from <filter> to check if the payload (a result from the REST call) is empty, meaning that we have done and no more pages on the service to retrieve. Otherwise you use <header-enricher> to increment and replace the page header and send the result into that our first <int-http:outbound-gateway>.

这篇关于从带有 spring 集成的分页 REST 服务中读取和下载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！