通过@Bean 提供的 RestTemplateBuilder 流式上传缓冲完整文件 [英] Streaming upload via @Bean-provided RestTemplateBuilder buffers full file
问题描述
我正在构建一个用于上传大文件(数 GB)的反向代理，因此想要使用不缓冲整个文件的流模型.大缓冲区会引入延迟，更重要的是，它们可能会导致内存不足错误.
I'm building a reverse-proxy for uploading large files (multiple gigabytes), and therefore want to use a streaming model that does not buffer entire files. Large buffers would introduce latency and, more importantly, they could result in out-of-memory errors.
我的客户端类包含
@Autowired private RestTemplate restTemplate;
@Bean
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
int REST_TEMPLATE_MODE = 1; // 1=streams, 2=streams, 3=buffers
return
REST_TEMPLATE_MODE == 1 ? new RestTemplate() :
REST_TEMPLATE_MODE == 2 ? (new RestTemplateBuilder()).build() :
REST_TEMPLATE_MODE == 3 ? restTemplateBuilder.build() : null;
}
和
public void upload_via_streaming(InputStream inputStream, String originalname) {
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);
InputStreamResource inputStreamResource = new InputStreamResource(inputStream) {
@Override public String getFilename() { return originalname; }
@Override public long contentLength() { return -1; }
};
MultiValueMap<String, Object> body = new LinkedMultiValueMap<String, Object>();
body.add("myfile", inputStreamResource);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body,headers);
String response = restTemplate.postForObject(UPLOAD_URL, requestEntity, String.class);
System.out.println("response: "+response);
}
这是有效的，但请注意我的 REST_TEMPLATE_MODE 值控制它是否满足我的流媒体要求.
This is working, but notice my REST_TEMPLATE_MODE value controls whether or not it meets my streaming requirement.
问题:为什么 REST_TEMPLATE_MODE == 3 会导致全文件缓冲?
Question: Why does REST_TEMPLATE_MODE == 3 result in full-file buffering?
参考文献:
- 如何使用 RestTemplate 转发大文件?
- 如何使用restTemplate Spring-mvc发送Multipart表单数据
- Spring - 如何将大型分段文件上传流式传输到数据库而不存储在本地文件系统上 -- 建立 InputStream
- 如何使用注释自动装配 RestTemplate
- 设计说明和使用注意事项，还有:
restTemplate不支持流式传输下载
- How to forward large files with RestTemplate?
- How to send Multipart form data with restTemplate Spring-mvc
- Spring - How to stream large multipart file uploads to database without storing on local file system -- establishing the InputStream
- How to autowire RestTemplate using annotations
- Design notes and usage caveats, also:
restTemplatedoes not support streaming downloads
推荐答案
简而言之，Spring Boot 以 @Bean 形式提供的 RestTemplateBuilder 实例包含一个拦截器(过滤器)与执行器/指标相关联——并且拦截器接口需要将请求主体缓冲到一个简单的 byte[] 中.
In short, the instance of RestTemplateBuilder provided as an @Bean by Spring Boot includes an interceptor (filter) associated with actuator/metrics -- and the interceptor interface requires buffering of the request body into a simple byte[].
如果您从头开始实例化您自己的 RestTemplateBuilder 或 RestTemplate，默认情况下它不会包含此内容.
If you instantiate your own RestTemplateBuilder or RestTemplate from scratch, it won't include this by default.
我似乎是唯一访问此帖子的人，但为了在我开始发布完整解决方案之前对某人有所帮助，我发现了一个重要线索:
I seem to be the only person visiting this post, but just in case it helps someone before I get around to posting a complete solution, I've found a big clue:
restTemplate.getInterceptors().forEach(item->System.out.println(item));
显示...
org.SF.boot.actuate.metrics.web.client.MetricsClientHttpRequestInterceptor
如果我通过setInterceptors清除拦截器列表，问题就解决了.此外，我发现任何拦截器，即使它只执行 NOP，也会引入全文件缓冲.
If I clear the interceptor list via setInterceptors, it solves the problem. Furthermore, I found that any interceptor, even if it only performs a NOP, will introduce full-file buffering.
我已经明确地设置了 bufferRequestBody = false，但是如果使用拦截器，显然这个代码会被绕过.要是早点知道就好了...
I have explicitly set bufferRequestBody = false, but apparently this code is bypassed if interceptors are used. This would have been nice to know earlier...
@Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
HttpURLConnection connection = openConnection(uri.toURL(), this.proxy);
prepareConnection(connection, httpMethod.name());
if (this.bufferRequestBody) {
return new SimpleBufferingClientHttpRequest(connection, this.outputStreaming);
}
else {
return new SimpleStreamingClientHttpRequest(connection, this.chunkSize, this.outputStreaming);
}
}
公共抽象类 InterceptingHttpAccessor extends HttpAccessor { ...
这表明如果 interceptors 列表不为空，则使用 InterceptingClientHttpRequestFactory.
public abstract class InterceptingHttpAccessor extends HttpAccessor { ...
This shows that the InterceptingClientHttpRequestFactory is used if the list of interceptors is not empty.
/**
* Overridden to expose an {@link InterceptingClientHttpRequestFactory}
* if necessary.
* @see #getInterceptors()
*/
@Override
public ClientHttpRequestFactory getRequestFactory() {
List<ClientHttpRequestInterceptor> interceptors = getInterceptors();
if (!CollectionUtils.isEmpty(interceptors)) {
ClientHttpRequestFactory factory = this.interceptingRequestFactory;
if (factory == null) {
factory = new InterceptingClientHttpRequestFactory(super.getRequestFactory(), interceptors);
this.interceptingRequestFactory = factory;
}
return factory;
}
else {
return super.getRequestFactory();
}
}
class InterceptingClientHttpRequest 扩展 AbstractBufferingClientHttpRequest { ...
接口清楚地表明，使用InterceptingClientHttpRequest 需要将body 缓冲到byte[].没有使用流接口的选项.
class InterceptingClientHttpRequest extends AbstractBufferingClientHttpRequest { ...
The interfaces make it clear that using InterceptingClientHttpRequest requires buffering body to a byte[]. There is not an option to use a streaming interface.
@Override
public ClientHttpResponse execute(HttpRequest request, byte[] body) throws IOException {
这篇关于通过@Bean 提供的 RestTemplateBuilder 流式上传缓冲完整文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
