Webflux上传大文件导致Java堆空间 [英] Webflux upload large files cause Java heap space
问题描述
可能没有多少开发人员面临与我相同的问题.
但是我想分享已经解决了将近1个月的解决方案.
我使用Kubernetes和docker-compose，此Webflux服务(容器)的内存限制设置为1g mem_limit:1g 我不允许增加内存限制.
Probably not many developers are facing the problem the same as me.
But I want to share the solution which I had been solved for almost 1 month.
I use Kubernetes and docker-compose, this Webflux service (container) is set the memory limit 1g mem_limit: 1g I am not allowed to increase the memory limit.
coRouter 已用作容器中的控制器.
coRouter has been used as a controller which is in the container.
@Configuration
class WebController(
) {
@Bean
fun endpoints() = coRouter {
contentType(MediaType.MULTIPART_FORM_DATA).nest {
POST("/uploadFile") { requestParm ->
requestParm.awaitMultipartData().multipartFormData["file"]?.first()
}
}
}
}
如果我使用API上传文件 http://localhost:9004/uploadFile 100MB，10个文件(使用JS Ajax)
它将产生 java.lang.OutOfMemoryError:Java堆空间，因为它没有足够的内存.
每当您调用 requestParm.awaitMultipartData()时，它将把这些字节流式传输到内存中.
If I upload files using the API
http://localhost:9004/uploadFile 100MB, 10 files (using JS Ajax)
It will produce java.lang.OutOfMemoryError: Java heap space because it doesn't have enough memory.
Whenever you call requestParm.awaitMultipartData() it will stream those bytes into memory.
推荐答案
我意识到，我当前的代码将直接将上传文件存储在内存中.有两种方法可以解决此问题.
I realized that, my current code will store the upload files in memory directly. There are 2 ways to solve this problem.
- 第一种方法，在路由器或控制器上添加
@EnableWebFlux
- 1st way, add
@EnableWebFluxon your Router or Controller
@Configuration
@EnableWebFlux
class WebController(
) { }
- 第二种方式，从
DelegatingWebFluxConfiguration扩展到路由器或控制器
参考:https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#webflux-config-advanced-java - 2nd way, extending from
DelegatingWebFluxConfigurationto your Router or Controller
Reference: https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#webflux-config-advanced-java
@Configuration
class WebController(
): DelegatingWebFluxConfiguration() { }
为什么可以解决上传大文件的问题?
Why does it solve the uploading large file problem?
以上两种方法将自动使用晚餐类的方法
configureHttpMessageCodecs(configurer:ServerCodecConfigurer)
https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#webflux-codecs-multipart
These 2 ways above will be automatically used a supper class's method configureHttpMessageCodecs(configurer: ServerCodecConfigurer)
https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#webflux-codecs-multipart
默认情况下，使用DefaultPartHttpMessageReader，但是可以通过ServerCodecConfigurer进行更改.有关DefaultPartHttpMessageReader的更多信息，请参考DefaultPartHttpMessageReader的javadoc.
By default, the DefaultPartHttpMessageReader is used, but this can be changed through the ServerCodecConfigurer. For more information about the DefaultPartHttpMessageReader, refer to to the javadoc of DefaultPartHttpMessageReader.
如您所见， ServerCodecConfigurer 将默认使用DefaultPartHttpMessageReader.
As you can see ServerCodecConfigurer will use DefaultPartHttpMessageReader by default.
DefaultPartHttpMessageReader 具有此重要属性 MaxInMemorySize (如果内存已满，则存储在磁盘中)
DefaultPartHttpMessageReader has this important properties MaxInMemorySize (if memory full, store in disk)
