将 BASE64 中的文件加载到服务器时改造 OutOfMemory 异常 [英] Retrofit OutOfMemory exception while loading a files in BASE64 to server

问题描述

当我尝试在 base64 中加载 pdf 文件作为模型 throw Retrofit 中的字段时出现 OOM 异常.我知道，这不是上传文件的正常方式，但这不是第三方实现.我该如何解决此类问题?

I had an OOM exception when tried to load a pdf file in base64 as a field in a model throw Retrofit. I know, than this is not a normal way to upload files, but it's not a third party realisation. How I can fix this kind of a problem?

当我关闭网络连接时应用程序崩溃

An application crached when I switching off my network connection

Failed to allocate a 30544558 byte allocation with 2085152 free bytes and 26MB until OOM

@Streaming
@POST("/api/order/")
fun makeOrder(@Header("Authorization") token: String, @Body order: OrderMainModel): Single<Response<PhoneNumberResponse>>

推荐答案

这是一种非常疯狂的文件发送方式.但是尽管它是错误的(正如您通常使用 @Multipart 做这些事情一样)，我发现您的问题是一个有趣的练习.我的解决方案充满了技巧，但如果您绝对确定不能以任何方式影响 API，那么也许这可以帮助您.

That's a pretty crazy way of sending files. But despite it being wrong (as you normally do those things with @Multipart), I found your problem an interesting exercise. My solution is full of hacks, but if you're absolutely sure you can't influence the API in any way, then maybe this can help you.

您需要一个 InputStream.没有其他方法可以在不最终耗尽内存的情况下发送文件.但是如何在这样的请求中发送 InputStream ?

You'll need an InputStream. There's no other way you'll be able to send files without eventually running out of memory. But how to send an InputStream in a request like this?

我假设 OrderMainModel 有一个 String 字段来保存你巨大的 Base64 字符串.我首先将其更改为 File(或 InputStream 本身).

I assume OrderMainModel has a String field that keeps your giant Base64 string. I'd start by changing this to the File (or an InputStream itself).

现在，假设您正在使用 Gson(这是一个非常大胆的假设，但这是我使用的解析器 - 我很确定您可以使用任何合理的 json 库实现类似的功能)，创建一个自定义 TypeAdapter 用于类型 File.这个 TypeAdapter 会强制你实现这个接口:

Now, assuming you're using Gson (that's a pretty bold assumption but that's a parser I use - I'm pretty sure you can achieve something similar with any reasonable json library), create a custom TypeAdapter for type File. This TypeAdapter will force you to implement this interface:

class Adapter : TypeAdapter<File>() {
    override fun write(out: JsonWriter, value: File) {
        // implement this

    }

    override fun read(`in`: JsonReader): File {
        // ignore
    }
}

你可以不理会read，你不需要它.

You can leave read alone, you won't need it.

现在，你在write方法中要做的就是逐个读取，不断写入JsonWriter.啊，无论您阅读什么，您都需要将其即时转换为 base64.android.util 中有一个 Base64InputStream 可用，但它似乎不能编码，你可以使用这个，来自 commons-codec:https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html(记得将 true 传递给 doEncode 在构造函数中，否则它将在解码模式下工作).

Now, what you have to do in the write method is to read it part by part, continuously writing it to JsonWriter. Ah, and whatever you read, you'll need to convert it to base64 on the fly. There's a Base64InputStream available in android.util but it doesn't seem to be capable of encoding, you can use this one, from commons-codec: https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html (remember to pass true to doEncode in a constructor, otherwise it'll work in decoding mode).

现在，您要做的就是让 Base64InputStream 包装 FileInputStream，它包装您在 TypeAdapter<中收到的 File/代码>.

Now, everything you have to do is to make Base64InputStream wrap FileInputStream, which wraps File you received in your TypeAdapter.

// 0 means no new lines in your Base64. null means no line separator. 
Base64InputStream(FileInputStream(file)), true, 0, null)

然后慢慢地将它重新写回你的 JsonWriter.

And slowly re-write it back to your JsonWriter.

但是等等，这也不容易！

But wait, that's not easy either!

JsonWriter 没有提供以流式方式编写单个字符串(值)的任何合理方式.我唯一的想法就是用反射来破解它.

JsonWriter doesn't offer any reasonable way of writing a single string (value) in a streaming fashion. The only idea I have is to hack it with reflection.

为此，您需要检索JsonWriter 的内部out 对象，类型为Writer.然后，为了能够在不破坏 JsonWriter 内部保持的状态的情况下进行编写，您需要访问 Writer 的两个私有方法 - writeDeferredValue 和 beforeValue 并按顺序调用它们.一切都变得非常复杂和不安全.但是，嘿，一切都是为了好玩，不是吗?

In order to do that, you need to retrieve JsonWriter's inner out object, type of Writer. Then, in order to be able to write to do it without breaking the state that JsonWriter keeps internally, you need to get an access on Writer's two private methods - writeDeferredValue and beforeValue and invoke them in order. Everything gets quite complicated and unsafe. But hey, it's all about fun, isn't it?

这是一个展示想法的小型 PoC，但尚未准备好.;-)

Here's a small PoC that presents the idea, rather not prod ready. ;-)

fun main() {
    val model = Model(File("file.txt"))
    val gson = GsonBuilder().registerTypeAdapter(File::class.java, Adapter()).create()
    // System.out just for PoC, don't try with large files because output will be massive
    gson.toJson(model, System.out)
}

data class Model(@SerializedName("file") val file: File)

class Adapter : TypeAdapter<File>() {
    override fun write(out: JsonWriter, value: File) {
        Base64InputStream(FileInputStream(value), true, 0, null).use { 
            out.writeFromStream(it)
        }
    }

    override fun read(`in`: JsonReader): File {
        throw UnsupportedOperationException()
    }

    // JsonWriter only offers value(String), which would need you to load the whole file to the memory.
    private fun JsonWriter.writeFromStream(inputStream: InputStream) {
        val declaredField = javaClass.getDeclaredField("out")
        val deferredName = javaClass.getDeclaredMethod("writeDeferredName")
        val beforeValue = javaClass.getDeclaredMethod("beforeValue")

        declaredField.isAccessible = true
        deferredName.isAccessible = true
        beforeValue.isAccessible = true

        val actualWriter = declaredField.get(this) as Writer

        deferredName.invoke(this)
        beforeValue.invoke(this)
        actualWriter.write("\"")
        for (byte in inputStream.buffered()) {
            actualWriter.write(byte.toInt())
        }
        actualWriter.write("\"")
    }
}

您可能可以通过集成一些较低级别的 HTTP API 来实现类似的行为，这将使您无需反射即可写入 OutputStream.也许其他一些解析器(Jackson，也许?)会让它稍微方便一些.

You can probably achieve a similar behaviour by integrating some lower-level HTTP APIs, which will let you write to the OutputStream without reflection. Perhaps some other parsers (Jackson, maybe?) would make it slightly more convenient.

...或者只是为 API 更改而战.

... or just fight for the API change.

祝你好运！

这篇关于将 BASE64 中的文件加载到服务器时改造 OutOfMemory 异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！