如何更有效地通过http下载大文件? [英] How can I more efficently download large files over http?

问题描述

我正在尝试在Kotlin中下载大文件(<1GB)，因为我已经知道我正在使用okhttp，并且几乎都遵循了

I'm trying to download large files (<1GB) in Kotlin since I already knew I'm using okhttp and pretty much followed just used the answer from this question. Except that I'm using Kotlin instead of java, so the syntax is slightly diffrent.

val client = OkHttpClient()
val request = Request.Builder().url(urlString).build()
val response = client.newCall(request).execute()

val is = response.body().byteStream()

val input = BufferedInputStream(is)
val output = FileOutputStream(file)

val data = ByteArray(1024)
val total = 0L
val count : Int
do {
    count = input.read(data)
    total += count
    output.write(data, 0, count)
} while (count != -1)

output.flush()
output.close()
input.close()

之所以有效，是因为它在不占用过多内存的情况下下载文件，但似乎不必要地效率低下，因为它会不断尝试写入更多数据，而不知道是否有新数据到达. 在资源非常有限的VM上运行它时，我自己的测试似乎也证实了这一点，因为它似乎比python中的可比脚本使用了更多的CPU，同时获得了更低的下载速度，并且使用了wget.

That works in that it downloads the file without using too much memory but it seems needlessly ineffective in that it constantly tries to write more data without knowing if any new data arrived. That also seems confirmed with my own tests while running this on a very resource limited VM as it seems to use more CPU while getting a lower download speed then a comparable script in python, and of cause using wget.

我想知道是否有一种方法可以给我一些回调，如果x字节可用或者它是文件的末尾，则该回调被调用，因此我不必不断尝试来获取更多数据而无需知道是否有.

What I'm wondering if there is a way where I can give something a callback that gets called if x bytes are available or if it's the end of the file so I don't have to constantly try and get more data without knowing if there is any.

如果使用okhttp不可能，那么我使用其他东西没有问题，只是它是我惯用的http库.

If it's not possible with okhttp I don't have a problem using something else, it's just that it was the http library I'm used to.

推荐答案

从版本11开始，Java具有内置的

As of version 11, Java has a built-in HttpClient which implements

具有无阻塞背压的异步数据流

asynchronous streams of data with non-blocking back pressure

这就是您要让代码仅在有数据要处理时才运行的需求.

and that's what you need if you want your code to run only when there's data to process.

如果您有能力升级到Java 11，则可以使用 HttpResponse.BodyHandlers.ofFile 主体处理程序.您不必自己实现任何数据传输逻辑.

If you can afford to upgrade to Java 11, you'll be able to solve your problem out of the box, using the HttpResponse.BodyHandlers.ofFile body handler. You won't have to implement any data transfer logic on your own.

科特琳(Kotlin)示例:

Kotlin example:

fun main(args: Array<String>) {    
    val client = HttpClient.newHttpClient()

    val request = HttpRequest.newBuilder()
            .uri(URI.create("https://www.google.com"))
            .GET()
            .build()

    println("Starting download...")
    client.send(request, HttpResponse.BodyHandlers.ofFile(Paths.get("google.html")))
    println("Done with download.")
}

这篇关于如何更有效地通过http下载大文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！