处理大数据的XMLHttpRequest响应 [英] Handle XMLHttpRequest response with large data
问题描述
使用Chrome，我设置了一个 XMLHttpRequest ：
$ b
const xhr = new XMLHttpRequest（）;
xhr.open（method，url，true）;
...
xhr.onreadystatechange =（）=> {
if（xhr.readyState === XMLHttpRequest.DONE）{
if（isStatusCodeSuccess（xhr.status））{
//句柄响应;
} else {
//处理错误
}
}
};
xhr.addEventListener（'progress'，event => {
console.log（event.loaded，event.target.response.length）;
}）;
xhr.send（data）;
在某些时候，我请求了一些JSON数据，总大小约为295MB未压缩/ 5.8 MB压缩。不幸的是，当成功调用以下代码时，响应是一个空字符串。
使用进度事件侦听器，我可以看到响应被正确处理大块，直到某一点。下面是它生成的控制台日志的摘录：
$ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ 7376739 $ 7376739
11525403 11525403
...
261489180 261489180
264684030 264684030
267880247 267880247
271037819 0
274232442 0
277428774 0
...
304018210 0
309230213 0
310445469 0
<看起来像Chrome上有一个字符串/分配的限制，但我没有收到任何错误。
在Firefox上我收到以下错误： InternalError：分配大小溢出。
我试图存储结果，但是我不能清空Xml Http请求对象作为属性是只读的。
浏览器中变量的官方限制是什么？我找不到正式的答案，只是一些实验性的答案 Javascript字符串大小限制：对我来说是256 MB - 是所有浏览器都一样吗？有什么办法解决这个问题，并处理大的POST结果？除了使用WebSocket。
我想：
application / octet-stream
您可以尝试使用获取API ，改进版本XMLHttpRequest使用直接承诺。它具有流式响应体的功能，而不是将整个身体响应作为文本进行缓冲。
基本上，使用 progress 事件，你在每个块上被调用，但是xhr响应不断累积响应，因为文本达到了内存限制。
获取API ，您可以在到达时读取收到的字节：
const consume = responseReader => {
return responseReader.read（）。then（result => {
if（result.done）{return;}
//对当前块做些什么
const chunk = result.value;
return consume（responseReader）;
}）;
//执行请求并使用响应流
fetch（url）.then（response => {
return consume（response.body.getReader （））;
}）
.catch（console.log.bind（console））;
欲了解更多信息，请参阅此文章： Fetch（或XHR的不可否认的局限性）
<通过Fetch API流式传输响应主体只有在编写本文之时才支持Chrome 43和Edge 14。 Cf 兼容性表。
Using Chrome, I setup a XMLHttpRequest:
const xhr = new XMLHttpRequest();
xhr.open(method, url, true);
...
xhr.onreadystatechange = () => {
if (xhr.readyState === XMLHttpRequest.DONE) {
if (isStatusCodeSuccess(xhr.status)) {
// handle response;
} else {
// handle error
}
}
};
xhr.addEventListener('progress', event => {
console.log(event.loaded, event.target.response.length);
});
xhr.send(data);
At some point I request some JSON data which total size is around 295MB uncompressed / 5.8 MB compressed. Unfortunately, when the following code is called on success, the response is an empty string.
Using the progress event listener, I can see that the response is correctly handled chunk by chunk, up to a certain point. Here is an excerpt of the console logs it produces:
32768 32768
4639135 4639135
7376739 7376739
11525403 11525403
...
261489180 261489180
264684030 264684030
267880247 267880247
271037819 0
274232442 0
277428774 0
...
304018210 0
309230213 0
310445469 0
It looks like there is a string/allocation limitation on Chrome but I don't receive any error.
On firefox I receive the following error: InternalError: allocation size overflow.
I tried storing the result as it comes but I can't "empty" the Xml Http Request object as attributes are readonly.
What is the official limitation for variables in browsers? I could not find an official answer, just some experimental ones Javascript string size limit: 256 MB for me - is it the same for all browsers?
Is there any way to workaround this issue and handle large POST result? apart from using WebSocket.
I am thinking of:
- specific content type to stream the data, like the
application/octet-stream - specific parameter to handle the data by chunk, like the range header although the content I'm fetching is not static (it changes over time)
- known algorithm to reduce the length of the JSON response, where I can unzip chunk by chunk the data to avoid hitting the above limitation
You can try using the Fetch API, an improved version of the XMLHttpRequest using directly promises. It comes with a feature to stream the response body instead of buffering the whole body response as text.
Basically, using the progress event, you are called on each chunk but the xhr response keeps accumulating the response as text thus reaching memory limit.
With the Fetch API, you can read the received bytes as they arrive:
const consume = responseReader => {
return responseReader.read().then(result => {
if (result.done) { return; }
// do something with the current chunk
const chunk = result.value;
return consume(responseReader);
});
}
// Perform the request and consume response stream
fetch(url).then(response => {
return consume(response.body.getReader());
})
.catch(console.log.bind(console));
For more information see this article: Fetch (or the undeniable limitations of XHR)
Streaming the response body through Fetch API is only supported since Chrome 43, and Edge 14 at the time of writing. Cf compatibility table.
这篇关于处理大数据的XMLHttpRequest响应的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
