如何发送和接收大型JSON数据 [英] How to send and receive large JSON data
问题描述
我对全栈开发相对较新，目前正试图找出一种有效的方法发送和 fetch 我的前端（React）和后端（Express）之间的大数据，同时最小化内存使用。具体来说，我正在构建一个映射应用程序，它需要我使用大型JSON文件（10-100mb）。
I'm relatively new to full-stack development, and currently trying to figure out an effective way to send and fetch large data between my front-end (React) and back-end (Express) while minimizing memory usage. Specifically, I'm building a mapping app which requires me to play around with large JSON files (10-100mb).
我目前的设置适用于较小的JSON文件：
My current setup works for smaller JSON files:
后端：
const data = require('../data/data.json');
router.get('/', function(req, res, next) {
res.json(data);
});
前端：
componentDidMount() {
fetch('/')
.then(res => res.json())
.then(data => this.setState({data: data}));
}
但是，如果数据大于~40mb，如果由于内存不足而在本地测试，后端会崩溃。另外，使用 require（）保留数据也需要相当多的内存。
However, if data is bigger than ~40mb, the backend would crash if I test on local due to running out of memory. Also, holding onto the data with require() takes quite a bit of memory as well.
我是做了一些研究并对JSON解析，字符串化，流式传输有了一般性的了解，我认为答案就在于使用chunked json流来逐位发送数据，但是在实现上却非常不知所措，特别是使用了单 fetch（）这样做（这是否可能？）。
I've done some research and have a general understanding of JSON parsing, stringifying, streaming, and I think the answer lies somewhere with using chunked json stream to send the data bit by bit, but am pretty much at a loss on its implementation, especially using a single fetch() to do so (is this even possible?).
绝对欣赏有关如何使用的任何建议接近这个。
Definitely appreciate any suggestions on how to approach this.
推荐答案
首先，40mb是巨大的，可能会对您的用户不体贴，特别是如果移动概率很高的话使用。
First off, 40mb is huge and can be inconsiderate to your users especially if there' s a high probability of mobile use.
如果可能的话，最好在后端收集这些数据，可能把它放到磁盘上，然后只向前端提供必要的数据，因为它是需要。由于地图需要更多数据，您可以进一步调用后端。
If possible, it would be best to collect this data on the backend, probably put it onto disk, and then provide only the necessary data to the frontend as it's needed. As the map needs more data, you would make further calls to the backend.
如果无法做到这一点，您可以使用客户端捆绑加载此数据。如果数据不经常更新，您甚至可以将其缓存在前端。这至少可以防止用户需要重复获取它。
If this isn't possible, you could load this data with the client-side bundle. If the data doesn't update too frequently, you can even cache it on the frontend. This would at least prevent the user from needing to fetch it repeatedly.
或者，您可以通过服务器上的流读取JSON并将数据流传输到客户端，使用 JSONStream 之类的内容来解析客户端上的数据。
Alternatively, you can read the JSON via a stream on the server and stream the data to the client and use something like JSONStream to parse the data on the client.
以下是如何通过套接字从服务器流式传输JSON的示例：如何通过套接字从服务器流式传输JSON
Here's an example of how to stream JSON from your server via sockets: how to stream JSON from your server via sockets
这篇关于如何发送和接收大型JSON数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
