从巨大的（> 200MB）阵列文件加载随机对象而不加载整个阵列 [英] Loading random objects from a huge (>200MB) array file without loading the entire array

问题描述

我正在构建一个静态Angular网站，其数据基于来自包含超过200,000个JSON对象的大型数组中的1000个随机对象。我在网上某处以文本形式存储大量JSON对象（如Github或Dropbox），文本文件大小超过200MB。由于性能问题，显然不是加载网站中所有数据的选项，但我也不想使用数据库，因为我正在构建一个静态网站。如何在不加载整个数组的情况下从200,000个对象中随机加载1000个对象？

I am building a static Angular website whose data is based on 1000 random objects from a huge array that contains more than 200,000 JSON objects. I am storing the huge array of JSON objects in text form somewhere online (like Github or Dropbox), and the text file size is larger than 200MB. It's obviously not an option to load all the data in the website due to performance concerns, but I also do not want to use a database as I am building a static website. How do I only load randomly 1000 objects out of the 200,000 objects without loading the entire array?

推荐答案

您可以尝试仅下载该文件的第一个部分块并尝试使用它。

You could attempt to download only the first partial chunk of the file and try to use that.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

您将使用Angular的HTTP服务来生成 GET 请求，但会添加范围：bytes = 0-1023 到标题。其中 0-1023 定义了你想要获取的字节数（使用开始和结束范围）。

You would use the Angular's HTTP service to make a GET request, but would add Range: bytes=0-1023 to the headers. Where 0-1023 defines the number of bytes you would like to fetch (using a start and end range).

如果服务器支持此功能，则必须清除从服务器获取的字符串JSON。由于JSON在完成之前将被截止。

If the server supports this feature, then you'll have to clean up the string JSON that is fetched from the server. Since the JSON will be cut-off before it's complete.

我不知道JSON的结构是什么。很难说这是否是一个实用的解决方案，但如果它是一个JSON对象数组。您可以尝试查找以数组分隔符，字符结尾的字节范围，然后只需将] 附加到结尾字符串。

Without me knowing what the structure of the JSON is. It's difficult to say if this is a practical solution, but if it's an array of JSON objects. You could try to find a byte range that ends on an array separator , character and just append ] to the end of the string.

如果你需要文件的随机部分。您将对不同的部分块进行多个 GET 请求，然后尝试清理JSON以使其有效。

If you need random parts of the file. You would make multiple GET requests for different partial chunks, and then try to clean the JSON so that it's valid.

这篇关于从巨大的（> 200MB）阵列文件加载随机对象而不加载整个阵列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！