通过http将大量数据从Angular发送到Express的最佳方法是什么? [英] What is the best approach to send huge chunks of data from Angular to Express via http?
问题描述
我有一个庞大的数据集，例如txt格式的 200MB .我能够通过http调用将数据发送到Angular .
I have a huge dataset, something like 200MB in txt format. I was able to send the data to Angular, via http call.
现在，我需要在本地进行少量编辑后将数据发送回，这可以在每条行"中完成(请参见
Now I need to send the data back, after small editing done locally, which can be done in every single "line" (see Reading an FASTA file, this is the file format).
令我惊讶的是，我无法寄回，它太大了；我要保存到MongoDB.我注意到，如果我发送1％的通知，它就可以了.因此，我现在的解决方案是以较小的数据块发送数据.尽管如此，我在这里尝试了一些建议的方法，但是什么也没有.
For my surprise, I cannot send it back, it is too big; I am saving to MongoDB. I have noticed that if I send say 1%, it goes. So, the solution I have now is sending the data in smaller chunks of data. Nonetheless, I have tried some suggested approaches here, but nothing.
我在这里找到了类似的问题: Angular 7处理大型帖子http请求
I have found a similar question here: Angular 7 Handling large post http request
我的问题是:为了接收数据，我们可以使用观察者；因此，以小卡盘形式接收数据.是否有一种清晰的方法(例如，一种方法)或可以输入整个数据集的方法来完成其余的工作?
My question is: for receiving the data we can use observer; thus, the data is received in small chucks. Is there a clear way to make it, say, a method, or something that I can input the whole dataset and it does the rest?
可能的提示:如果我可以发送1％，则可以将其分成1％的块，然后依次发送；但我需要确保Angular在上一个操作完成之前不会发送HTTP调用，否则，由于JavaScript内存问题，Express应用程序将崩溃.
Possible hint: if I can send 1%, I could divide it in 1% chunks, and send then in sequence; but I need to make sure that Angular will not send the HTTP call until the previous one is finished, otherwise, the express app crashes due to JavaScript memory issues.
See here the first version of this question, where everything started: How can I hold on http calls for a while?
我不确定这是否是一个未解决的问题，因为我没有提供代码.我仍在学习使用该平台.有时我们从鸡身上拿了太多的金蛋！提前致谢！:)
I am not sure if this is an opened question, since I am not providing codes. I am still learning to use this plataform; sometimes we take too many golden eggs from the chicken! thanks in advance! :)
讨论
@zzfs明智地给了我一些见解，我会考虑的；尽管如此，我仍然愿意以代码为例进行示例；至少对我来说，从代码中学习更容易.
@zzfs wisely gave me some insights, I will consider them; nonetheless, I am opened to code as example, samples; it is easier to learn from code, at least for me.
@zzfs提到要考虑差异"，只保存更改，这是我当前的方法.我坚持认为，我已经打开了这个赏金的原因是，如果可能的话，我想拥有一个强大的系统，这意味着如果一个人添加过多的评论，差异将很小，并且系统可能崩溃；我认为发生这种情况的可能性很小，因为有些人不太可能会一次添加1.000-10.000条评论.
@zzfs mention to consider the "difference", and save just the changes, that is my current approach. The reason I am insisting, that I have opened this bounty is that I want to have, if possible, a robust system, which means if a person adds too much comments, the difference will be small, and the system can crash; the probability of that happening, I believe, is small since it is most unlikely that some will add say 1.000-10.000 comments in a single time.
另一个问题是，我不知道我当前的上传数据集的方式是否最好，我是从本地保存的Mongo和Express应用程序上传的.
The other problem is that do not know if my current way of uploading the dataset is the best, I am uploading from an app in Mongo and Express, locally saved.
推荐答案
将如此大的文件发送到&仅当例如下载zip之类的Blob或将Blob上传到服务器时(例如youtube或Dropbox上传)，才可以从客户端进行访问.所有其他用例通常都无需进行如此大的转移就可以处理...
Sending such large files to & from the client only makes sense when for example downloading blobs like zips or, say, uploading blobs to the server (think youtube or dropbox uploads). All other use cases are usually handled without such large transfers...
您提到您只会执行小修改".这意味着您应该只发送差异，而不是整件事.您的问题太笼统了，所以我不能给您一个更清晰的答案，但是当您的大文件是JSON或文本文件并且用户更改了其中的内容时，您可以轻松地标记出更改的位置并发送更改后的实际有效负载.然后，您的服务器(节点正在运行express?)应该能够进行差异检查并将更改应用于200MB文件.
You mentioned you'd only perform "small editing". This means you should only send the difference, not the whole thing. Your question is too broad so I cannot give you a clearer answer but when your large file is, say, a JSON or a text file and the user changed something in it, you can quite easily mark the position of that change and send it along with the actual, changed payload. Your server (node running express?) should then be able to diff-check and apply the change to the 200MB file.
当然支持对多个HTTP请求进行分块处理&经常发生.当您确实要上传200MB文件时，可以将其拆分并使用 依次上传的switchMap 或
Chunking multiple HTTP requests is of course supported & happens frequently. When you really want to upload your 200MB file, you can split it and use either switchMap for sequential uploading or forkJoin for parallelized requests.
但是随后您仍然会在服务器上进行某种 后处理(在最琐碎的情况下合并块)，因此您应该将注意力集中在实现diff-checker方法上如上所述.
But then you'd still do some kind of post-processing on the server (combining the chunks in the most trivial case) so your attention should rather be directed at implementing the diff-checker method described above.
这篇关于通过http将大量数据从Angular发送到Express的最佳方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
