在NodeJS服务器上处理大量数据检索 [英] Handling large amounts of data retrieval on NodeJS server

问题描述

关于如何降低从NodeJS后端服务器上的数据库中获取和处理大量物联网时间序列数据的延迟的一般性问题.有时即使将超时设置为15秒，我也会超时.

A general question on how to lower latency of fetching and processing large amounts of iot time-series data from the database on my NodeJS back-end server. Sometimes I get timed-out even when I've set the time-out to be 15 seconds.

我当前的设置使用Google数据存储作为我的数据库来流传输数据，使用NodeJS后端服务器与数据存储进行交互，然后再将数据传递到前端.我还将后端服务器本地托管的MongoDB用作缓存层.

My current setup uses Google Datastore as my database for streaming data, NodeJS backend server for interacting with Datastore before passing the data to the front-end. I'm also using MongoDB hosted locally on the backend server as a caching layer.

1个请求的数据检索工作流程大致如下:从MySQL db查询设备的MAC地址，并用于查询请求时间范围内的缓存，然后从时间间隔(缓存未涵盖)中检索数据数据存储，然后汇总为请求的时间分辨率，并根据需要对某些类型的数据进行额外的解析.

The data-retrieval workflow for 1 request is roughly as follows: Device mac-addresses are queried from MySQL db and used to query cache for the requested time range, data from time gaps (not covered by cache) are then retrieved from Datastore, and aggregated into the requested time resolution, and additional parsing is done for some types of data if required.

我可以想到的一些提高绩效的策略.欢迎对以下策略提出任何其他建议/意见.

Some strategies for improving performance I can think of include. Any other suggestions/ comments on the below strategies are welcome.

1. 异步-使用异步从缓存以及数据存储中获取数据(已实现)

1. Async - Using async to fetch data from cache as well as Datastore (implemented)

流-使用fs返回流中的数据以提高内存利用率

Streams - Use fs to return data in streams to improve memory utilization

压缩-使用诸如压缩之类的库来减少数据量 发送到前端

Compression - Use libraries like compression to reduce the size of data sent over to front-end

集群-使用群集利用多核处理器(更多用于处理多个请求，而与减少每个请求的等待时间无关)

Clustering - Use cluster to utilize multi-core processors (more for serving multiple requests and not so relevant for reducing latency for each request)

推荐答案

您想集中精力减少延迟，但是从我的角度来看，您使用的系统过于复杂，涉及的技术并未得到有效利用.您说您的工作流程是这样的:

You want to focus on reducing latency, however from my point of view you are using a too complex system involving technologies not efficiently used. You say your workflow is something like this:

MySQL->缓存(使用MongoDB)/数据存储-> NodeJS->前端

MySQL -> Cache(Using MongoDB) / Datastore -> NodeJS -> FrontEnd

首先，请参见您使用的是三种不同的数据库解决方案.我知道它们旨在用于不同的任务，但这似乎并不是最有效的方法.我想说Datastore + MongoDB的组合也不是很自然.为什么不使用以下之一?

First of all, see that you are using three different DB solutions. I know they are intended for different tasks, still it doesn't seem to be the most efficient way to go. I would say that the combination of Datastore+MongoDB doesn’t seem very natural either. Why don’t you use one of the following?:

1. 数据存储+ Redis的组合(用于缓存功能).在这里，您将获得有关如何使用Redis缓存应用程序数据的官方文档.在这里，您可以找到有关如何从Node连接到Redis的更详细的教程. js在Google App Engine Flexible 上完成所有交互，包括部署使用Deploy Launcher在Google Cloud上Redis
2. 直接使用Mongodb.这里有一个有关如何连接的教程在Google App Engine灵活环境中从Node.js迁移到MongoDB .

1. A combination of Datastore + Redis (for caching functionalities). Here you have the official docs on how to Cache Application Data Using Redis. Here you have a more detailed tutorial on how to Connect to Redis from Node.js on Google App Engine Flexible , doing all the interconection including deploying Redis on Google Cloud using Deploy Launcher
2. Use Mongodb directly.Here you have a tutorial on how to Connect to MongoDB from Node.js on Google App Engine Flexible environment.

无论选择第一个还是第二个选项:您都无法使用MySQL删除吗?

And whatever you choose the first or second option: Aren't you able to drop using MySQL?

如果您使用 Google App Engine 中的任何一种解决方案，您将可以更轻松地回答与聚类"相关的问题.增加服务器CPU/RAM.您可以尝试使用 app.yaml 的不同配置，以检查最适合您的内核和RAM的数量.

If you use any of those solutions using Google App Engine, you will have it easier to answer your questions related to Clustering & Increase Server CPU/ RAM. You can try with different configurations of your app.yaml to check the amount of cores and RAM that fits you best.

我没有想到与异步和压缩策略有关的其他任何东西.根据Streams，您可以使用@MichałCzapracki提出的有关使用 scramjet 的建议.

Nothing else comes to my mind related to async and compression strategies. As per Streams, you can use the suggestions by @Michał Czapracki about using scramjet.

这篇关于在NodeJS服务器上处理大量数据检索的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！