socket.io性能每数据库行发出一次 [英] socket.io performance one emit per database row

问题描述

我试图了解什么是读取并将大量数据库行(50K-100K)发送给客户端的最佳方法.

I am trying to understand what is the best way to read and send a huge amount of database rows (50K-100K) to the client.

1. 我是否应该简单地一次从后端的数据库中读取所有行，然后以json格式发送所有行?由于用户只是等待了很长时间，所以响应速度不是那么快，但是对于较小的用户来说，响应速度更快.行.

1. Should I simply read all the rows at once from the database at the backend and then send all the rows in a json format? This isn't that much responsive as user is just waiting for a long time, but this is faster for small no. of rows.

我应该从数据库中流传输行，并且在每次从数据库中读取行时，都调用socket.emit()吗?这会导致套接字发出过多的消息，但响应速度更快，但速度很慢...

Should I stream the rows from the database and upon each reading of the row from the database, I call a socket.emit()? This causes too many socket emits, but is more responsive, but slow...

我正在使用node.js，socket.io

I am using node.js, socket.io

推荐答案

重新思考界面

首先，在客户端上显示50-100k行的用户界面设计可能并不是最佳的用户界面.在某些移动设备中，不仅要发送大量数据到客户端并由客户端进行管理，而且可能不切实际，而且显然，与任何单个用户相比，在与之进行的任何给定交互中，实际读取的行数都将超过任何单个用户.这一页.因此，首要任务可能是重新考虑用户界面设计并创建某种更多由需求驱动的界面(分页，虚拟滚动，按字母键入等等).不同(希望更好)的用户界面设计有很多不同的可能性，它们可以减少数据传输量.哪种设计最好，完全取决于用户的数据和可能的使用模型.

First off, a user interface design that shows 50-100k rows on a client is probably not the best user interface in the first place. Not only is that a large amount of data to send down to the client and for the client to manage and is perhaps impractical in some mobile devices, but it's obviously way more rows than any single user is going to actually read in any given interaction with the page. So, the first order might be to rethink the user interface design and create some sort of more demand-driven interface (paged, virtual scroll, keyed by letter, etc...). There are lots of different possibilities for a different (and hopefully better) user interface design that lessens the data transfer amount. Which design would be best depends entirely upon the data and the likely usage models by the user.

批量发送数据

也就是说，如果您要将这么多的数据传输到客户端，则可能要分块(一次一组行)发送数据.分块的想法是，您在一个分块中发送了可消耗的数据量，以便客户端可以对其进行解析，处理，显示结果，然后为下一个分块做好准备.客户端可以一直保持活动状态，因为它在块之间有可用的周期来处理其他用户事件.但是，分块发送可以减少为每一行发送单独消息的开销.如果您的服务器正在使用压缩，则块也将为压缩效率提供更大的机会.一块应该有多大(例如，应包含多少行数据)取决于许多因素，并且可能是通过对可能的客户端或功耗最低的客户端进行试验来最好地确定的.例如，您可能希望每条消息发送100行.

That said, if you were going to transfer that much data to the client, then you're probably going to want to send it in chunks (groups of rows at a time). The idea with chunks is that you send a consumable amount of data in one chunk such that the client can parse it, process it, show the results and then be ready for the next chunk. The client can stay active the whole time since it has cycles available between chunks to process other user events. But, sending it in chunks reduces the overhead of sending a separate message for each single row. If your server is using compression, then chunks gives a greater chance for compression efficiency too. How big a chunk should be (e.g. how many rows of data is should contain) depends upon a bunch of factors and is likely best determined through experimentation with likely clients or the lowest power expected client. For example, you might want to send 100 rows per message.

为数据使用有效的传输格式

并且，如果您正在使用socket.io传输大量数据，则可能需要重新使用JSON格式.例如，发送100,000个都重复完全相同的属性名称的对象不是很有效.您通常可以发明自己的优化方法，以避免重复每个对象中完全相同的属性名称.例如，而不是发送其中的100,000:

And, if you're using socket.io to transfer large amounts of data, you may want to revisit how you use the JSON format. For example, sending 100,000 objects that all repeat exactly the same property names is not very efficient. You can often invent your own optimizations that avoid repeating property names that are exactly the same in every object. For example, rather than sending 100,000 of these:

 {"firstname": "John", "lastname": "Bundy", "state": "Az", "country": "US"}

如果每个对象都具有完全相同的属性，则可以将属性名称编码为自己的代码，也可以发送一次属性名称，然后仅将逗号分隔的值列表发送到接收代码可以放入的数组中放入具有适当属性名称的对象:

if every single object has the exact same properties, then you can either code the property names into your own code or send the property names once and then just send a comma separated list of values in an array that the receiving code can put into an object with the appropriate property names:

 ["John", "Bundy", "Az", "US"]

有时，只需删除冗余信息，数据大小就可以减少2-3倍.

Data size can sometimes be reduced by 2-3x by simply removing redundant information.

这篇关于socket.io性能每数据库行发出一次的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！