速度快，内存对+ 5M记录表查找范围 [英] Fast, in-memory range lookup against +5M record table

问题描述

我有到位+ 5M静态记录的数据库表。结构简单：（INT开始，结束INT，结果INT）。所以我有一个一定的INT，我需要找到它对应的结果（INT）。目前，查表是在DB中，但它需要驻留在内存中，最有可能在环境中没有数据库的访问。

I have a database table with +5M static records in place. Simple structure: (starting int, ending int, result int). So i have an certain INT and i need to find it's corresponding result(int). Currently, the look-up table is in DB, but it needs to reside in-memory, most likely in the environment without a database access.

我的解决方案需要在没有数据库访问执行此逻辑，存储器和超级快，我需要处理1000每秒的交易。该集的大小超过50MB少，所以我可以扔了整个事情到针对它的内存和运行范围看起坐，每次这个帖子：<一href=\"http://stackoverflow.com/questions/8948205/doing-a-range-lookup-in-c-sharp-how-to-implement/8949322#8949322\">Doing在C＃中查找范围 - 如何实现。但我不知道怎么会这样规模的演出。

My solution needs to perform this logic without a database access, in memory and super fast as I need to process 1000s of transactions per second. The size of the set is little over 50MB so I could throw the whole thing into the memory and run range look-ups against it, per this post: Doing a range lookup in C# - how to implement. But I don't know how it will perform on such scale.

• 请我pre加载启动时该表？这可能需要一段时间。


• 任何方式对表加载到一些.dat文件，并有超高效的查询在运行时？

顺便说一句，我在Azure上，如果不能确定使用存储表上查找帮助...

BTW, I am on Azure, not sure if using Storage Tables helps on lookups...

推荐答案

二进制搜索是超级快。在50M记录二进制搜索只需要27比较，以找到答案。只要将其加载到内存中，并使用你的链接查找范围

binary searches are super fast. A binary search on 50M records only takes 27 comparisons to find the answer. Just load it into memory and use the Range lookup you linked.

如果你发现它是缓慢的，开始优化：

If you find it is slow, start optimizing:

• 更改范围对象，而不是在struct类


• 手 - code你自己的二进制搜索算法（一）实现了相等比较，而不是直接呼叫到的的IEqualityComparer 和（b）使用指针和其他不安全的技巧，以禁用，而做搜索数组边界检查。

• Change the Range object to struct instead of class
• Hand-code your own binary search algorithm that (a) implements the equality comparer directly instead of calling out to an IEqualityComparer and (b) uses pointers and other unsafe tricks to disable array bounds checking while doing the search.

这篇关于速度快，内存对+ 5M记录表查找范围的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！