50GB HttpRuntime.Cache持久性可能吗？ [英] 50GB HttpRuntime.Cache Persistence Possible?

问题描述

我们有一个ASP.NET 4.0应用程序，该应用程序从数据库中提取了一个复杂的数据结构，该结构需要花费12个小时以上的时间才能推入内存中的数据结构（以后存储在HttpRuntime.Cache中）。数据结构的大小正在迅速增加，如果应用程序重新启动，我们将无法再等待12个多小时才能将其存入内存。如果您要更改web.config或Web应用程序中导致重启的任何代码，则这是一个主要问题-这意味着需要等待很长时间才能使用该应用程序，并且阻碍了开发或更新部署。

We have an ASP.NET 4.0 application that draws from a database a complex data structure that takes over 12 hours to push into an in memory data structure (that is later stored in HttpRuntime.Cache). The size of the data structure is quickly increasing and we can't continue waiting 12+ hours to get it into memory if the application restarts. This is a major issue if you want to change the web.config or any code in the web application that causes a restart - it means a long wait before the application can be used, and hinders development or updating the deployment.

数据结构必须在内存中，以使网站可用的速度工作。与HttpRuntime.Cache相比，内存数据库（例如memcache或Redis）运行缓慢，并且在我们的情况下不起作用（在内存数据库中，数据库必须序列化put / get，而且它们无法互相引用，它们使用作为查找键的键-降低性能，再加上大量按键，性能会迅速下降）。性能在这里是必须的。

The data structure MUST be in memory to work at a speed that makes the website usable. In memory databases such as memcache or Redis are slow in comparison to HttpRuntime.Cache, and would not work in our situation (in memory db's have to serialize put/get, plus they can't reference each other, they use keys which are lookups - degrading performance, plus with a large amount of keys the performance goes down quickly). Performance is a must here.

我们要做的是在应用程序结束之前（重新启动时）将HttpRuntime.Cache快速转储到磁盘上，并能够在应用程序结束时立即将其重新加载再次启动应用程序（希望在几分钟之内，而不是12个小时或几天以上）。

What we would like to do is quickly dump the HttpRuntime.Cache to disk before the application ends (on a restart), and be able to load it back immediately when the application starts again (hopefully within minutes instead of 12+ hours or days).

内存结构约为50GB。

The in-memory structure is around 50GB.

有解决方案吗？

推荐答案

在内存数据库中，例如memcache或Redis与HttpRuntime.Cache相比，速度较慢。

In memory databases such as memcache or Redis are slow in comparison to HttpRuntime.Cache

是的，但是它们非常快相比起12小时以上的加速就我个人而言，我认为您在强迫50 GB结构的负载时采用了错误的方法。只是一个建议，但是我们将HttpRuntime.Cache用作多层缓存策略的一部分：

Yes, but they are very fast compared to a 12+ hour spin-up. Personally, I think you're taking the wrong approach here in forcing load of a 50 GB structure. Just a suggestion, but we use HttpRuntime.Cache as part of a multi-tier caching strategy:

• 首先检查本地缓存，等等


• 否则，redis用作下一层缓存（比基础数据快，持久且支持许多应用程序服务器）（然后更新了本地缓存）


• 否则，将命中基础数据库（然后更新redis和本地缓存）

，在加载时，我们不会需要内存中的任何内容-它会按需填充，并且此后速度很快。我们还使用pub / sub（同样由redis提供）确保高速缓存失效及时。最终结果是：寒冷时足够快，温暖时非常快。

The point being, at load we don't require anything in memory - it is filled as it is needed, and from then on it is fast. We also use pub/sub (again courtesy of redis) to ensure cache invalidation is prompt. The net result: it is fast enough when cold, and very fast when warm.

基本上，我会观察避免需要50GB的数据，然后再执行任何操作。

Basically, I would look at anything that avoids needing the 50GB data before you can do anything.

如果数据不是真正的 cache ，而是您的数据，我将在适当的对象模型上查看序列化。我建议protobuf-net（我偏向作者）在这里是个不错的候选人-速度非常快而且输出量很小。

If this data isn't really cache, but is your data, I would look at serialization on a proper object model. I would suggest protobuf-net (I'm biased as the author) as a strong candidate here - very fast and very small output.

这篇关于50GB HttpRuntime.Cache持久性可能吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！