Azure的Redis的缓存 - 对GET调用超时 [英] Azure Redis cache - timeouts on GET calls

问题描述

我们已经有了在Azure的多个Web和工人角色，通过StackExchange.Redis库连接到我们的Azure Redis的缓存，我们正在接收正在我们的终端到终端的解决方案，慢慢停下来经常超时。其中之一的例子是如下：

We've got several web and worker roles in Azure connecting to our Azure Redis cache via the StackExchange.Redis library, and we're receiving regular timeouts that are making our end-to-end solution grind to a halt. An example of one of them is below:

System.TimeoutException：超时执行GET流：459，研究所：4，
  经理：无效，队列：12，曲= 0，QS = 12，QC = 0，WR = 0/0，在= 65536/0时
  StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl [T]（消息
  消息，ResultProcessor ​​ 1个处理器，ServerEndPoint服务器）的
  c:\\TeamCity\\buildAgent\\work\\58bc9a6df18a3782\\StackExchange.Redis\\StackExchange\\Redis\\ConnectionMultiplexer.cs:line
  1785在StackExchange.Redis.RedisBase.ExecuteSync [T]（消息
  消息，ResultProcessor ​​ 1个处理器，ServerEndPoint服务器）的
  c:\\TeamCity\\buildAgent\\work\\58bc9a6df18a3782\\StackExchange.Redis\\StackExchange\\Redis\\RedisBase.cs:line
  79在StackExchange.Redis.RedisDatabase.StringGet（RedisKey键，
  的commandflags标志）的
  c:\\TeamCity\\buildAgent\\work\\58bc9a6df18a3782\\StackExchange.Redis\\StackExchange\\Redis\\RedisDatabase.cs:line
  在1346
  。OptiRTC.Cache.RedisCacheActions＆LT;> c__DisplayClass4 1＆LT;获取与GT; b__3（）的
  C：\\ dev的\\ OptiRTCAzure \\ OptiRTC.Cache \\ RedisCacheActions.cs：行104
  Polly.Retry.RetryPolicy.Implementation（动作动作，IEnumerable的 1
  shouldRetry predicates，Func`1 policyStateFactory）在
  OptiRTC.Cache.RedisCacheActions.Get [T]（字符串键，布尔
  allowDirtyRead）在
  C：\\ dev的\\ OptiRTCAzure \\ OptiRTC.Cache \\ RedisCacheActions.cs：行107
  OptiRTC.Cache.RedisCacheAccess.d__e4.MoveNext（）
  在C：\\ dev的\\ OptiRTCAzure \\ OptiRTC.Cache \\ RedisCacheAccess.cs：行1196;
  TraceSourceWaWorkerHost.exe事件

System.TimeoutException: Timeout performing GET stream:459, inst: 4, mgr: Inactive, queue: 12, qu=0, qs=12, qc=0, wr=0/0, in=65536/0 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 1785 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 79 at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in c:\TeamCity\buildAgent\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 1346 at OptiRTC.Cache.RedisCacheActions.<>c__DisplayClass41.<Get>b__3() in c:\dev\OptiRTCAzure\OptiRTC.Cache\RedisCacheActions.cs:line 104 at Polly.Retry.RetryPolicy.Implementation(Action action, IEnumerable1 shouldRetryPredicates, Func`1 policyStateFactory) at OptiRTC.Cache.RedisCacheActions.Get[T](String key, Boolean allowDirtyRead) in c:\dev\OptiRTCAzure\OptiRTC.Cache\RedisCacheActions.cs:line 107 at OptiRTC.Cache.RedisCacheAccess.d__e4.MoveNext() in c:\dev\OptiRTCAzure\OptiRTC.Cache\RedisCacheAccess.cs:line 1196; TraceSource 'WaWorkerHost.exe' event

所有超时有不同的队列和QS号，但邮件的其余部分是一致的。这些StringGet调用是横跨在高速缓存不同的密钥。在每一个我们的服务，我们使用一个单独的缓存访问类的是与在Web或辅助角色的启动我们的Io​​C容器注册一个ConnectionMultiplexer：

All the timeouts have different queue and qs numbers, but the rest of the messages are consistent. These StringGet calls are across different keys in the cache. In each of our services, we use a singleton cache access class with a single ConnectionMultiplexer that is registered with our IoC container in the web or worker role startup:

        container.RegisterInstance<ICacheAccess>(cacheAccess);

在我们的实现ICacheAccess，我们正在创建的复用器如下：

In our implementation of ICacheAccess, we're creating the multiplexer as follows:

            ConfigurationOptions options = new ConfigurationOptions();
            options.EndPoints.Add(serverAddress);
            options.Ssl = true;
            options.Password = accessKey;                    
            options.ConnectTimeout = 1000;
            options.SyncTimeout = 2500;

            redis = ConnectionMultiplexer.Connect(options);

在这里Redis的对象在整个实例使用。我们有这个通过实施ICacheAccess连接到高速缓存中约20网络和工人的角色实例，但管理控制台显示平均200个并发连接到缓存中。

where the redis object is used throughout the instance. We've got about 20 web and worker role instances connecting to the cache via this ICacheAccess implementation, but the management console shows an average of 200 concurrent connections to the cache.

我见过使用StackExchange.Redis，我们通过这样的NuGet其中1.0.333版本引用其他职位，但是当我看到添加的StackExchange.Redis.dll参考的实际版本，它表明1.0 .316.0。我们已经尝试添加和删除的NuGet引用，以及将它添加到一个新的项目，我们始终获得版本差异。

I've seen other posts that reference using version 1.0.333 of StackExchange.Redis, which we're doing via NuGet, but when I look at the actual version of the StackExchange.Redis.dll reference added, it shows 1.0.316.0. We've tried adding and removing the NuGet reference as well as adding it to a new project, and we always get the version discrepancy.

任何有识之士将AP preciated。谢谢你。

Any insight would be appreciated. Thanks.

其他信息：

我们已升级到1.0.371。我们有各自不同的时间间隔访问相同的缓存对象两个服务，一个用来编辑和偶尔读取和一个读取该对象数次。两种服务都部署了相同的缓存code和StackExchange.Redis库版本。我几乎从来没有看到超时在编辑对象的服务，但我的时间的50和75％取得超时上读取它的服务。超时具有相同的格式对视了一眼上面指出，他们继续在波利重试块同时处理RedisException和System.TimeoutException和500毫秒后重试一次包裹db.StringGet调用后发生。

We've upgraded to 1.0.371. We have two services that each access the same cache object at different intervals, one to edit and occasionally read and one that reads this object several times a second. Both services are deployed with the same caching code and StackExchange.Redis library version. I almost never see time outs in the service that edits the object but I get timeouts between 50 and 75% of the time on the services that reads it. The timeouts have the same format as the one indicated above, and they continue to occur after wrapping the db.StringGet call in a Polly retry block that handles both RedisException and System.TimeoutException and retries once after 500ms.

我们联系了微软对这个问题，他们确认他们什么也看不见了Redis的日志，表明对Redis的服务端的问题。我们的高速缓存未命中％是Redis的服务器上非常低，但我们将继续得到这些超时，这大大阻碍了我们的应用程序的功能。

We contacted Microsoft about this issue, and they confirm that they see nothing in the Redis logs that indicate an issue on the Redis service side. Our cache miss % is extremely low on the Redis server, but we continue to get these timeouts, which substantially hinder our application's functionality.

在回应的评论，是的，我们总是有一个QS号码，从未在质量控制。我们总是在中的第一部分和从未在第二若干

In response to the comments, yes, we always have a number in qs and never in qc. We always have a number in the first part of the in and never in the second.

更附加信息：

当我运行在更高的CPU用更少的情况下一个服务，我得到更多的显著这些超时错误的情况时，在较低的CPU上运行比。更具体地说，我从我们的服务今天早上拉一些数字。当他们在大约30％的CPU运行时，我看到很少的超时问题 - 只有42多30分钟。当我删除了一半的情况下，他们开始在大约60-65％的CPU运行速度提高10倍，达到536超过30分钟。

When I run a service with fewer instances at a higher CPU, I get significantly more of these timeout errors than when instances are running at lower CPUs. More specifically, I pulled some numbers from our services this morning. When they were running at around 30% CPU, I saw very few timeout issues - just 42 over 30 minutes. When I removed half the instances and they started to run at around 60-65% CPU, the rate increased 10-fold to 536 over 30 minutes.

推荐答案

我知道这个线程个月大，但我觉得我自己的经验可以在这里添加一些价值。我曾与天青Redis的缓存同样的问题（上获取超时），但意识到，这是几乎完全发生在获取其中字符串值都比较大（> 250K长）。我实现的gzip上都获取和设置（当字符串值大），现在我几乎从来没有得到一个超时。

I know this thread is months old but I think my own experiences can add some value here. I had the same problem with Azure Redis Cache (timeouts on Gets) but realized that it was almost exclusively happening on Gets where the string value was relatively large (> 250K in length). I implemented gzip on both Gets and Sets (when the string value is large) and now I almost never get a timeout.

即使这没有解决您的特定问题，它可能是很好的做法，COM preSS一般值，以降低成本，提高性能。

Even if this doesn't solve your particular problem, it's probably good practice to compress the values in general to reduce costs and improve performance.

这篇关于Azure的Redis的缓存 - 对GET调用超时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！