如何调试挂起的动作 [英] How to debug a hanging action
问题描述
我们有一个MVC应用，每个人都知道，然后(从每天几次到每2-3天一次)遭受整个站点的挂起(它不响应任何请求并且不会在其上重新启动)拥有).我们设法将可疑对象的数量减少到一个页面(只要站点被挂起，该页面就会显示在挂起请求的顶部).但是，知道是哪页是罪魁祸首并没有帮助，因为我们无法在开发机器上重现该问题.实际上，我们甚至无法在生产机器上重现问题(在大多数情况下，仅打开有故障的页面不会破坏任何内容).
We have an MVC app that every know and then (anything from few times a day to once per 2-3 days) suffers a a site-wide hang (it doesn't respond to any requests and doesn't restart on its own either). We managed to reduce the number of suspects to a single page (whenever the site would suffer a hang this page would show up at the top of hanging requests). Knowing which page is the culprit doesn't help though, because we're unable to recreate the problem on development machine. In fact we can't even recreate the problem on the production machine (most of the time, simply opening the faulty page doesn't break anything).
我们所知道的是，有时当用户转到此页面(控制器操作)时，网站将挂起，并且我们知道发出请求的方式并没有太多可能导致挂起(来自IIS)日志中，我们知道用户是如何进入错误页面的，并且我们正在处理一个简单的GET，没有查询字符串参数.
What we know is that, sometimes when a user goes to this page (controller action) the site will hang and we know there isn't much in the way the request is being made that could contribute the the hang (from IIS logs we know how user got to the faulty page and that we're dealing with a simple GET with no query string params).
我们想知道挂起在代码中的确切位置，但是我们不知道如何从应用程序/工作线程/IIS/Windows Server中获取此类信息.没有异常会保存到Windows日志中，我们的内置记录器也不会选择任何异常(可能与挂起有关).
We would like to know where exactly in the code the hang happens, but we don't know how to get such information out of the app/worker thread/IIS/Windows Server. No exception is saved to Windows log, nor is our built into the app logger picking any exceptions (probably has something to do with the hang).
有什么方法可以告诉IIS工作者线程在给定时间到底在做什么(例如获取相应的文件和代码行)?
What are the ways to tell what exactly is an IIS worker thread doing at a given time (like get the corresponding file and line of code)?
P.S.我在另一个问题中描述了站点范围挂起的确切症状，但与该问题无关.加上问题最终变得太广泛，只能带来一般性的答案. P.S.2 .我们通过查看IIS中的Worker Processes \ Requests找到了有问题的页面.
P.S. I described the exact symptoms of the site-wide hang in another question, but those are irrelevant to this question. Plus the question ended up being too broad which only brought generic answers. P.S.2 We found the offending page by looking into Worker Processes\Requests in IIS.
推荐答案
在过去，我遇到了同样的问题，而真正迅速帮助我的是使用以下命令将跟踪代码行插入DLL的能力: Mono.Cecil.下面的代码将跟踪注入到DLL的无符号版本中，并输出一个新的DLL，该代码可通过在开始处插入日志记录行来创建缩进的跟踪"(如果需要，则在末尾插入日志行...未在下面显示)每个方法的时间，以便您可以计时每个调用的存在时间和存在时间.有很多第三方工具，但是这很容易(最多需要一天的工作)，可以完全控制开发人员，并且免费提供.
In the past I've had the same issue and what really quickly helped me was the ability to inject a line of tracing code int the DLL(s) using Mono.Cecil. The code below injects tracing into the unsigned version of the DLL's and outputs a new DLL that can be used to create an "indented trace" into the code by injecting logging lines in the start (and end if needed...not shown below) of each method so that you can time the entry and exist of each call. There are a lot of third party tools but this is easy (a days work at best), gives full control to the developer, and as a bonus, is free.
您还需要创建一个具有跟踪类和可用于记录数据的静态调用"TraceStep"的DLL(下面的IISStackTraceProvider).要跨工作进程和线程重建数据，请使用"HttpContext.Items属性"将GUID和秒表与每个BeginRequest/EndRequest关联.由于您的电话挂起...您想跟踪开始"但绝不结束"或因超时而结束的所有电话，并将其余电话扔掉以保持通话速度.
You also need to create a DLL (IISStackTraceProvider below) that has the trace class and the static call "TraceStep" which can be used to log the data. To reconstruct that data across worker processes and threads, you associate a GUID and stopwatch with each BeginRequest/EndRequest using the "HttpContext.Items Property". Since your call hangs...you want to trace any call that "Begin" but never "End" or Ends with a time out and toss the rest away to keep things fast.
我已经在生产环境中的Web场中针对每小时服务器每服务器一百万个呼叫测试了这一点，同时又不影响性能，但是请记住要记录的请求和丢弃的请求.另外，我使用Redis来捕获日志，因为写入时间非常快而且还免费，然后一旦发现问题就可以读取Redis数据.
I've tested this against ~one million calls an hour/per server in a web farm in our production environment without affecting performance but be mindful of what requests you want to log and which requests get thrown away. Also, I used Redis to capture the logs since write times are insanely fast and also it's free, then just read the Redis data once I trap the issue.
class TraceInjection
{
private ELogLevel logLevel;
public enum ELogLevel
{
eLow,
eMid,
eHigh
}
public TraceInjection(ELogLevel logLevel)
{
this.logLevel = logLevel;
}
public bool InjectTracingLine(string assemblyPath, string outputDirectory)
{
CustomAttribute customAttr;
AssemblyDefinition asmDef;
// New assembly path
string fileName = Path.GetFileName(assemblyPath);
string newPath = outputDirectory + "\\" + fileName;
// Check if Output directory already exists, if not, create one
if (!Directory.Exists(outputDirectory))
{
Directory.CreateDirectory(outputDirectory);
}
ModuleDefinition modDefCopy = null;
TypeDefinition typDefCopy = null;
try
{
var resolver = new DefaultAssemblyResolver();
resolver.AddSearchDirectory(System.IO.Path.GetDirectoryName(assemblyPath));
var parameters = new ReaderParameters
{
AssemblyResolver = resolver,
};
// Load assembly
asmDef = AssemblyDefinition.ReadAssembly(assemblyPath, parameters);
String functionsFound = "";
foreach (var modDef in asmDef.Modules)
{
modDefCopy = modDef;
foreach (var typDef in modDef.Types)
{
typDefCopy = typDef;
foreach (MethodDefinition metDef in typDef.Methods)
{
try
{
// Skipping things I personally don't want traced...
if (metDef.IsConstructor ||
metDef.IsAbstract ||
metDef.IsCompilerControlled ||
metDef.IsGetter ||
metDef.IsSetter
) continue;
functionsFound += String.Format("{0}\r\n", metDef.Name.Trim());
// Get ILProcessor
ILProcessor ilProcessor = metDef.Body.GetILProcessor();
/*** Begin Method ******/
// Load fully qualified method name as string
Instruction i1 = ilProcessor.Create(
OpCodes.Ldstr,
String.Format(">,{0},{1}", metDef.Name.Replace(",", ""), asmDef.Name.Name)
);
ilProcessor.InsertBefore(metDef.Body.Instructions[0], i1);
// Call the method which would write tracing info
Instruction i2 = ilProcessor.Create(
OpCodes.Call,
metDef.Module.Import(
typeof(IISStackTraceProvider).GetMethod("TraceStep", new[] { typeof(string) })
)
);
ilProcessor.InsertAfter(i1, i2);
}catch(Exception ex)
{
// ...
}
}
}
}
Console.Write(functionsFound);
Console.ReadKey();
// Save modified assembly
asmDef.Write(newPath, new WriterParameters() { WriteSymbols = true });
}
catch (Exception ex)
{
modDefCopy = null;
typDefCopy = null;
// Nothing to be done, just let the caller handle exception
// or do logging and so on
throw;
}
return true;
}
public bool TryGetCustomAttribute(MethodDefinition type, string attributeType, out CustomAttribute result)
{
result = null;
if (!type.HasCustomAttributes)
return false;
foreach (CustomAttribute attribute in type.CustomAttributes)
{
if (attribute.Constructor.DeclaringType.FullName != attributeType)
continue;
result = attribute;
return true;
}
return false;
}
}
这篇关于如何调试挂起的动作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
