多线程软件的测试方法 [英] Testing approach for multi-threaded software

问题描述

我有一个成熟的地理空间软件，最近已经改写了一些领域，以更好地利用现代PC中的多处理器。具体来说，显示，GUI，空间搜索和主要处理都已经分离到单独的线程。该软件有一个相当大的GUI自动化套件的功能回归，另一个较小的一个用于性能回归。虽然所有自动化测试都通过了，但我不相信它们提供了几乎足够的覆盖范围，找到与种族条件，死锁和其他与多线程相关的缺陷相关的错误。你会使用什么技术来查看这样的错误是否存在？你会倡导什么技术，把他们出来，假设有一些在那里根除？

到目前为止，我所做的是在运行在调试器下的应用程序上运行GUI功能自动化，这样我可以突破死锁并捕获崩溃，并计划做一个边界检查器构建和重复针对该版本的测试。我还通过PC-Lint对源进行了静态分析，希望找到潜在的死锁，但没有任何有价值的结果。

应用程序是C ++，MFC，多文档/视图，每个文档有多个线程。我使用的锁定机制是基于一个对象，其中包含一个指向CMutex的指针，它被锁定在ctor中并在dtor中释放。我使用这个对象的局部变量根据需要锁定各种位的代码，我的互斥体有一个超时，如果超时，触发我的警告。我尽可能避免锁定，尽可能使用资源副本。

您还要进行什么其他测试？

编辑：我已经在一些不同的测试和编程论坛上发布了这个问题，因为我渴望看到不同的思维方式和思想方法将会解决这个问题。非常抱歉，如果你看到它在其他地方交叉。

解决方案

首先，非常感谢您的回应。对于通过不同的演示发布的回复，请参阅：

http://www.sqaforums.com/showflat.php?Cat=0&Number=617621&an=0&page=0#Post617621 a>

http：/ /stackoverflow.com/questions/2437763/testing-approach-for-multi-threaded-software

http://www.softwaretestingclub.com/forum/topics/testing-approach-for?xg_source=activity

和以下邮寄名单; [e-mail]

测试花费的时间比预期的长，因此这个迟到的答复，导致我的结论是，添加多线程到现有的应用程序是负责任的在测试方面非常昂贵，即使编码是相当直接的。这对SQA社区可能是有意义的，因为越来越多的多线程开发正在进行。

根据Joe Strazzere的建议，我发现打击错误的最有效方法是通过具有不同输入的自动化。我最终在三台电脑上这样做了，它们在大约六个星期内反复进行了一系列测试，输入不同。最初，我每天看到每个PC崩溃一到两次。当我跟踪这些，它结束了三个PC之间每周一个或两个，我们没有任何进一步的问题，在过去的两个星期。在过去两周内，我们还有一个版本的用户进行了测试，并且正在内部使用该软件。

除了在自动化下改变输入外，

• 添加允许从配置文件读取互斥量超时的测试选项




• 扩展互斥体超时超出执行一段线程代码的典型时间，并触发调试异常超时。




• 与调试器（VS2008）一起运行自动化，使得当发生问题时，有更好的机会跟踪




• 在没有调试器的情况下运行，以确保调试器不会隐藏其他与时间相关的错误。 li>

  针对正常发布，调试和完全优化的构建运行自动化。 FWIW，优化的构建引起的错误在其他构建中不可重现。

未覆盖的错误类型往往是严重的性质，例如解除引用无效指针，甚至在调试器下进行了相当多的跟踪。正如在其他地方已经讨论过的，SuspendThread和ResumeThread函数最终成为主要的罪魁祸首，并且这些函数的所有使用被互斥体替代。类似地，所有关键部分由于缺乏超时而被移除。关闭文档和退出程序也是一个错误源，在一个实例中，一个文档被销毁，工作线程仍然活动。为了克服这个问题，每个线程添加了一个单独的互斥量来控制线程的生命周期，并由文档析构函数获得，以确保线程按预期终止。

，非常感谢所有的详细和多样的回应。下一次我参加这种类型的活动，我会更好地准备。

I have a piece of mature geospatial software that has recently had areas rewritten to take better advantage of the multiple processors available in modern PCs. Specifically, display, GUI, spatial searching, and main processing have all been hived off to seperate threads. The software has a pretty sizeable GUI automation suite for functional regression, and another smaller one for performance regression. While all automated tests are passing, I'm not convinced that they provide nearly enough coverage in terms of finding bugs relating race conditions, deadlocks, and other nasties associated with multi-threading. What techniques would you use to see if such bugs exist? What techniques would you advocate for rooting them out, assuming there are some in there to root out?

What I'm doing so far is running the GUI functional automation on the app running under a debugger, such that I can break out of deadlocks and catch crashes, and plan to make a bounds checker build and repeat the tests against that version. I've also carried out a static analysis of the source via PC-Lint with the hope of locating potential dead locks, but not had any worthwhile results.

The application is C++, MFC, mulitple document/view, with a number of threads per doc. The locking mechanism I'm using is based on an object that includes a pointer to a CMutex, which is locked in the ctor and freed in the dtor. I use local variables of this object to lock various bits of code as required, and my mutex has a time out that fires my a warning if the timeout is reached. I avoid locking where possible, using resource copies where possible instead.

What other tests would you carry out?

Edit: I have cross posted this question on a number of different testing and programming forums, as I'm keen to see how the different mind-sets and schools of thought would approach this issue. So apologies if you see it cross-posted elsewhere. I'll provide a summary links to responses after a week or so

解决方案

Firstly, many thanks for the responses. For the responses posted across different forumes see;

http://www.sqaforums.com/showflat.php?Cat=0&Number=617621&an=0&page=0#Post617621

http://stackoverflow.com/questions/2437763/testing-approach-for-multi-threaded-software

http://www.softwaretestingclub.com/forum/topics/testing-approach-for?xg_source=activity

and the following mailing list; [e-mail]

The testing took significantly longer than expected, hence this late reply, leading me to the conclusion that adding multi-threading to existing apps is liable to be very expensive in terms of testing, even if the coding is quite straightforward. This could prove interesting for the SQA community, as there is increasingly more multi-threaded development going on out there.

As per Joe Strazzere's advice, I found the most effective way of hitting bugs was via automation with varied input. I ended up doing this on three PCs, which have ran a bank of tests repeatedly with varied input over about six weeks. Initially, I was seeing crashes one or two times per PC per day. As I tracked these down, it ended up with one or two per week between the three PCs, and we haven't had any further problems for the last two weeks. For the last two weeks we have also had a version with users beta testing, and are using the software in-house.

In addition to varying the input under automation, I also got good results from the following;

• Adding a test option that allowed mutex time-outs to be read from a configuration file, which in turn could be controlled by my automation.

• Extending mutex time-outs beyond the typical time expected to execute a section of thread code, and firing a debug exception on time-out.

• Running the automation in conjunction with a debugger (VS2008) such that when a problem occurred there was a better chance of tracking it down.

• Running without a debugger to ensure that the debugger was not hiding other timing related bugs.

• Running the automation against normal release, debug, and fully optimised build. FWIW, the optimised build threw up errors not reproducible in the other builds.

The type of bugs uncovered tended to be serious in nature, e.g. dereferencing invalid pointers, and even under the debugger took quite a bit of tracking down. As has been discussed elsewhere, the SuspendThread and ResumeThread functions ended up being major culprits, and all use of these functions were replaced by mutexes. Similarly all critical sections were removed due to lack of time-outs. Closing documents and exiting the program were also a bug source, where in one instance a document was destroyed with a worker thread still active. To overcome this a single mutex was added per thread to control the life of the thread, and aquired by the document destructor to ensure the thread had terminated as expected.

Once again, many thanks for the all the detailed and varied responses. Next time I take on this type of activity, I'll be better prepared.

这篇关于多线程软件的测试方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！