一个人应该在小检查中控制`Depth`参数？ [英] How much should one control the `Depth` parameter in smallcheck?

问题描述

我正在做一些关于 smallcheck 的实际工作，我对如何使用 Depth code>参数。在讨论这个问题之前，让我说明我正在使用 smallcheck 。

在工作中，在我们自己的内部数据库之前重新构建一个简单的Web服务。 Web服务执行一些查询，并将查询结果序列化为JSON。我目前的工作是确保：给定一个表示查询结果的对象，该对象产生预期的JSON。例如：

  data action 
 = Action {actionType ::！ActionType 
，actionDescription ::！ Text 
，actionPerformedAt ::！UTCTime 
，actionAgentName ::！Text 
} 
  

必须产生JSON，例如：

  {
type：Booking，
description：无论，
performAt：2012-01-04，
agent：Tom
} 
  

对于 smallcheck 来说，这看起来是一个理想的任务，我将其制定为以下内容：

  testAction :: Tasty.TestTree 
 testAction = Tasty.testGroupAction
 [SmallCheck .testPropertytype$ 
 SmallCheck.over操作$ match $ 
 Aeson.keytype - > Aeson.toJSON。 actionType 
 
，SmallCheck.testPropertydateActioned$ 
 SmallCheck.over操作$ match $ 
 Aeson.keydateActioned - >预期的UTCTimeEncoding。 actionPerformedAt 
 
  - 等等
] 
 
  - （ - >）::等式a =>透镜 - 艾森遍历a  - > （b→a）→> b  - > Bool 
  -  actions :: Monad m => SmallCheck.Series m Action 
  

默认 smallcheck 在 Delicious 框架中的深度是5，这导致了我还没有看到完成的测试运行。 smallcheck 具有 changeDepth 和 changeDepth1 函数，所以我 could 将这些作为 changeDepth（const 3）使用，以确保我的测试总是在合理的时间内运行。但是，通过这样做，我不禁感到我在某个地方错过了这一点。例如，现在不可能通过改变命令行选项来运行测试来运行更长时间的测试，也许在一夜之间。另一方面，如果我使用 changeDepth（ - 2），它仍然感觉好像我在假设测试如何运行！也许最好假定全局测试深度为5分钟，并且每个属性都可以根据需要调整深度。

很想听到 smallcheck 这个更实用的一些反馈。

解决方案<尽管smallcheck（无论如何，对于小型案例）的穷举是一个有趣的属性，但我宁愿在这种情况下提出quickcheck。虽然JSON具有轻量级的结构，但从实际数据的角度来看，它非常重要。

测试时间也非常关键取决于您如何定义大小（深度）为你的类型的Series实例中的smallcheck。如果你的类型有很多分支（许多构造函数），那么测试的数量将会迅速增长。 深度是指数级指数，而指数的基数是指与特定测试用例相关的Series实例中的分支数量。换句话说，如果你平均有2个构造函数，你就会看32个测试用例，但是如果你有20个构造函数，它就更像3200000.

然而，您的覆盖率也会受到影响 - 如果您在测试用例中减少分支（使深度增加得更快），那么给定深度的覆盖范围就会减少。通过快速检查，您可能会损失一些小型测试用例，而不喜欢用小检查来取样一些更大的示例。

I'm doing my first bit of real work with smallcheck, and I'm a little confused about how to use the Depth parameter. Before I go into that, let me state what I'm using smallcheck for.

At work, we're building a simple web service in front of our own in-house database. The web service does some queries, and responds with the query results serialized to JSON. What I'm working on at the moment is the assurance that: given an object that represents the results of a query, this object produces the expected JSON. For example:

data Action
  = Action { actionType :: !ActionType 
           , actionDescription :: !Text
           , actionPerformedAt :: !UTCTime
           , actionAgentName :: !Text
           }

Must produce JSON such as:

{
  "type": "Booking",
  "description": "Whatever",
  "performedAt": "2012-01-04",
  "agent": "Tom"
}

This looked like an ideal task for smallcheck, and I have this formulated as the following:

testAction :: Tasty.TestTree
testAction = Tasty.testGroup "Action"
  [ SmallCheck.testProperty "type" $
      SmallCheck.over actions $ match $
        Aeson.key "type" --> Aeson.toJSON . actionType

  , SmallCheck.testProperty "dateActioned" $
      SmallCheck.over actions $ match $
        Aeson.key "dateActioned" --> expectedUTCTimeEncoding . actionPerformedAt

  -- and so on
  ]

-- (-->) :: Eq a => lens-aeson traversal a -> (b -> a) -> b -> Bool
-- actions :: Monad m => SmallCheck.Series m Action

The default smallcheck depth in the tasty framework is 5, which results in test runs that I've yet to see finish. smallcheck has the changeDepth and changeDepth1 functions, so I could use these as changeDepth (const 3) to ensure that my tests always run in a sensible amount of time. However, by doing this I can't help but feel I'm missing the point somewhere? For example, it is now not possible to run a longer test, perhaps overnight, by just changing the command line options to run the test. On the otherhand, if I used changeDepth (- 2), it still feels as though I am making an assumption of how the tests are ran! Perhaps it's best to assume that a global test depth of 5 runs in n seconds, and it's up to each property to adjust the depth as it sees fit?

Would love to hear some feedback on this more practical side of smallcheck.

解决方案

While the "exhaustiveness" of smallcheck (for the small cases anyway) is an intriguing property, I would rather suggest quickcheck in this context. While JSON has a lightweight structure, from the point of view of actual bits of data, it is pretty heavy.

The testing time also quite crucially depend on how you define "size" (depth) for smallcheck in the Series instances for your types. If your types have a lot of branching (many constructors), the number of tests will grow quickly. It is exponential in "depth", while the base of the exponential is the amount of branching in your Series instances relevant to a particular testcase.

In other words, if you have 2 constructors on average, you are looking at something like 32 testcases to run, but if you have 20, it's more like 3200000.

However, your coverage is affected as well -- if you reduce the branching (making depth increase faster) in your testcases, you will get less coverage with given depth. With quickcheck, you trade losing some of the "small" testcases in favour of sampling a number of bigger examples which you can't reach with smallcheck.

这篇关于一个人应该在小检查中控制`Depth`参数？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！