对基于Django的项目中的A / B测试有何想法？ [英] Any thoughts on A/B testing in Django based project?

问题描述

我们现在才开始对基于Django的项目进行A / B测试。我能否获得有关此A / B测试的最佳做法的一些信息或有用的见解。

We just now started doing the A/B testing for our Django based project. Can I get some information on best practices or useful insights about this A/B testing.

理想情况下，每个新的测试页都将使用单个参数进行区分（就像Gmail）。 mysite.com/?ui=2应该提供不同的页面。因此，对于每个视图，我都需要编写一个装饰器来基于ui参数值加载不同的模板。而且我不想在装饰器中对任何模板名称进行硬编码。那么urls.py url模式将如何？

Ideally each new testing page will be differentiated with a single parameter(just like Gmail). mysite.com/?ui=2 should give a different page. So for every view I need to write a decorator to load different templates based on the 'ui' parameter value. And I dont want to hard code any template names in decorators. So how would urls.py url pattern will be?

推荐答案

退后一步并抽象一下A / B测试很有用在深入研究代码之前正在尝试做的事情。我们到底需要进行什么测试？

It's useful to take a step back and abstract what A/B testing is trying to do before diving into the code. What exactly will we need to conduct a test?

• 有条件的目标


• 在至少有两个符合目标条件的路径


• 一个用于将观看者吸引到其中一个路径的系统


• 一个用于记录测试结果的系统

• A Goal that has a Condition
• At least two distinct Paths to meet the Goal's Condition
• A system for sending viewers down one of the Paths
• A system for recording the Results of the test

考虑到这一点，让我们考虑一下实现。

With this in mind let's think about implementation.

目标

当我们想到网络上的目标时，通常是指用户到达某个页面或他们完成了特定操作，例如成功注册为用户或进入结帐页面。

When we think about a Goal on the web usually we mean that a user reaches a certain page or that they complete a specific action, for example successfully registering as a user or getting to the checkout page.

在Django中，我们可以通过两种方式对此进行建模-也许是天真地在视图内部，只要有已达到目标：

In Django we could model that in a couple of ways - perhaps naively inside a view, calling a function whenever a Goal has been reached:

    def checkout(request):
        a_b_goal_complete(request)
        ...

但这没有帮助，因为我们必须添加到处都需要该代码-而且，如果我们使用任何可插拔的应用程序，我们不希望不编辑其代码以添加A / B测试。

But that doesn't help because we'll have to add that code everywhere we need it - plus if we're using any pluggable apps we'd prefer not to edit their code to add our A/B test.

如何我们在不直接编辑视图代码的情况下引入了A / B目标？中间件呢？

How can we introduce A/B Goals without directly editing view code? What about a Middleware?

    class ABMiddleware:
      def process_request(self, request):
          if a_b_goal_conditions_met(request):
            a_b_goal_complete(request)

这将允许我们跟踪A / B网站上任何地方都有目标。

That would allow us to track A/B Goals anywhere on the site.

我们如何知道目标条件已得到满足？为了便于实施，我建议我们知道一个目标，当用户到达特定的URL路径时，它已经满足条件。作为奖励，我们可以在不弄脏视图的情况下进行测量。回到我们注册用户的示例，我们可以说当用户到达URL路径时已经达到了此目标：

How do we know that a Goal's conditions has been met? For ease of implementation I'll suggest that we know a Goal has had it's conditions met when a user reaches a specific URL path. As a bonus we can measure this without getting our hands dirty inside a view. To go back to our example of registering a user we could say that this goal has been met when the user reaches the URL path:

/ registration / complete

因此我们定义 a_b_goal_conditions_met ：

     a_b_goal_conditions_met(request):
       return request.path == "/registration/complete":

路径

在Django中考虑路径时，很自然地跳到使用不同路径的想法模板。是否还有其他方法有待探索。在A / B测试中，您可以使两页之间的差异很小，并测量结果。因此，最好的做法是定义一个基本路径模板，所有目标路径应从该模板扩展。

When thinking about Paths in Django it's natural to jump to the idea of using different templates. Whether there is another way remains to be explored. In A/B testing you make small differences between two pages and measure the results. Therefore it should be a best practice to define a single base Path template from which all Paths to the Goal should extend.

应如何渲染这些模板？装饰器可能是一个不错的开始-在Django中，最好的做法是在您的视图中包含参数 template_name ，装饰器可以在运行时更改此参数。

How should render these templates? A decorator is probably a good start - it's a best practice in Django to include a parameter template_name to your views a decorator could alter this parameter at runtime.

    @a_b
    def registration(request, extra_context=None, template_name="reg/reg.html"):
       ...

您可以看到此装饰器内省包装的函数并修改 template_name 参数或从某个地方（例如模型）查找正确的模板。如果我们不想在每个函数中添加装饰器，则可以将其作为ABMiddleware的一部分实现：

You could see this decorator either introspecting the wrapped function and modifying the template_name argument or looking up the correct templates from somewhere (like a Model). If we didn't want to add the decorator to every function we could implement this as part of our ABMiddleware:

    class ABMiddleware:
       ...
       def process_view(self, request, view_func, view_args, view_kwargs):
         if should_do_a_b_test(...) and "template_name" in view_kwargs:
           # Modify the template name to one of our Path templates
           view_kwargs["template_name"] = get_a_b_path_for_view(view_func)
           response = view_func(view_args, view_kwargs)
           return response

我们还需要添加一些方法来跟踪哪些视图正在运行A / B测试等。

We'd need also need to add some way to keep track of which views have A/B tests running etc.

用于将观众引导到某个路径的系统

从理论上讲，这很简单，但有很多不同之处实施方案，因此尚不清楚哪种方案最好。我们知道一个好的系统应该沿着路径平均划分用户-必须使用某种哈希方法-也许您可以使用内存缓存计数器的模数除以路径数-也许有更好的方法。

In theory this is easy but there are lot of different implementations so it's not clear which one is best. We know a good system should divide users evenly down the path - Some hash method must be used - Maybe you could use the modulus of memcache counter divided by the number of Paths - maybe there is a better way.

记录测试结果的系统

我们需要记录有多少用户沿什么路径前进-当用户达到目标时，我们还需要访问此信息（我们需要能够说出他们达到目标条件所遵循的路径）-我们将使用某种模型来记录数据以及Django会话或Cookies来保留Path信息，直到用户满足目标条件为止。

We need to record how many users went down what Path - we'll also need access to this information when the user reaches the goal (we need to be able to say what Path they came down to met the Condition of the Goal) - we'll use some kind of Model(s) to record the data and either Django Sessions or Cookies to persist the Path information until the user meets the Goal condition.

结束思想

我已经给出了很多在Django中实现A / B测试的伪代码-上面的方法绝不是一个完整的解决方案，而是一个为A /创建可重用框架的良好起点在Django中进行B测试。

I've given a lot of pseudo code for implementing A/B testing in Django - the above is by no means a complete solution but a good start towards creating a reusable framework for A/B testing in Django.

作为参考，您可以想在GitHub上查看Paul Mar的7分钟A / B-这是上述内容的ROR版本！
http://github.com/paulmars/seven_minute_abs/tree/master

For reference you may want to look at Paul Mar's Seven Minute A/Bs on GitHub - it's the ROR version of the above! http://github.com/paulmars/seven_minute_abs/tree/master

更新

打开对Google网站优化工具的进一步反思和研究，很明显，上述逻辑中存在漏洞。通过使用不同的模板表示路径，您可以中断视图上的所有缓存（或者如果视图被缓存，它将始终使用相同的路径！）。取而代之的是，我不使用Paths，而是窃取GWO术语，并使用 Combinations 的想法-这是模板更改的一个特定部分-例如，更改<$ c站点的$ c>< h1> 标签。

On further reflection and investigation of Google Website Optimizer it's apparent that there are gaping holes in the above logic. By using different templates to represent Paths you break all caching on the view (or if the view is cached it will always serve the same path!). Instead, of using Paths, I would instead steal GWO terminology and use the idea of Combinations - that is one specific part of a template changing - for instance, changing the <h1> tag of a site.

该解决方案将包含可转换为JavaScript的模板标签。在浏览器中加载页面后，JavaScript会向您的服务器发出请求，以获取可能的组合之一。

The solution would involve template tags which would render down to JavaScript. When the page is loaded in the browser the JavaScript makes a request to your server which fetches one of the possible Combinations.

这样，您可以在保存每个页面的同时测试多个组合

This way you can test multiple combinations per page while preserving caching!

更新

仍有进行模板切换的空间-例如，您引入了一个全新的主页，并希望对照旧主页测试其性能-您仍想使用模板切换技术。要记住的事情是，您将必须找出某种方法来在X个页面的缓存版本之间进行切换。为此，您需要覆盖标准的缓存中间件，以查看它们是否是在请求的URL上运行的A / B测试。然后它可以选择要显示的正确的缓存版本！！！

There still is room for template switching - say for example you introduce an entirely new homepage and want to test it's performance against the old homepage - you'd still want to use the template switching technique. The thing to keep in mind is your going to have to figure out some way to switch between X number of cached versions of the page. To do this you'd need to override the standard cached middleware to see if their is a A/B test running on the requested URL. Then it could choose the correct cached version to show!!!

更新

使用上述想法，我实现了可插拔的应用程序，用于Django的基本A / B测试。您可以关闭Github：

Using the ideas described above I've implemented a pluggable app for basic A/B testing Django. You can get it off Github:

http://github.com/johnboxall/django-ab/tree/master

这篇关于对基于Django的项目中的A / B测试有何想法？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！