You are here:

AB Analysis Tool

This tool compares the percentage change in a performance measure to the same measure either over the same time period one year earlier (controlling for possible seasonal effects), or a user specified time period, for two different groups (a treatment group and control group). The test performed to make the comparison is Welch's two sample t-test*.

Since the measures can involve using over a year's worth of historic data, it is important to have sufficient historic data available that is prior to the start of the test. If this condition is not met, then an error will occur.

This tool uses the R programming language. Go to Options > Download Predictive Tools to install R and the packages used by the R Tool.

Input

C anchor: A Controls data stream that contains a field with a unique identifier or each control unit that allows the control units to be identified in the performance data (described below) and a field that gives the identifier of the treatment unit to which each control unit is assigned.
T anchor: A Treatments data stream that contains a field with a unique identifier for each treatment unit that allows the treatment units to be identified in the performance data (described below).
P anchor: A Performance data stream that contains a field with a performance measure (e.g., sales, visits) of interest to be compared between the treatment and control groups, a date field identifying the period each value of the performance measure to which the value pertains (this allows for the creation of the correct comparison time periods), and a unit identifier field that allows the treatment and control units to be properly identified.

Configure the tool

Input Data

Controls: Use the drop downs to:

Select the field with the Control unit identifier: For best results, this field should be some sort of character based type such as String or V-String.
Select the control unit to treatment unit mapping fielding: This field should contain the identifier of the treatment unit to which each control unit is assigned.

Treatments: Use the drop down to Select the field with the Treatments unit identifier. For best results, this field should be some sort of character based type such as String or V-String.

Performance Data: Use the drop downs to specify:

Select the field with the performance data unit identifier: The field with the test unit identifier. For best results, this field should be some sort of character based type such as String or V-String.
Select the field with the reporting period data information: The field with the (Date or DateTime type) reporting period identifier.
Select the field with the performance measure: The numeric field with the performance (e.g., sales, traffic) measure of interest.

Test Dates

Test Start Date: The date the test started, selected using a calendar interface.

Test End Date: The date the test ended, selected using a calendar interface.

Set the dates of the comparison period: Checking this box allows the user to specify a custom period to use for comparison purposes by selecting the start and end dates of the comparison using a calendar interface. By default, the comparison period is taken to be the same time period one year earlier.

Additional Information

The test name: This information is used to identify the test the analysis pertains to in the report.

Provide any additional information about the test: Any other relevant information that makes sense to include in the report, such as the DMA(s) in which the test was carried out, additional information about characteristics of the test, and so on.

Optional alternative name for the performance measure: This enables the performance measure to be identified in the report in terms that will be easier for readers to understand (e.g., "Customer Traffic" instead of "T_Cust_Day").

The required or target percentage growth level (a value of 0.0 implies not target): This number is likely to be based on the incremental increase needed in revenues needed to achieve breakeven or a minimal target needed target to justify the roll out of the treatment program to the relevant population.

Graphics Options

Save Dashboard: Specify the file location to save the interactive report (I) output.

The format used to display dates in the line plot comparison: Specify the Date format to use in the interactive report (I) output.

Graph Resolution: Specify the image resolution. Lower resolution creates a smaller file and is best for viewing on a monitor. Higher resolution creates a larger file with much better print quality.

Outputs

O anchor: Includes a textual summary of the test results, basic summary statistics of the test results, a number of visualizations of the test results, and a table of the results of a Welch's two sample (the treatment and control samples) t-test of the difference in the percentage change for the performance measure of interest.

E anchor: The Extended output is a data table consisting of the values used to populate the Dot Plot found in the O and I outputs.

In a Dot Plot, each column of dots gives the percentage change in Traffic from the same period as the test period, but one year earlier, and the test period for a treatment unit and the control units assigned to that treatment unit. An examination of a dot plot chart allows for a rapid determination of whether (and which of) the treatment units outperformed the control units with respect to Traffic.

G anchor: The Grouped Data output is a data table consisting of the values used to populate the Time Comparison Plot found in the O and I outputs.

The Time Comparison Plot provides a visualization of the average effect of the test treatment across the Treatment units for the performance measure of interest. The performance measure is normalized to be the percentage difference between the value of that measure in each reported time unit and the average of that measure for the entire comparison period, and then averaged across stores of each unit type (Treatment or Control) in each reported time unit.

I anchor: The Interactive Dashboard is HTML that displays in your browser. You can interact with the visualizations by clicking on the different graphic elements to reveal more information, values, metrics and analytics.

*en.wikipedia.org/wiki/Welch's_t_test