You are here: Predictive Analytics > AB Testing > AB Analysis

AB Analysis Tool

This tool compares the percentage change in a performance measure to the same measure either over the same time period one year earlier (controlling for possible seasonal effects), or a user specified time period, for two different groups (a treatment group and control group). The test performed to make the comparison is Welch's two sample t-test*.

Since the measures can involve using over a year's worth of historic data, it is important to have sufficient historic data available that is prior to the start of the test. If this condition is not met, then an error will occur.

This tool uses the R programming language. Go to Options > Download Predictive Tools to install R and the packages used by the R Tool.

Input

Configuration Properties

Input Data

    1. Controls: Use the drop downs to:

      1. Select the field with the Control unit identifier: For best results, this field should be some sort of character based type such as String or V-String.

      2. Select the control unit to treatment unit mapping fielding: This field should contain the identifier of the treatment unit to which each control unit is assigned.

    2. Treatments: Use the drop down to select the field with the Treatments unit identifier. For best results, this field should be some sort of character based type such as String or V-String.

    3. Performance Data: Use the drop downs to specify:

    1. Performance Data ID (String) Field: The field with the test unit identifier. For best results, this field should be some sort of character based type such as String or V-String.

    2. Period (Date) Field: The field with the (Date or DateTime type) reporting period identifier.

    3. Performance Measure Field: The numeric field with the performance (e.g., sales, traffic) measure of interest.

Test Dates

    1. Test Start Date: The date the test started, selected using a calendar interface.

    2. Test End Date: The date the test ended, selected using a calendar interface.

    3. Set the dates of the comparison period: Checking this box allows the user to specify a custom period to use for comparison purposes by selecting the start and end dates of the comparison using a calendar interface. By default, the comparison period is taken to be the same time period one year earlier.

Additional Information

    1. The test name: This information is used to identify the test the analysis pertains to in the report.

    2. Provide any additional information about the test: Any other relevant information that makes sense to include in the report, such as the DMA(s) in which the test was carried out, additional information about characteristics of the test, and so on.

    3. Optional alternative name for the performance measure: This enables the performance measure to be identified in the report in terms that will be easier for readers to understand (e.g., "Customer Traffic" instead of "T_Cust_Day").

    4. The required or target percentage growth level: This number is likely to be based on the incremental increase needed in revenues needed to achieve breakeven or a minimal target needed target to justify the roll out of the treatment program to the relevant population.

Graphics Options

    1. Specify the file location to save the interactive report (I) output.
    2. Specify the Date format to use in the interactive report (I) output.
    3. Graph Resolution: Specify the image resolution. Lower resolution creates a smaller file and is best for viewing on a monitor. Higher resolution creates a larger file with much better print quality.

Outputs

O Output: Includes a textual summary of the test results, basic summary statistics of the test results, a number of visualizations of the test results, and a table of the results of a Welch's two sample (the treatment and control samples) t-test of the difference in the percentage change for the performance measure of interest.

E Output: The Extended output is a data table consisting of the values used to populate the Dot Plot found in the O and I outputs.

G Output: The Grouped Data output is a data table consisting of the values used to populate the Time Comparison Plot found in the O and I outputs.

I Output: The Interactive Dashboard is HTML that displays in your browser. You can interact with the visualizations by clicking on the different graphic elements to reveal more information, values, metrics and analytics.

*en.wikipedia.org/wiki/Welch's_t_test