This document provides an explanation of the Crossing Minds report, providing an example of what one may look like following a Proof of Concept (PoC) test. Please note that every report can be tailored based on the Key Performance Indicators (KPIs) and other metrics relevant to your organization.

Where to Find those Reports

You can find your analytics reports (monthly or weekly) under the section "Results and KPIs > Reports"

Structure of the Report

The report is structured into several sections, each offering valuable insights into the performance of the A/B test conducted.

Overview

The report commences with an overview that outlines the test duration, the number of unique users in both the Beam and Custom groups, and the calculated return on investment (ROI) when using Beam. This section provides an initial snapshot of the test's scale and potential financial implications.

Results

The report then delves into a detailed examination of the results. This is segmented into results for all users and a subset deemed as 'qualified users'. Metrics included are:

• Recommendations Click-Through Rate (CTR): This measures the percentage of users who clicked on a recommended item. It helps assess the effectiveness of the recommendation system.
• Product Views: An indicator of user engagement, showing the number of times products were viewed.
• Average Order Value (AOV): This signifies the average monetary value of each order placed. It offers insights into consumer purchasing behavior.
• Conversion Rate: This ratio of users who made a purchase to the total number of users provides a direct measure of the test's effectiveness.
• Revenue: Actual revenue earned during the test period is shown for both the Beam and Custom groups.
• Projected Revenue per Month: Based on the test results, this projects potential future earnings.

Interpreting the Results

The report comes with an instruction manual on how to comprehend these results. For example, every new user visiting the site has a 50-50 chance of being assigned to the Beam Group (A Group) or the Custom Group (B Group). The report data includes all users who performed any interaction on the site, with the exception of those who exited without interaction.

A 'qualified user' is defined as one who clicked on a recommendation. Not all users will see recommendations due to factors such as carousel position or a strong intention to view a specific product. By concentrating on activities performed by qualified users, we ensure the results primarily represent the impact of the recommendation system.

Statistical Significance

The report also highlights the statistical significance of the results, explaining the concept of 'lift' and its associated p-value. When a lift value is colored and accompanied by a checkmark, it signifies the direction (+/-) is very unlikely to change with additional data. This understanding aids in determining the reliability of the results and future predictions.

Graphs and Insights

A synopsis of the transactions is offered, including the total number and revenue separated into distinct categories, such as price ranges, number of items purchased, and item popularity. This comprehensive breakdown provides deeper insights into customer purchasing patterns and product performance.

Non-AB Analysis

Additionally, the report explores interaction insights, including the count of total interactions per day and a further breakdown by interaction types. It also provides user-side and item-side insights, which provide information about the total users/items, one-time users/items, moderate users/items, and frequent users/items, along with their distributions. This section offers an understanding of user behavior and item interaction trends beyond the A/B testing scope.

Bear in mind that this is a standard report structure, and individual reports may vary depending on the specific KPIs and business requirements. For further clarity or to discuss your unique A/B test reports, please feel free to reach out to the Beam team.