---
title: "Building an A/B testing analysis framework for mobile gaming on Databricks"
description: "HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount."
---

# Building an A/B testing analysis framework for mobile gaming on Databricks

[Databricks](https://yomu.fyi/company/databricks) · Sanjay Ashok, Jack Holdsworth, Tingting Wan, Joel Dias, Richard Carr, Monika Kolodziejczyk · Sep 18, 2026

**Type:** Problem & solution

## Summary

HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.

## Context

HARDlight found that manual analysis, inconsistent statistical methods, and dashboards that could not serve both casual and expert audiences were slowing experimentation and reducing confidence in results. The studio needed standardised inference, accessible insights at different levels of detail, and durable records of completed experiments.

## Approach / What changed

The framework ingests standardised experiment definitions, player telemetry, and outcome metrics into governed Databricks tables. Notebooks compute statistical outputs upstream, including effect estimates, uncertainties, and segment impacts, while Databricks AI/BI presents them through daily-refresh dashboards with progressive disclosure. Databricks Workflows standardise experiment runs, and completed experiments are preserved as frozen dashboard snapshots containing outcomes, decisions, and recommended actions.

## Takeaways

- Upstream notebooks materialise statistical outputs into a unified experiment analytics model, avoiding duplicated logic and re-running analysis inside dashboards.
- Progressive dashboard layers serve different audiences, from LLM-generated status summaries to KPIs, diagnostics, revenue, segment, monetisation, gameplay, and statistical detail.
- Frozen dashboards preserve final outcomes, decisions, and recommended actions, creating a durable reference for completed experiments and reducing repeated historical analysis.

## FAQ

### How does the dashboard support both quick status checks and expert validation?

It starts with an LLM-generated summary for a high-level view, then progressively reveals KPIs, diagnostics, segment analysis, and deeper statistical detail. Users can stop at the summary or continue into the underlying analytical layers.

### Why are experiment dashboards frozen after a run ends?

Freezing preserves the final snapshot, decisions, and recommended actions rather than allowing later refreshes to replace the result. This keeps completed experiments available with their original context and reduces ambiguity when teams revisit them.

### What operational gains did the framework produce?

It reduced manual effort by more than eight hours per week and saved roughly one day per experiment through standardised runs with Databricks Workflows. The changes enabled a targeted two-times increase in monthly A/B testing capacity without increasing headcount.

**Tags:** [AI](https://yomu.fyi/topic/ai), [Databricks](https://yomu.fyi/topic/databricks), [Experimentation](https://yomu.fyi/topic/experimentation), [MLflow](https://yomu.fyi/topic/mlflow), [Unity Catalog](https://yomu.fyi/topic/unity-catalog)

- Source: [Databricks](https://www.databricks.com/blog/building-ab-testing-analysis-framework-mobile-gaming-databricks)
- Source URL: https://www.databricks.com/blog/building-ab-testing-analysis-framework-mobile-gaming-databricks
- Ingested by Yomu: 2026-09-17T14:12:57.022Z

[Read original post](https://www.databricks.com/blog/building-ab-testing-analysis-framework-mobile-gaming-databricks)