---
title: "Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy"
description: "Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings."
---

# Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy

[Databricks](https://yomu.fyi/company/databricks) · Saad Ali, David Poulet, Daniel Taylor, Ismail Makhlouf · May 23, 2026

**Type:** Problem & solution

## Summary

Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.

## Context

Market-wide Half-Hourly Settlement changes industry data from monthly readings to 48 data points per meter per day. For a supplier with over 8 million customers, the resulting 48x data increase made the legacy monthly, single-grain margin architecture expensive and unable to align monthly profiled revenue with half-hourly actual costs.

## Approach / What changed

The team rebuilt the pipeline as three independently tunable streams for settlement cost, half-hourly smart-tariff revenue, and daily standard-tariff revenue. A unified multi-grain consumption layer provides the source of truth, while a Databricks “Job of Jobs” workflow manages dependencies. Incremental processing uses Change Data Feed where available and microbatch dbt otherwise, alongside measured Spark and Delta optimisations.

## Takeaways

- Margin processing uses three grains: half-hourly settlement cost, half-hourly revenue for EV, heat pump, and time-of-use customers, and daily revenue for standard-tariff customers.
- Change Data Feed and microbatch dbt reduced rows processed per run from 25 billion to 300 million, a 98.8% reduction; freshness improved from weekly to daily.
- Projected cost per settlement date fell from $23.63 to $0.48, approximately a 50x reduction, with about $1 million in annualised cost avoidance excluding upstream savings.

**Tags:** [Databricks](https://yomu.fyi/topic/databricks), [Data Engineering](https://yomu.fyi/topic/data-engineering), [dbt](https://yomu.fyi/topic/dbt), [Spark](https://yomu.fyi/topic/spark)

- Source: [Databricks](https://www.databricks.com/blog/scaling-mhhs-how-octopus-energy-achieved-50x-cost-reduction-margin-data-engineering)
- Source URL: https://www.databricks.com/blog/scaling-mhhs-how-octopus-energy-achieved-50x-cost-reduction-margin-data-engineering
- Ingested by Yomu: 2026-08-31T03:33:12.274Z

[Read original post](https://www.databricks.com/blog/scaling-mhhs-how-octopus-energy-achieved-50x-cost-reduction-margin-data-engineering)