Loading…
From test bench to lakehouse: how AVL modernizes measurement data analytics with Impulse
Dr. Thomas Bonfert, Jonathan Bräuer, Fabian Ade, Maxim Hammer, Florian Gorzitzke, David Crescence, Christa Simon, Jörg Zimmermann, Hannes Schneider
- Source
- Databricks
- Published
- Added to Yomu
Summary
AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.
Context
Automotive test campaigns generate hundreds of thousands of measurement recordings and hundreds of terabytes of time-series sensor data. Traditional desktop tools and isolated scripts do not scale easily, are difficult to reproduce, and leave measurement data outside the governance used elsewhere in the enterprise.
Approach / What changed
AVL uses a Databricks Medallion Architecture with Unity Catalog and Databricks Workflows. Raw files and metadata move through Bronze and a quality-assured hierarchical Silver model before Impulse compiles declarative Python TSAL expressions into distributed Spark execution, producing Gold-layer reporting tables, ad-hoc DataFrames, or ML feature matrices.
Takeaways
- Impulse’s TSAL lets engineers define signal arithmetic, event conditions, and aggregations without writing Spark transformations, joins, window functions, or user-defined functions.
- Virtual-channel computation includes channel alias resolution, unit conversion, common time-axis alignment, and interpolation before arithmetic is performed.
- AVL reports reducing analysis time from days to minutes, processing many recordings in one run, saving infrastructure costs versus on-premises solutions, and improving reproducibility and cross-team standardization.