--- title: "Data Engineering" description: "5 posts about Data Engineering, summarised, each linking to the original." --- # Data Engineering > 5 posts about Data Engineering, summarised, each linking to the original. ## Articles ### [Databricks at SIGMOD 2026](https://yomu.fyi/post/databricks-at-sigmod-2026.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Indrajit Roy - Published: May 29, 2026 Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current\_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution. ### [Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy](https://yomu.fyi/post/scaling-for-mhhs-50x-cost-efficient-margin-data-engineering-at-octopus.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Saad Ali, David Poulet, Daniel Taylor, Ismail Makhlouf - Published: May 23, 2026 Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings. ### [Announcing the Databricks analytics engineer learning pathway](https://yomu.fyi/post/announcing-the-databricks-analytics-engineer-learning-pathway.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Maroua Lazzarou, Pratyarth Rao - Published: May 18, 2026 Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions. ### [Announcing Databricks student fellows](https://yomu.fyi/post/announcing-databricks-student-fellows.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Trang Le, Elise Hollowed, Joe Nash - Published: May 12, 2026 Databricks is launching the Student Fellows Program, an opportunity for university and college students interested in computer science, AI, and data engineering to become leaders in their campus communities. Fellows receive specialized training from Databricks experts, build hands-on skills with the Databricks platform, and organize campus activities such as hackathons, tech talks, and study groups while connecting peers with resources. The initiative also offers opportunities for free or discounted certification exam vouchers and practical experience intended to support applications for internships with Databricks, its customers, or partners. Student Fellows’ primary mission is to foster a community of learners and serve as a bridge between Databricks and their university communities, with the first cohort currently accepting applications. ### [Data Science vs Data Engineering: Choosing Analysis or Infrastructure](https://yomu.fyi/post/data-science-vs-data-engineering-choosing-analysis-or-infrastructure.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Databricks Staff - Published: May 1, 2026 This guide distinguishes data engineering from data science by their roles in the data value chain. Data engineers build and maintain ingestion pipelines, warehouses, lakes, access controls, and production systems, using tools such as SQL, Python, Scala, Java, Apache Spark, Kafka, orchestration platforms, and cloud infrastructure. Data scientists work with clean, accessible data to perform exploratory analysis, engineer features, train and validate statistical or machine-learning models, and communicate findings through visualization and storytelling. The roles collaborate through feedback on data quality, shared documentation, model deployment infrastructure, and versioned schemas, feature pipelines, and experiments. For choosing a path, the guide recommends trying a two-week pipeline project and a two-week ML project; it also outlines portfolio deliverables for each and says both roles require strong analytical and computer-science fundamentals.