---
title: "Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code"
description: "Genie Code’s traditional machine learning workflows are difficult to evaluate because notebook quality depends on code, best practices, and adaptation to data and task. The team built an evaluation pipeline with nine LLM judges, each scoring applicable dimensions—including data exploration, imputation, training, MLflow logging, and cell organization—from 1 to 3, using test cases across classification, regression, and forecasting. On 50 human-graded notebooks, initial judge alignment was weak in several areas, including model training (MAE 0.680) and model use (0.562), with positivity bias contributing to missed technical flaws. Applying MLflow’s MemAlign, which combines generalized semantic guidelines with retrieved episodic examples, reduced judge error by 74–89% on the most important dimensions; removing episodic memory weakened most gains, while data exploration showed a nonsignificant regression."
---

# Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code

[Databricks](https://yomu.fyi/company/databricks) · Stepan Nosov, Pavle Martinović, Tejas Sundaresan, Alkis Polyzotis, Nemanja Petrovic · May 8, 2026

**Type:** Benchmark

## Summary

Genie Code’s traditional machine learning workflows are difficult to evaluate because notebook quality depends on code, best practices, and adaptation to data and task. The team built an evaluation pipeline with nine LLM judges, each scoring applicable dimensions—including data exploration, imputation, training, MLflow logging, and cell organization—from 1 to 3, using test cases across classification, regression, and forecasting. On 50 human-graded notebooks, initial judge alignment was weak in several areas, including model training (MAE 0.680) and model use (0.562), with positivity bias contributing to missed technical flaws. Applying MLflow’s MemAlign, which combines generalized semantic guidelines with retrieved episodic examples, reduced judge error by 74–89% on the most important dimensions; removing episodic memory weakened most gains, while data exploration showed a nonsignificant regression.

## Context

Genie Code’s generated traditional ML notebooks must be evaluated for code quality, ML best practices, and data-informed tailoring. LLM judges did not consistently interpret evaluation rubrics like human experts, missing technical flaws and showing positivity bias, which made the original evaluation system unreliable for iteration, regression testing, benchmarking, and CI.

## Approach / What changed

The team created nine rubric-based LLM judges covering common ML workflow dimensions and compared their scores with human ratings on 50 Genie Code-generated notebooks. They used MemAlign in MLflow to distill human feedback into semantic guidelines and retrieve relevant episodic examples at scoring time, then tested the contribution of episodic memory separately.

## Takeaways

- The evaluation framework scores nine dimensions, including library installation, data exploration, imputation, feature engineering, model training and use, metrics, MLflow logging, and cell organization, using scores from 1 to 3 plus N/A.
- Before alignment, judge disagreement was highest for model training (MAE 0.680), model use (0.562), data imputation (0.474), and data exploration (0.407).
- Removing episodic memory reduced or eliminated improvements across most dimensions; model training still improved, but data imputation showed no gain and MLflow logging regressed from 0.062 to 0.396 MAE.

**Tags:** [Genie](https://yomu.fyi/topic/genie), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [MLflow](https://yomu.fyi/topic/mlflow)

- Source: [Databricks](https://www.databricks.com/blog/using-memalign-improve-evaluation-traditional-machine-learning-genie-code)
- Source URL: https://www.databricks.com/blog/using-memalign-improve-evaluation-traditional-machine-learning-genie-code
- Ingested by Yomu: 2026-08-31T03:35:14.756Z

[Read original post](https://www.databricks.com/blog/using-memalign-improve-evaluation-traditional-machine-learning-genie-code)