Loading post…
DatabricksFabian Halkivaha, Mukrram Ur Rahman, Maria Vedenina, Timur YüreExplainer
How Zalando built a unified data foundation for AI and analytics on Databricks
Summary
Zalando describes how it is building a unified data foundation for analytics and AI across a microservices-driven ecosystem serving more than 50 million active customers and 7,000 brands and partners. Its architecture combines private domain-owned catalogs for autonomous development with a centrally governed shared catalog that exposes cross-team data through Dynamic Views, enabling identity-based policies, GDPR checks, column classification, and auditability. Databricks Metric Views provide a governed semantic layer where business logic and KPI definitions are centralized and reusable across SQL, dashboards, notebooks, and AI agents. Genie uses those metrics plus enriched metadata to answer natural-language questions more reliably, although poorly curated or unaggregated data can increase costs; Zalando plans broader enterprise adoption through curated spaces and additional orchestration.
Context
Zalando’s microservices architecture streams terabytes of events into a central data lake, but distributed ownership created governance difficulties and divergent metric definitions. The described foundation addresses the separation between transactional and analytical data while supporting secure sharing across hundreds of teams.
Approach / What changed
Zalando uses Unity Catalog with a dual-catalog pattern: private catalogs support domain-level development, while a central shared catalog serves governed cross-team data through Dynamic Views. A GitOps workflow provisions and classifies shared views after pull-request approval. Metric Views centralize business definitions, and Genie uses those governed metrics, enriched metadata, and curated spaces for conversational analytics.
Takeaways
Private catalogs give domain teams autonomy, while the shared catalog restricts organization-wide data exposure to centrally governed Dynamic Views.
Metric Views decouple business-metric definitions from their consumers, allowing the same governed logic to serve SQL, dashboards, notebooks, and AI agents.
Genie’s accuracy improves when it uses governed metrics and enriched metadata, but unaggregated or poorly curated data can make the service expensive.
FAQ
Why does Zalando expose shared data through Dynamic Views instead of direct table pointers?
Dynamic Views let Zalando apply centralized access logic, including GDPR and antitrust checks, while maintaining an audit trail for row- and column-level decisions. Queries for sensitive columns without the required authorization fail with a permission denied error, preventing partial or misleading results.
How do Metric Views improve Genie’s answers?
Genie recognizes governed metrics such as Net Merchandise Value instead of reconstructing their calculations from raw tables. Descriptions, synonyms, and sample queries in Unity Catalog provide additional context for interpreting business terms such as “Cancellations.”
What trade-off does Zalando identify when using Genie?
Genie can become expensive when it is configured over unaggregated tables and views. Zalando therefore emphasizes carefully curating the data and context used by each Genie space.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.