Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 541 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
What Is Serverless PostgreSQL?
Serverless PostgreSQL is presented as a fully managed cloud database model that decouples compute and storage, allowing each to scale independently with demand. Traditional deployments require teams to size infrastructure, manually manage scaling, and absorb costs from idle capacity. In serverless systems, the provider provisions compute on demand, can suspend it when idle, and bills according to active usage; scale-to-zero may introduce cold-start latency. The architecture can also support database branching through copy-on-write, creating isolated environments without duplicating data. The article distinguishes this model from lakebase architecture, which combines transactional and analytical workloads on a shared foundation using decoupled compute, durable object storage, log-based storage systems, and orchestration.
The Rise of Sports Intelligence: How the Lakehouse Turns Tracking Data into Competitive Advantage
Professional basketball’s Hawk-Eye SkeleTRACK feed produces roughly 22,620 positional updates per second—about 65 million records per 48-minute game—yet teams often cannot turn that volume into timely, trusted decisions. The integration gap comes from separate vendors for tracking, wearables, video, scouting, and medical data, alongside calibration differences, weak provenance, and compute limits. The Databricks Data + AI Platform is presented as a governed lakehouse that ingests feeds with Lakeflow, refines them through medallion layers, and uses Unity Catalog for lineage, access control, and auditing. Models for shot probability, injury risk, and fatigue can run alongside serving and custom applications, with Lakebase supporting sub-second interactive queries. Applications include proactive load management, real-time coaching intelligence, and enriched broadcast or fan experiences across tracking-rich sports.
How Daikin Applied Americas builds consistent data pipelines at scale with Genie Code
Daikin Applied Americas needed to scale reliable data pipelines across growing analytics and AI use cases involving operational, manufacturing, and service data while coordinating development across teams. It adopted Databricks Genie Code within a structured operating model, using Unity Catalog context, reusable MECE skills, and explicit checkpoints across Bronze, Silver, and Gold layers to guide planning and execution. The framework defines competencies such as source grain, transformation patterns, canonical alignment, governance, and business-entity modeling, moving standards out of long prompts and into the development environment. The team reports that pipelines that once took days to prototype could be generated in minutes, with faster iteration, more consistent outputs, less structural correction, reduced architectural drift, and greater trust in AI-assisted results.
What are Dashboards?
A dashboard is a visual interface that combines metrics, KPIs, and visualizations from one or more sources so users can judge performance against a goal at a glance. It acts as an organized, live visual layer over databases, warehouses, cloud applications, spreadsheets, or other sources, rather than being merely a collection of individual charts or reports. A typical flow sends source data through a query or pipeline into a visualization layer; modern dashboards may query underlying data directly and refresh on a schedule or in real time, avoiding stale copied data and definition drift associated with older extract-based architectures. Filters and drill-downs support investigation, while dashboard types—operational, analytical, strategic, and tactical—differ by audience, time horizon, and update frequency; usefulness ultimately depends on a clear purpose, defined audience, trusted data, and enough context for action.
What if the answer was already in your data?
Kythera Labs is building an AI-native healthcare strategy platform on Databricks to give health systems access to strategic intelligence that historically required specialized analysts or consulting firms. Its foundation converts 339 billion medical and prescription claims covering more than 300 million patients into governed, event-based data, resolving providers, harmonizing codes across 130 vocabularies, and reconstructing patient journeys. Healthcare Strategy Agent, built with Agent Bricks, lets executives ask questions such as where oncology referrals are going and receive analyses of leakage, competing providers, physicians, and reimbursement opportunity in minutes. A Louisiana health system went live within ten days and reported 150% greater visibility into encounters, 12% more keepage, 22% less leakage, and $3.8 million in estimated annualized retained-encounter value. Unity Catalog, Lakebase, Delta Lake, Delta Sharing, and serverless infrastructure provide shared governance, lineage, access controls, and operational integration.
Databricks positioned highest in execution and furthest in vision for the second consecutive year in Gartner Magic Quadrant
Databricks says Gartner positioned it highest for execution and furthest for vision in the Magic Quadrant for the second consecutive year. The post connects this recognition to a category reclassified from “Data Science and Machine Learning” to “AI Platforms for Data Science and Machine Learning,” and argues that agentic applications require enterprise data, governance, observability, and business context. Databricks presents a unified approach combining the lakehouse, Lakebase, Agent Bricks, Unity Catalog, and Unity AI Gateway to build, monitor, and govern agents, models, data, apps, and tools. Reported examples include YipitData’s 20x increase in company coverage with 92–95% tagging accuracy, Block’s unified AI and data estate, and Novo Nordisk’s attribution of more than $157 million in net new value to governed clinical-trial optimization.
Genesis Workbench: A blueprint for industry AI in life sciences, powered by Databricks and NVIDIA
Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.
Guide to Agentic Systems and AI Agents
Agentic AI systems are goal-directed software platforms that perceive context, plan and execute multi-step workflows, and adapt based on outcomes with minimal human intervention. The guide distinguishes them from traditional and generative AI, defining agents, broader system architecture, and the role of LLMs as reasoning cores connected to memory, APIs, databases, and other tools. It describes a perceive-reason-act-learn loop, multi-step planning, external tool integration through interfaces such as the Model Context Protocol (MCP), and orchestration patterns for coordinating specialized agents. Production concerns include retries, queues, observability, permissions, privacy, logging, and human escalation, while stated risks include reward-hacking, unintended actions, and explainability gaps. It identifies repetitive, data-rich workflows with clear success criteria and bounded error consequences as the best current enterprise candidates.
Top 10 AI Business Solutions Driving Company Growth
The article identifies ten AI business solution categories presented as growth drivers, while arguing that value is concentrated in workflows where AI changes the economics of work. It frames successful adoption around three conditions: clean, governed data; process-first use-case selection; and governance designed in from the start. Examples include customer-service agents, forecasting, personalization, intelligent process automation, and supply-chain optimization; the text says customer service accounts for 40% of top use cases, while data quality accounts for roughly 75% of what makes an AI solution work. It also describes productivity, automation, and business reimagination as distinct value paths, including a payments-data forecasting product that became an eight- to nine-figure annual revenue stream. The conclusion favors unified platforms that connect data, analytics, AI, governance, and agentic workflows.
End-to-End RAG Workflow: How Retrieval Augmented Generation Works
Retrieval Augmented Generation (RAG) connects a large language model to external knowledge at inference time, addressing outdated training data and access to proprietary or real-time information. The workflow ingests and normalizes sources, splits documents into chunks, embeds them in a vector store, retrieves context, assembles a prompt, and generates an answer. Semantic search can be combined with BM25 keyword search through reciprocal rank fusion, while reranking can improve precision; the same embedding model must be used during ingestion and querying. The guide presents evaluation and deployment considerations, including separate measurement of retrieval precision and generation faithfulness, versioning, monitoring, and containerized components. It identifies poor retrieval as the most common failure mode and explains that RAG reduces, but does not eliminate, hallucinations.
What is Vector Search?
Vector search retrieves results by comparing embeddings that represent meaning across text, images, audio, and other content rather than matching exact words. A model creates embeddings, an index stores them for fast similarity search, and a query embedding is matched against the index using nearest-neighbor methods. Exhaustive k-nearest neighbor search can become too slow at millions of items, so production systems commonly use approximate nearest neighbor search, trading some precision for speed. The guide positions vector search behind semantic search, RAG, recommendations, and multimodal or cross-language retrieval, while hybrid search combines dense and sparse vectors, keyword results, metadata filtering, and reranking to improve reliability. Quality depends on embeddings, filters, index freshness, and infrastructure, with vector search requiring more memory and compute; Databricks AI Search is presented as a managed service supporting these capabilities and Unity Catalog governance.
Data Lake vs. Cloud Data Warehouse: A Practical Guide for Data Scientists
The guide contrasts data lakes and cloud data warehouses for storing and querying data at scale. Data lakes retain raw structured, semi-structured, and unstructured data in low-cost object storage with schema-on-read, while warehouses enforce schema-on-write for structured analytical workloads. Lakes fit petabyte-scale machine learning, data science, and undefined future use cases; warehouses fit fast, concurrent SQL for dashboards, reporting, and operational analytics. It describes Bronze, Silver, and Gold zones, with Parquet and ORC supporting columnar scans and open-format portability. For teams combining ML and BI, lakehouses use Delta Lake, Apache Iceberg, or Apache Hudi to add ACID transactions, schema enforcement, and quality monitoring to lake storage without duplication; catalogs, staged checks, and access controls help prevent data swamps.
DataOps Strategy for Modern Data Engineering
DataOps is presented as a collaborative practice that applies DevOps principles—continuous integration, automated testing, and rapid delivery—to the full data lifecycle, from ingestion and transformation through trusted data-product delivery. It treats data as a continuously delivered product, with technical and non-technical participants sharing an operational cadence to improve quality and shorten time-to-insight. The recommended operating model standardizes source onboarding, enforces schemas at ingestion, uses idempotent jobs, version-controls transformations, and promotes them through tested staging and production environments, with the Bronze, Silver, and Gold layers serving as quality gates. Automated unit, data-contract, and regression tests, alongside observability, anomaly detection, SLAs, lineage, and documented rollback procedures, support reliable delivery and faster incident diagnosis. Organizations with mature DataOps practices report up to 99% fewer data-downtime incidents and 30–50% less reactive maintenance.
Data scientists: Powering the future of AI and analytics
Data scientists connect statistics, programming, and domain knowledge to turn raw data into models, recommendations, and decisions with business consequences. The role now extends beyond classical modeling to large language models, generative AI applications, agentic systems, and production workflows involving deployment, monitoring, and retraining. Modern practice combines Python, SQL, and libraries such as scikit-learn, PyTorch, Spark, and MLflow with data engineering basics, communication, and domain expertise. Data scientists contribute across problem framing, data access, preparation, feature engineering, experimentation, deployment, and lifecycle monitoring, with governed access and lineage helping align training data with production data. The article concludes that AI assistants and agents can automate routine work, but human judgment remains necessary for trustworthy evaluation, business framing, and actionable recommendations.
Data Pipeline Best Practices: Architecture, Modern Pipelines, and Deployment
Modern data pipelines move raw data from source systems through transformation and storage to downstream consumers, with batch, streaming, or hybrid operation selected according to latency needs. The guidance places service-level agreements first, defining acceptable latency, uptime, and error rates before architecture choices, and compares ETL, ELT, and zero-ETL patterns. It recommends immutable raw landing zones, CDC for transactional sources, schema registries, event replay, incremental loads, and idempotent writes using techniques such as MERGE operations. Performance and reliability practices include partitioning, clustering, data-quality checks, observability, automated CI/CD testing, and cost controls such as autoscaling and storage tiering. Adoption guidance advises piloting a representative pipeline, measuring telemetry and cost, then iterating through regular architecture reviews.
How Stagwell built privacy-safe ID matching on Databricks
Stagwell built a Databricks Marketplace App to let brands match first-party records against its Identity Spine without sending raw customer data outside their own workspaces. It combines Databricks Clean Rooms, Unity Catalog, Jobs and Notebooks, Delta Sharing, and a React/Express interface. Workspace OAuth and On-Behalf-Of authorization enforce the user's Unity Catalog permissions, while separate service principals and a scoped personal access token support app and clean-room operations. Within the clean room, a packaged notebook joins brand data with the Identity Spine, resolves multiple identifiers, and computes match rates, coverage metrics, and household and consumer IDs; results include aggregated audience insights for activation, while raw customer data and proprietary matching code remain separated.
Building a SQL ETL Pipeline: The Complete Guide for Data Engineers
SQL ETL pipelines are presented as repeatable workflows that extract data from sources, transform it, and load it into warehouses, lakes, or lakehouses for analysis and machine-learning use. The guide addresses source connectivity, extraction patterns, transformation logic, loading targets, governance, performance, testing, and operational design, while contrasting ETL with ELT and broader data pipelines. It explains that SQL can serve as the primary implementation language for transformations and load operations, with techniques including JOIN and GROUP BY, window functions, MERGE upserts, and deduplication with ROW_NUMBER() or DISTINCT. It also covers full versus incremental extraction, batch and streaming needs, schema-on-write versus schema-on-read, and layered validation using row counts, checksums, business rules, and schema-drift monitoring.
What is Artificial Intelligence (AI)?
Artificial intelligence (AI) is a branch of computer science that enables machines to perform tasks associated with human intelligence, including learning, reasoning, pattern recognition and decision-making. Modern AI generally learns patterns from large datasets, tunes internal weights and parameters during training, evaluates outputs on held-out data, and applies the resulting model during inference to classify, predict, generate content or trigger actions. Most organizations fine-tune existing foundation models rather than train from scratch, while output quality remains dependent on the completeness, bias and quality of training data. The page separates reactive machines and limited memory from theoretical theory of mind and self-aware systems, and distinguishes today’s narrow AI from theoretical general AI and superintelligence. It also describes generative AI, common applications, risks including hallucinations, bias, privacy and security gaps, and governance, concluding that practical adoption depends on real problems, trusted data and responsible oversight.
Data Engineering for AI: A Practical Guide for Data Professionals
Data engineering for AI extends traditional ETL by adding requirements for model-ready data, unstructured inputs, feature engineering, governance, and production monitoring. The guide addresses data professionals building or scaling AI infrastructure, covering ingestion, architecture, data quality, privacy compliance, generative AI, and career development. It emphasizes shared feature schemas, pipeline data contracts, lineage, statistical validation, drift detection, and PII stripping or anonymization as responsibilities shared across data engineering and data science. For generative AI systems, it describes preparing RAG pipelines by ingesting and chunking documents, creating vector embeddings, and indexing them for semantic retrieval, while evaluating vector databases for latency, scale, and integration. It concludes that reliable AI depends on fresh, accurate, compliant data and ongoing operational and architectural review.
Data Warehouse Types: A Complete Guide to Architectures and Use Cases
A data warehouse is a centralized repository for structured data, supporting complex queries, reporting, and business intelligence rather than transaction processing. The guide compares architectures by scale, latency, cost, scope, ownership, and governance. Enterprise data warehouses integrate organization-wide sources through ETL, apply cleansing and validation, and provide a governed source of truth, while data marts focus on departmental analysis and may be dependent or independent. Operational Data Stores replicate current or recent operational data for reporting refreshed from minutes to hours, whereas virtual, cloud, hybrid, and lakehouse designs trade physical consolidation, scalability, flexibility, and governance differently. The comparison also frames lakehouses as combining open-format data lake flexibility with warehouse-style governance and transactional reliability.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.