Loading…
Topic
Databricks
228 posts about Databricks. Every summary links to the original.
- 228 posts
- 3 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Stripe Data Pipeline now syncs real-time Stripe data to Lakebase Postgres
Stripe Data Pipeline now supports real-time sync to Postgres, with Databricks Lakebase as a supported destination. Stripe streams real-time API data into Lakebase and manages the sync end to end, eliminating the need for custom integrations with APIs or webhooks that require handlers, retry logic, and reconciliation processes for missed or out-of-order events. One Stripe account can feed two Databricks destinations: Unity Catalog for analytics and AI workloads, and Lakebase for applications. With real-time sync, payment, customer, subscription, refund, and payout data updates arrive in Lakebase within seconds, enabling applications to control product and feature access based on subscription status, keep billing and account details up to date without calling the Stripe API on every page load, and provide real-time financial visibility with dashboards showing daily sales volume, refunds, and payouts as they happen. The feature is available in public preview and free to get started.
Geospatial queries are up to 38X faster with Lakehouse//RT
Databricks announced that Spatial SQL and spatial joins now run on Lakehouse//RT, delivering up to 38X faster performance under high concurrency compared to traditional cloud data warehouses. The platform supports GEOMETRY and GEOGRAPHY data types, over 100 spatial functions, and high-performance spatial joins, all within the open lakehouse using Apache Spark and Parquet files. Organizations in insurance, agriculture, energy, telecom, retail, and logistics can consolidate geospatial workloads onto a single governed platform, eliminating separate stacks like PostGIS. Benchmarks using SpatialBench show Lakehouse//RT is 3X faster on average and up to 20X faster for single-query concurrency, with no code changes required. The service is available in Beta, allowing users to run existing spatial queries directly on Delta and Iceberg tables without copying data.
Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks
Atlassian faced a security data pipeline ingesting 44 billion events daily from 20 years of growth and acquisitions, each with proprietary logging formats. To unify this fragmented telemetry, the team standardized on the Open Cybersecurity Schema Framework (OCSF) and built a security lakehouse on Databricks. They embedded the entire OCSF schema into Go structs (TypeGo), guaranteeing compile-time correctness and reducing AI hallucinations during automated transformation. Table optimization with partitioning and Z-ORDER cut query times by 95%, and Graviton clusters with Photon reduced cost and runtime by two-thirds. The result was a self-service platform where SOC analysts can onboard new data sources in days, and detection teams moved from one-to-one alerts to signal-based clustering using standardized OCSF values.
How FactSet cut model training and deployment from days to hours with Databricks AI Runtime
FactSet’s Text-to-Formula (TTF) system converts natural-language investment queries from professionals into correct FactSet Query Language (FQL) formulas. Initially built on retrieval-augmented generation (RAG), the approach hit a quality ceiling as queries became more specialized and formulas more complex. FactSet adopted fine-tuned open models within a Compound AI architecture, reducing end-to-end latency by about 60 percent and improving quality for complex formulas. Fine-tuning runs on Databricks AI Runtime with serverless GPU infrastructure, where engineers choose compute based on prompt length and output needs, and the pipeline integrates with MLflow, the Lakehouse, and Lakeflow Jobs for a unified workflow. This setup cut model setup, training, and deployment from days to hours, enabling the team to retrain every two to three weeks as new data and models become available.
Unlock powerful insights and forecasts with Predictive AI Functions
Data teams frequently field questions about future sales, metric movements, and anomalous data, but answering them typically requires data science teams to build and manage models, creating a bottleneck. Predictive AI Functions, now in Beta, remove that bottleneck by providing SQL-native functions for forecasting, driver analysis, and anomaly detection, backed by state-of-the-art ML models maintained centrally by Databricks. The launch includes ai_forecast for time series projection, ai_top_drivers for segment driver analysis, ai_detect_anomalies for outlier detection, and ai_predict_value and ai_predict_class for regression and classification. These functions can be called directly in SQL, invoked through Genie with plain-language questions, or embedded in dashboards, delivering consistent, high-quality answers without infrastructure management. Benchmarking on fev-bench shows ai_forecast outperforms forecasting functions from leading cloud data warehouses, achieving industry-leading accuracy out of the box with no tuning required.
Power real-time applications and agents with Document Intelligence
Databricks announces General Availability of Document Intelligence REST APIs for real-time document parsing, extraction, and classification, with the ai_decide API for routing in beta. These synchronous APIs eliminate the manual bottleneck of reading documents by transforming unstructured contracts, invoices, and insurance claims into structured insights in real time. The ai_parse_document API converts tables and figures into structured text with citation bounding boxes, while ai_extract pulls specified fields with optional citations and confidence scores for verification. The ai_classify API assigns labels using a business taxonomy with optional rationales, and ai_decide uses content to route or flag documents for review. Developers can integrate these APIs into Databricks Apps for human-in-the-loop workflows, such as a contract intake flow where a lawyer uploads an agreement and sees key terms prefilled, or into custom agents on Agent Bricks.
Sizing Lakebase Search: lessons from a real app and 4 million passages
Slide Hub, an internal Databricks app for slide search, uses Lakebase Postgres with Lakebase Search extensions (lakebase_vector and lakebase_text) to run hybrid vector and BM25 search on the same database as application data. The app stores 23k slides and indexes up to 4M passages from the MIRACL English corpus. Benchmarking shows that below 8 Compute Units (CU), Lakebase Postgres is the bottleneck; above 8 CU, the app’s worker count limits throughput. Candidate pool size significantly affects cold-start latency: 200 candidates preserves 90.5% of top-10 results while touching a third of the pages of 1,000 candidates. A 14-minute load test cost 0.13, with Lakebase compute billing only while active. For production, a 0.5 CU minimum costs ~ 31-40/month and avoids cold-start delays.
Introducing Private Network Gateway
Databricks announced Private Network Gateway (PNG) in public preview on Azure and private preview on AWS, addressing the networking friction that prevented teams from moving workloads to serverless compute. PNG provides a single managed gateway from serverless compute into a customer’s virtual network, eliminating the need for per-resource private endpoints, load balancers, and proxy VMs. Customers configure the gateway via a Network Connectivity Configuration (NCC) and delegate one subnet, with Databricks managing the gateway infrastructure while customers retain control over routing and security. PNG supports use cases such as ingesting from private databases via Lakeflow Connect, connecting AI agents to private MCP servers, using private package repositories, and routing egress through customer firewalls. The gateway offers two traffic modes for precise control or broad egress inspection and works alongside existing private connectivity without disruption.
Databricks is a Visionary the Gartner® Magic Quadrant™ for Analytics and Business Intelligence Platforms
Databricks debuted as a Visionary in the 2026 Gartner Magic Quadrant for Analytics and Business Intelligence Platforms, achieving the highest debut performance in the report’s 20+ year history. The company also received one of the highest Completeness of Vision scores among all vendors evaluated. Databricks attributes this recognition to its ability to deliver on three major trends reshaping businesses: data and AI platform convergence, agentic analytics, and semantics at the core. The Databricks lakehouse integrates BI directly, eliminating data silos and the need for extracts, so agents can reason over complete, governed context. With Genie and AI/BI, the platform automates complex workflows and provides a unified semantic layer through Unity Catalog Business Semantics and Genie Ontology.
What's New in Lakeflow Designer - October 2026 Roundup
Lakeflow Designer, a visual no-code data preparation tool in Databricks, added three new operators for cleaning, deduplication, and quality checks, addressing customer requests for more no-code capabilities. The Prepare operator standardizes data by changing case, converting types, replacing values, and applying formulas; Unique removes duplicates across selected columns with ordering control; Guardrails enforces quality rules with configurable failure handling. Existing operators gained multi-table joins with separate outputs for matched, left unmatched, and right unmatched rows, branching in Filter, and dynamic column selection in Select. Output options expanded to materialized views in Unity Catalog that refresh automatically, merge into existing tables, and direct file writes to CSV, Excel, or JSON with dynamic worksheets. Canvas improvements include a redesigned interface with better contrast and larger connection targets for large workflows, a searchable table of contents, visual version history, and live cursor presence.
Closing the AI context gap: what it takes to make enterprise AI trustworthy
The post argues that model intelligence is no longer the bottleneck for enterprise AI; the real gap is business context — the unique knowledge about an organization’s processes, customers, and products. Without proper grounding, agents resort to brute force, iterating through guesses and burning tokens while producing unreliable answers. While building a well-grounded agent for a single domain is now easier, scaling context across hundreds of agents in an enterprise remains unsolved due to fragmentation, fast-changing definitions, and differing team-specific interpretations of the same concepts. Genie Ontology addresses this by providing a unified, governed context layer that combines intentionally curated knowledge with automatically learned observations from system usage. This enables every agent connected to Databricks to work from the same consistent business understanding, supporting accurate answers without each agent rediscovering the business from scratch.
How Flo Health built safer, more useful open models with Databricks AI Runtime
Flo Health, a women’s health app with 81 million monthly active users, aimed to improve generative AI responses for medical safety and usefulness while operating at scale. Relying on general-purpose models created cost, dependency, and domain-expertise limitations, so Flo built a post-training pipeline with Databricks AI Runtime. The team defined 26 AI judges across three categories: 12 medical-safety judges acting as hard constraints, 6 micro judges for nuanced behaviors, and 8 usefulness criteria. Using these judges as a reward signal, they generated over 15,000 synthetic questions from privacy-first principles and applied reinforcement fine-tuning with supervised fine-tuning (SFT) and direct preference optimization (DPO). After fine-tuning on 32 NVIDIA H100 GPUs using PyTorch and Fully Sharded Data Parallel (FSDP), medical safety scores rose from 34% to 61%, usefulness from 42% to 76%, and the combined metric improved nearly threefold to 47%.
Brickster Voices: Meet Clara Kurian
Clara Kurian, Databricks’ first university recruiter for APJ, leads the company’s early career hiring in India. Databricks has invested over $250 million in India, treating it as a strategic hub rather than a satellite office, with workforce growth of 270% since 2022 and leadership expansion of 350% across all business units. Clara designed the first structured India Intern Program with 51 interns, aiming to give them a genuine experience of the company’s pace, people, and culture through Speed Mentoring with engineering leadership and activities like Sneaker Art and Pickleball socials. She advises early career candidates to build strong fundamentals, curiosity, and a builder mindset, noting that the field moves fast and thrives on continuous learning. Candidates stand out by showing genuine curiosity, authenticity, and a willingness to figure out unknown problems, as the technical bar is almost a given.
From post-training to production: How TinyFish runs its web-native model on Databricks AI Runtime
TinyFish builds infrastructure for enterprises that need agents to work across the live web and is developing its own specialized web-native model. The company uses Databricks AI Runtime to manage GPU compute for post-training with supervised fine-tuning and moving toward Direct Preference Optimization, MLflow for evaluation, and Model Serving for production deployment. TinyFish evaluates its model by measuring full agent task completion rather than individual responses, achieving a 77.6% success rate on its internal go/no-go benchmark where 1,691 of 2,178 tasks were completed successfully. Over a 15-day period, the web-native model handled 294,781 agent runs across 6.4 million tool steps at an estimated serving cost of $0.088 per run or $0.0040 per tool step. By consolidating training, evaluation, serving, and operational data on Databricks, TinyFish reduces infrastructure overhead and maintains a continuous development cycle from agent behavior to deployment.
How California built its Behavioral Health Public County Profile on Databricks
The California Department of Health Care Services built the Behavioral Health Public County Profile on Databricks, a public dashboard consolidating behavioral health data from all 58 counties for nearly 40 million Californians. Launched on October 1, 2026, following the 2024 passage of Proposition 1, the dashboard supports the new Behavioral Health Services Act (BHSA) tying funding to statewide goals for outcomes, equity, and transparency. The platform uses Spark Declarative Pipelines and Lakeflow Connect for unified data ingestion, Semantic Views for standardized metrics across counties, and serverless compute to process data at scale without infrastructure management. It operates on a FedRAMP-authorized environment meeting federal security standards for health data, with data quality and lineage ensuring every number can be traced to its source. The County Profile enables public comparison of county plans, funding, service delivery, and performance measures, creating a feedback loop between policy intent and measurable outcomes for democratic oversight.
Solving defense supply chain visibility through governed data sharing
The Department of War cannot obtain cost transparency across its supply chain of over 200,000 suppliers due to reliance on manual data calls and fragmented spreadsheets. The proposed solution is a federated data exchange where each contractor publishes validated cost and pricing evidence from existing records to named government recipients using open-source sharing protocols. Databricks provides the platform, enforcing governance, access controls, CUI markings, and audit lineage without storing raw records centrally or requiring contractors to adopt Databricks. Small and mid-tier suppliers submit through validated intake forms that produce the same governed data contracts as large primes, requiring no new accounting systems. The reporting obligation already exists through CSDR, FlexFile, and DD Form 1921, so a pilot exchange can launch in months by defining two data contracts on a single program.
Introducing Funke: Native HL7v2 Parsing on Databricks
HL7v2 is the dominant healthcare messaging standard but is difficult to parse due to its nested, delimiter-encoded structure. Existing workarounds like converting to FHIR or using third-party engines add cost and latency. Funke, the successor to the open-source Smolder library, is a new open-source Python and PySpark library under the Databricks License that parses HL7v2 messages directly into native Spark types, preserving the full hierarchy losslessly. It deploys as a Databricks Asset Bundle with a Declarative Pipeline that uses Auto Loader to ingest raw messages into a Unity Catalog volume, then parses them into structured silver tables following the medallion pattern. The library supports all HL7 message types and versions, includes accessor helpers for extracting fields by segment and component, and ships with a demo using synthetic ADT events to illustrate end-to-end ingestion and gold table creation, including a live census and bed utilization dashboard.
Biomedical Imaging's Real Bottleneck Is the Data, Not the Model
Medical imaging’s real bottleneck is not model architecture but data access. Hospitals, academia, medtech, and pharma all struggle to extract, de-identify, and query imaging data locked inside PACS and vendor archives. The approach proposed is a data lakehouse model on Databricks using a medallion architecture that ingests raw DICOM into Bronze, extracts de-identified metadata into Silver, and curates queryable Delta tables in Gold. This foundation unifies imaging with multi-omics, clinical-trial, and real-world data under a single governance model. The algorithms exist; the missing piece is a governed, queryable, and linkable data layer that makes the model the easy 10% again.
Set Budgets and Alerts for Cloud Data Warehouse Costs
Reactive cost management for cloud data warehouses fails because usage is unattributed, budgets are absent, and alerts arrive too late. Databricks’ proactive framework addresses this through five pillars: cost attribution via warehouse and query tags, layered budgets with email alerts, real-time dashboards, and optimizations. SQL Serverless warehouses eliminate idle compute costs, as Lumen Technologies demonstrated with a 30-40% reduction. Query tags (Public Preview) enable per-query attribution, linking spend to specific dashboards or dbt models. The post provides step-by-step instructions for tagging, budget creation in the Account Console, and tuning auto-stop and statement timeouts to reduce waste.
Announcing Workday Data Connect federation in Unity Catalog
Databricks announced the public beta of a Workday Data Connect federation connector for Unity Catalog, enabling zero-copy queries of Workday HR and finance data directly from the Databricks Data and AI Platform without ingestion pipelines. Workday shares tables via an Iceberg REST catalog; Databricks creates a foreign catalog in Unity Catalog and reads data from Workday’s managed object storage, with Unity Catalog governing read-only access through table-level permissions, lineage, and auditing. Teams can query the federated data using Databricks SQL, AI/BI, notebooks, or Genie for natural-language questions about workforce and financial trends. The connector requires Databricks Runtime 19 or above and a workspace admin to enable it from the Previews page. The release builds on the Databricks-Workday partnership announced at Workday Rising 2025 and complements existing Lakeflow Connect and Live Data Query options.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.