Every post, summarised

Sep 18, 2026

• Databricks

  What is Intelligent Document Processing?

  Intelligent document processing (IDP) uses AI to extract, classify, validate, and structure information from PDFs, images, emails, forms, and other structured, semi-structured, or unstructured documents. Its pipeline combines document ingestion and preprocessing with OCR, document classification, NLP, layout analysis, data validation, human review for uncertain results, and delivery into downstream systems such as ERP, CRM, or accounting software. The post explains how embeddings support classification, how continuous learning adapts models to new layouts and corrections, and how metrics such as accuracy, throughput, confidence scores, and model drift support optimization. IDP can reduce manual effort, improve accuracy, lower costs, scale document-heavy workflows, and supply structured data for analytics, automation, AI agents, and retrieval-augmented generation, while high document variability and edge cases still make human-in-the-loop review relevant.

• Databricks

  What are Analytic Applications?

  Analytic applications are packaged business intelligence systems designed to measure, analyze and improve performance for defined business domains. They combine data integration, modeling, visualization, reporting and workflow support so technical and non-technical users can turn operational data into decisions. Typically, they connect ERP, CRM and supply chain systems with a data warehouse or lakehouse, process historical and real-time data, and deliver dashboards, reports, scorecards or embedded insights through preconfigured schemas, connectors, governance controls and business logic. Unlike general-purpose BI tools built for open-ended exploration, they support standardized, domain-specific workflows such as sales performance management, financial planning, risk assessment and inventory optimization. Organizations can start with descriptive analytics and adopt predictive or prescriptive capabilities as data maturity increases, while choosing standalone or embedded deployment based on workflow needs and data readiness.

• Databricks

  Building an A/B testing analysis framework for mobile gaming on Databricks

  HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.

• Databricks

  Tevogen Bio’s Journey to Streamlining Life

  Tevogen Bio faced drug development costing upwards of $3 billion and requiring 10–12 years; selecting targets for its single-HLA-restricted SARS-COV2 product took 18–24 months of wet-lab work. Tevogen.AI partnered with Microsoft and Databricks to build and curate a multi-terabyte protein-sequence library and parallelize processing for machine-learning training. Databricks Data + AI Platform, Medallion Architecture, Unity Catalog, and distributed computing cut processing from 50 days to 24 hours under governed, fine-grained access controls. The dataset reached 24 million proteins, 16 billion datapoints, ~700 million unique peptides, and ~37 million cross-matched expert articles. An MLOps framework automated training, inference, monitoring, and retention; the alpha PredicTcell model used XGBoost and ESM models and reported 93–97% recall and 38–43% accuracy, while beta PredicTcell and alpha AdapTcell work began.

• Databricks

  What is a Cloud

  A cloud-based database management system stores, manages, and queries data on third-party cloud infrastructure instead of local or on-premises servers. It retains capabilities such as query optimization, security, transactions, and support for relational SQL, NoSQL, data warehouse, and in-memory workloads, while operating across distributed nodes. Organizations can self-manage database software on cloud virtual machines or use DBaaS, where the provider manages the software and infrastructure; connections typically use SQL, APIs, or web consoles. Virtualized resources can scale, and providers may handle patching, backups, maintenance, and failover, while replication across availability zones supports recovery and availability. The comparison with on-premises systems covers ownership, scalability, cost models, maintenance, accessibility, control, compliance, total cost, and emerging capabilities including AI integration, vector search, autonomous operation, multicloud, hybrid, and serverless architectures.

• Databricks

  Fizz accelerates ecommerce analytics with Databricks SQL

  Fizz.hu, an ecommerce marketplace backed by OTP Group, outgrew a Microsoft SQL Server and Power BI setup that relied on daily batch loads and separate virtual-machine Python workloads. The company migrated about 50 tables and several stored procedures to Databricks SQL, recreated core views, and moved to a lakehouse architecture using an MVP-first approach rather than redesigning everything. Completed in three months, the migration switched off SQL Server while keeping Power BI reports running on the new engine. Nightly processing fell from three to four hours to roughly 90 minutes, reports became consistently available by 4:30 a.m., Power BI refreshes dropped about 50%, and gigabyte-scale exports finish in minutes. Databricks now unifies SQL, Python, machine learning, governance work, and Hungarian-language Genie self-service, with about 20 users reclaiming roughly 20% of an analyst’s time.

• Databricks

  Zero downtime patching lakebase part-1 prewarming

  Lakebase addresses the workload disruption caused by planned PostgreSQL maintenance, where a restart clears in-memory buffer and local file caches. Although the database can return in about one second at P99, the source reports a roughly 70% pgbench TPS reduction while caches warm, potentially causing timeouts. Lakebase uses stateless Postgres compute replicas and disaggregated shared storage to prewarm a replacement compute before a scheduled restart. The control plane transfers cached-page information, loads pages from shared storage, and follows relevant WAL from Safekeepers, while the current primary continues serving traffic. After prewarming, Lakebase promotes the new compute with PostgreSQL’s pg_promote without restarting the server; tests on 10 GB pgbench workloads found nearly instant throughput recovery, especially for read-only queries.

• Databricks

  Protect Performance and Reduce Surprise Costs with Default Warehouse

  Databricks SQL’s generally available Default Warehouse lets administrators choose the SQL warehouse automatically used for ad hoc experiences, addressing manual selection and fallback behavior such as “Last Selected” or alphabetical ordering. A workspace-level setting applies across SQL Editor, Catalog Explorer, AI/BI Dashboards, Alerts, and Genie Spaces, while users can override it or retain “Last Selected”; admin APIs and a Terraform resource support per-user assignment, auditing, and control. The feature supports serverless, pro, and classic SQL warehouses and respects existing governance and access controls, with the setting opt-in and new assets automatically using it. In customer results, Catalog Explorer queries sent to smaller warehouses rose from 77% to 96%, and users running ad hoc SQL across multiple warehouses fell by 15%, while future coverage is planned for MLflow, Lakeflow CLI, and DBSQL MCP.

• Databricks

  The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks

  Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.

• Databricks

  What is Agentic Analytics?

  Agentic analytics is an approach in which autonomous AI agents continuously monitor data, detect patterns and anomalies, generate contextual insights, and take actions with minimal human intervention. Unlike traditional business intelligence, which relies on people to query dashboards and interpret reports, these systems operate as goal-directed decision loops that can recommend or trigger alerts, forecasts, workflows, and system changes within defined boundaries. Their architecture combines data access, agent orchestration, LLM-based reasoning, a semantic layer, action integrations, and feedback signals. The source identifies faster insight, lower reporting overhead, reduced analytics backlogs, and potentially better forecasting as benefits, while stressing governance, auditability, security, monitoring, and human oversight to control errors and unintended outcomes.

• Databricks

  Database Branching in Postgres: Git

  Shared staging databases often drift as schemas diverge, test data accumulates, and developers avoid creating isolated environments because traditional pg_dump copies are slow and expensive. Databricks Lakebase Postgres addresses this with database branching, which creates fully isolated environments from a parent’s exact schema and data at a point in time. Branches use copy-on-write: they share underlying versioned storage until a branch changes, then record only new data, while each branch has independent compute and a separate connection string. Idle branches scale compute to zero, and storage cost tracks changes rather than the full database size. This supports per-developer, per-pull-request, per-test-run, and recovery workflows, including point-in-time branching within the restore window, with the source describing branch creation through the console, CLI, or API in under a minute.

• Databricks

  Memory scaling for AI agents

  Memory scaling is proposed as an agent’s performance improving when it accumulates past conversations, feedback, interaction trajectories, and business context in persistent external memory. Unlike model-parameter or inference-time scaling, it relies on selective retrieval to supply relevant information without filling long context windows. In MemAlign experiments on Databricks Genie Spaces, adding labeled examples raised test scores from near zero to 70%, about 5% above an expert-curated baseline, while average reasoning steps fell from about 20 to 5; filtered user logs produced similarly strong gains, exceeding 50% after 62 records. Production use requires storage and retrieval for episodic and semantic memories with organizational and user-level scoping, plus controls that preserve permissions and sensitivity labels.

• Together AI

  How a global fintech scaled coding agent traffic with Dedicated Model Inference

  A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.

• Stripe

  Building custom financial reconciliation workflows with Stripe

  The post examines how enterprises can replace SFTP-based financial reconciliation with Stripe’s reporting APIs. Stripe’s Reports API v2 and Query Run API provide SQL-based report customization, consistent schemas, API-key permissioning, audit trails, and event-driven completion notifications instead of fixed files and polling schedules. Using fictional retailer Galtee, the examples run payout queries across multiple Stripe accounts with an Organization secret key, joining transfers with balance_transactions and using a succeeded webhook to trigger retrieval. Further queries join charges to add presentment amounts and exchange rates, then use aggregation and a common table expression to itemize IC+ fees; the conclusion is that this supports an on-demand pipeline that evolves with business requirements.

• Auth0

  Auth0 Agent Gateway: The Identity Control Plane for Customer-Facing AI

  Auth0 Agent Gateway is announced in beta as a Customer Identity and Access Management (CIAM) control plane for governing product-native, customer-facing AI agents in multi-tenant SaaS applications. It addresses a trust model involving organizations, delegated user access, customer consent, independently owned connected accounts, and cross-tenant data boundaries, rather than workforce-only identity. Positioned between an agent and its tools, the gateway evaluates each call using agent, user, and organization context; it uses Organizations, Agents as Principals, Token Vault, token exchange, tenant logs, and administrative controls. The described capabilities include short-lived agent-only tokens, user-scoped credentials, runtime tool authorization, audit logging with optional SIEM streaming, and an agent kill switch. Auth0 is seeking product and engineering teams to join the beta.

• Auth0

  Move and Scale Faster with FGA Permissions Index

  Auth0 FGA Permissions Index is announced in Early Access as a way for B2B SaaS applications to enforce permissions while handling expanding AI-agent activity and complex relationship graphs. Customers select relations used for search and filtering, typically high-cardinality relations such as can_view, and the index precomputes the access those relations imply. It streams the resulting data into the customer’s database alongside application data, allowing indexed permission checks through SQL JOINs instead of repeated authorization API calls or runtime graph traversal. Computation moves to relationship-write time, while relations outside the index continue to be evaluated by FGA in real time. The announcement positions this design as a path to consistently low-latency filtering at enterprise scale, with permission decisions described as precomputed, traceable, and compliant.

• Auth0

  Optimize Your Identity Flows with Auth0 Identity Conversion Suite

  Auth0 Identity Conversion Suite addresses identity friction across the visitor-to-user journey, where forced account creation and complicated checkout are cited as contributors to abandonment. It combines Anonymous Sessions, which uses OAuth 2.0 access tokens to track pre-authentication activity and transfer session data into an authenticated profile, with Experiment Center, which supports controlled authentication-flow tests. Visitors can be assigned to percentage- or segment-based variants, kept in the same variant during their journey, and measured through exposure, success-rate, and time-to-auth events across supported Auth0 surfaces. The Early Access release is available to Enterprise-plan Auth0 customers and is presented as a way to improve personalization, onboarding, guest checkout, and authentication decisions without changing application APIs or code.

• Apple

  Dynamically Scaled Activation Steering

  Activation steering has emerged as a way to guide generative models toward desired behaviors, but uniform interventions can reduce performance when steering is unnecessary. DSAS, or Dynamically Scaled Activation Steering, addresses this by separating when to steer from how to steer and adapting intervention strength across layers and inputs. At generation time, it computes context-dependent scaling factors that modify existing steering transformations, and it can also be jointly optimized end-to-end with the steering function. Combined with existing methods, DSAS improves the reported Pareto front between toxicity mitigation and utility preservation, while adding minimal computational overhead; experiments also apply it to a text-to-image diffusion model for modulating specific concepts and show token-level interpretability through the strength and location of steering.

• Databricks

  coSTAR: How we ship AI agents at Databricks fast, without breaking things

  Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.

• Databricks

  Business Analytics Tools: A Complete Guide for Data

  Business analytics tools have evolved beyond static reporting as organizations seek explanations, forecasts, and immediate actions. This guide categorizes visualization and dashboard platforms, self-service analytics, statistical tools, spreadsheets, and SQL-based query tools, then examines how AI adds natural-language querying, anomaly detection, summaries, and forecasting. It argues that inconsistent, duplicated, or differently governed data can undermine even sophisticated dashboards. Lakehouse architecture combines data-lake scalability with warehouse governance, performance, and SQL compatibility, enabling tools such as Tableau, Power BI, and Looker to use broader, fresher datasets alongside analytics and AI workloads. Reported examples include 94% faster BI queries at Anker Innovations, 98% faster query performance at AnyClip, and consolidation across 120+ global analysts at JLL.