--- title: "Latest reads" description: "The engineering internet, summarised so you can actually read it." --- # Latest reads > The engineering internet, summarised so you can actually read it. ## Articles ### [What is data pipeline architecture?](https://yomu.fyi/post/what-is-data-pipeline-architecture.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Databricks Staff - Published: Jun 16, 2026 - Type: Explainer Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability. ### [Build Your Own Transaction Foundation Model for Financial Intelligence](https://yomu.fyi/post/build-your-own-transaction-foundation-model-for-financial-intelligence.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Benjamin Wu - Published: Jun 16, 2026 - Type: Tutorial Production financial intelligence systems frequently rely on hand-engineered tabular features and rule sets that are brittle, expensive to maintain, and blind to historical sequential patterns. NVIDIA demonstrates an accelerated reference pipeline that replaces standard BPE tokenization with a GPU-based domain tokenizer, converting raw transactions into semantic tokens with an 8,192-token context window. Using the NeMo AutoModel library, a compact 29-million parameter decoder-only transformer is pretrained from scratch on unlabeled transaction sequences using causal language modeling. Learned sequence embeddings are extracted, compressed using PCA, and concatenated with raw tabular features to train a downstream GPU-accelerated XGBoost fraud detection model. On the IBM TabFormer benchmark dataset, this combined approach achieves a 41.76% lift in Average Precision over the baseline model relying solely on raw tabular features. ### [NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance](https://yomu.fyi/post/nvidia-blackwell-tops-mlperf-training-6-0-with-industry-leading-scale.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Farshad Ghodsian - Published: Jun 16, 2026 - Type: Benchmark NVIDIA submitted results across all MLPerf Training v6.0 benchmarks, setting performance records on workloads including the DeepSeek-V3 and GPT-OSS-20B Mixture of Experts models. To scale training up to 8,192 Blackwell GPUs, NVIDIA combined hardware cluster designs with scale-out networking via Spectrum-X Ethernet and Quantum InfiniBand. The engineering team eliminated CPU-GPU synchronization bottlenecks in token-dropless MoEs by implementing full-iteration CUDA graphs with synchronization-free operators and paged stashing. Further software optimizations included CuTe DSL kernel fusions, an MXFP8 attention block in the Transformer Engine, and transitioning router elementwise math to FP32. Across these benchmarks, the GB300 NVL72 platform achieved the fastest time to train at scale and the highest normalized per-accelerator performance. ### [Enabling Governed Vibe Coding for Enterprise Apps on Databricks](https://yomu.fyi/post/enabling-governed-vibe-coding-for-enterprise-apps-on-databricks.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Evan Pandya, Justin DeBrabant, Cong Xu - Published: Jun 16, 2026 - Type: Announcement Databricks introduces three capabilities intended to bring vibe coding to enterprise applications, where speed alone does not provide business-data context, deployment safety, or cost control. App Spaces lets admins define resource and data access, on-behalf-of-user API scopes, and security policies for groups of apps, with each app inheriting those settings. Genie App Builder turns plain-language descriptions into working internal apps through generated plans, live previews, AppKit, and awareness of workspace data assets and Unity Catalog semantics. Serverless micro apps run in isolated lightweight virtual machines, start quickly when needed, scale to zero while idle, and use usage-based rather than reserved-capacity infrastructure. Together, the capabilities are presented as a way for business-proximate users to build on enterprise data while organizations apply consistent governance and support broader app portfolios; all three are coming to Databricks Apps, with private previews coming soon. ### [Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era](https://yomu.fyi/post/introducing-opensharing-the-next-evolution-of-delta-sharing-for-the-ag.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Huey Han, Harish Gaur, Akram Chetibi, Mengxi Chen - Published: Jun 16, 2026 - Type: Announcement OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls. ### [Announcing Apps on Databricks Marketplace](https://yomu.fyi/post/announcing-apps-on-databricks-marketplace.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Tia Chang, Akram Chetibi, Harish Gaur, Stephen Orban, Mengxi Chen - Published: Jun 16, 2026 - Type: Announcement Databricks announces the Public Preview of Apps on Databricks Marketplace, allowing customers to discover, install, and run third-party data and AI applications inside secure Databricks workspaces. The offering addresses procurement challenges involving data movement, lengthy security reviews, custom integrations, and fragmented identity management by bringing applications to the customer’s data. Installed apps run in isolated sandboxes within the consumer’s Databricks account, inherit Unity Catalog governance, and use dedicated serverless compute with consumer-controlled external access through Serverless Egress Gateway policies. Providers can publish closed-source containerized apps once for no-egress distribution without maintaining per-customer infrastructure, while applications connect natively to services including SQL Warehouse, Lakebase, Model Serving, and Foundation Model APIs. The Public Preview launches with 20 partners, and planned additions include bundled assets, provider analytics, and commercial monetization. ### [Introducing OpenSharing SecureConnect](https://yomu.fyi/post/introducing-opensharing-secureconnect.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Huey Han, William Chau, Harish Gaur - Published: Jun 16, 2026 - Type: Announcement OpenSharing SecureConnect addresses the networking burden of sharing live data from provider storage behind private networks, where providers and recipients otherwise exchange firewall and egress details manually. It is a Databricks-managed proxy that routes recipient storage access through Databricks endpoints after a one-time provider setup, while the data remains in the provider’s bucket. Providers allowlist Databricks Serverless Data Plane endpoints and enable SecureConnect for a metastore; serverless recipients require no configuration, while classic and open recipients allowlist stable inbound IPs. Optional NCC provides private link connectivity, and mutual TLS is available for recipients. SecureConnect is in Public Preview and supports cross-region, cross-cloud sharing plus customer-managed and Databricks Default Storage. ### [Build On-Device AI Companions with the NVIDIA ACE Game Agent SDK and Unreal Engine 5 Plugins](https://yomu.fyi/post/build-on-device-ai-companions-with-the-nvidia-ace-game-agent-sdk-and-u.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Phillip Singh - Published: Jun 16, 2026 - Type: Announcement Game developers integrating on-device artificial intelligence companions face challenges such as conversational latency, state synchronization, and loop prevention. NVIDIA announced the open-source ACE Game Agent SDK alongside a suite of Unreal Engine 5 plugins to streamline native, hardware-accelerated character pipelines. The SDK offers Agent, Chat, and RAG APIs, enabling characters to execute multi-step tool-assisted reasoning and query game databases. Unreal Engine 5 plugins introduce local runtime models including nemo-conformer-ctc-120m for speech recognition, Qwen 3.5 4B for language generation, and Chatterbox Turbo 350M for speech synthesis. Additional tooling expands to motion generation via Animotive Kimodo and rendering updates with the DLSS 4.5 Unreal Engine plugin. ### [How to Optimize Transformer-Based Models for Low-Precision Training](https://yomu.fyi/post/how-to-optimize-transformer-based-models-for-low-precision-training.md) - Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md) - Author: Jonathan Mitchell - Published: Jun 16, 2026 - Type: Benchmark Accelerating transformer training requires understanding how low-precision formats such as FP8 and NVFP4 impact specific general matrix multiplication (GEMM) workloads. Transformer configurations do not explicitly reveal these shapes, making microbenchmarks necessary before committing to full training runs. NVIDIA Transformer Engine enables quantization and kernel dispatch across precisions, which can be evaluated in realistic autocast mode or kernel-only prequantized mode. Profiling the ESM2-15B model on NVIDIA B300 GPUs demonstrated that NVFP4 achieved a 1.79x blended forward propagation speedup over MXFP8 and up to 4.01x over BF16 in prequantized execution. Although large GEMM dimensions successfully overcome quantization overheads, dynamic scaling, Hadamard transforms, and kernel selection asymmetries moderate real-world gains. ### [Cart Assistant: Agentic Grocery Shopping on Uber Eats](https://yomu.fyi/post/cart-assistant-agentic-grocery-shopping-on-uber-eats.md) - Company: [Uber](https://yomu.fyi/company/uber.md) - Published: Jun 16, 2026 - Type: Explainer Cart Assistant addresses grocery requests that begin as lists, recipe images, or vague plans and turns them into draft carts shoppers can review and edit. It uses a multi-prompt state graph that assigns planning, candidate retrieval, semantic relevance, constraint enforcement, quantity selection, guardrails, assembly, and content refinement to focused stages. LLMs handle ambiguity and structured outputs, while backend APIs, deterministic arithmetic, validation, optimization, and aggregation manage inventory, pricing, eligibility, sellable units, and cart construction. The design separates retrieval language from reasoning context, preserves quantity provenance, supports parallel item processing, and uses deterministic checks plus an LLM judge against synthetic and production-derived cases. According to the conclusion, this decomposition makes the system more controllable and inspectable while producing useful draft carts without removing shopper review, editing, and decision-making. ### [Sciene AI Companion: building an autonomous Customer Success platform on Databricks](https://yomu.fyi/post/sciene-ai-companion-building-an-autonomous-customer-success-platform-o.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Renata Fencz, Solano Campos, Rodrigo Mohr, Ricardo Morandini - Published: Jun 16, 2026 - Type: Problem & solution Sciene built AI Companion for Quartile’s Customer Success organization, where CSMs support more than 1,000 brands and previously spent substantial time preparing decks, reconstructing context, and investigating account changes. The platform addresses personalization at scale, high-volume content generation, and root-cause diagnosis by combining account data, CSM communication styles, company principles, and cross-domain business data. Its Email Hub cuts reply time from 15–30 minutes to about three minutes, Meeting Hub reduces preparation for 80+ slide decks from over two hours to around 10 minutes, and Account Flagging reduces diagnosis of flagged accounts from 30+ minutes to about five. Databricks provides the shared governed foundation: Delta Sharing supplies data without copies, Lakebase stores operational state, and SQL Warehouses serve analytical, AI, and operational workloads from the same tables. The design keeps CSMs responsible for judgment while giving them current context for customer interactions. ### [What’s new with Unity Catalog at Data + AI Summit 2026](https://yomu.fyi/post/what-s-new-with-unity-catalog-at-data-ai-summit-2026.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: The Unity Catalog Product and Engineering Team - Published: Jun 16, 2026 - Type: Announcement At Data + AI Summit 2026, Unity Catalog announcements position the catalog as a runtime governance layer for enterprise data and AI, organized around control, context, and choice. Control additions include Unity AI Gateway for governing models, agents, MCP services, skills, and tools; contextual service policies can allow, deny, or require approval for runtime actions, while budgets, hard caps, tracing, and guardrails address spend, investigation, and safety. Context additions include Glossary and Domains for business meaning and scoped asset organization, plus Metrics that standardize KPIs for SQL, BI tools, APIs, and agents; Genie Ontology is described as a continuously learned enterprise context layer. Choice additions span cross-cloud and cross-region addressability, managed disaster recovery, Delta and Iceberg interoperability, multimodal and geospatial types, and open sharing of data, AI assets, and applications across organizations. ### [Agent Bricks: Data + AI Summit 2026](https://yomu.fyi/post/agent-bricks-data-ai-summit-2026.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Hanlin Tang, Kasey Uhlenhuth, Akhil Gupta, Patrick Wendell - Published: Jun 16, 2026 - Type: Announcement At Data + AI Summit 2026, Databricks announced Agent Bricks as a comprehensive developer platform for building and operating agents, extending a product launched the previous year. The announcement frames the core agent loop as only 1% of the work, with token capacity, deployment, security, evaluation, monitoring, context, and sharing forming the remaining infrastructure burden. Agent Bricks addresses choice, context, and control through support for multiple proprietary, open-source, and custom models, any agent harness, MCP-connected data, Genie Ontology, managed memory, document intelligence, sandboxes, and governed tools. Unity AI Gateway adds catalogs, fine-grained access controls, budgets, traffic routing, contextual policies, monitoring, and registry support for agents, tools, and models. Databricks says more than 100,000 agents have been built and customers including AstraZeneca, 7-Eleven, Fox Corporation, and Block have shipped agents on the platform. ### [AI governance at Data + AI Summit 2026: What’s new with Unity AI Gateway](https://yomu.fyi/post/ai-governance-at-data-ai-summit-2026-what-s-new-with-unity-ai-gateway.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: David Nasi, Stefania Leone, Ahmed Bilal, Kevin Stumpf, Martin Grund, Vladimir Kolovski, Kelly Albano - Published: Jun 16, 2026 - Type: Announcement Databricks announces new Unity AI Gateway capabilities for governing enterprise AI as organizations operate multi-model, multi-agent, and multi-vendor estates connected to models, MCP services, APIs, and tools. The update adds unified spend visibility, granular attribution, hard spend caps, and smart routing, alongside Unity Catalog support for registering and governing models, MCP services, agents, and skills. Contextual Service Policies, in Beta, can allow, deny, or require approval for actions based on users, agents, models, tools, services, or request and response contents, with guardrails for risks such as PII exposure and prompt injection. The announcement also covers end-to-end tracing, coding-agent analysis with Genie, incident investigation with Lakewatch, ecosystem integrations, and Managed Omnigent on Databricks in Beta. ### [Lakeflow: A new era of agentic data engineering](https://yomu.fyi/post/lakeflow-a-new-era-of-agentic-data-engineering.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Bilal Aslam, Ray Zhu, Manish Dalwadi, Saad Ansari, Giselle Goicochea - Published: Jun 16, 2026 - Type: Announcement Databricks announces a major evolution of Lakeflow, its unified platform for data engineering across ingestion, transformation, and orchestration, with capabilities centrally governed by Unity Catalog. Genie Code and generally available Lakeflow Designer support agentic and no-code pipeline development, while Genie ZeroOps monitors production assets, analyzes failures, proposes fixes, and validates them in a governed sandbox before human approval. Lakeflow Connect expands to more than 100 managed connectors, and Zerobus Ingest adds Kafka-compatible, gRPC, REST, SDK, and OpenTelemetry interfaces for high-volume event ingestion. Real-Time Mode for Spark Declarative Pipelines reaches Public Preview with end-to-end latency as low as 5 milliseconds, alongside declarative APIs and expanded Lakeflow Jobs integrations. The release also adds data-readiness triggers and external orchestration for systems including Snowflake, REST APIs, Slack, and PagerDuty. ### [Introducing Genie ZeroOps: Put your data and AI operations on autopilot](https://yomu.fyi/post/introducing-genie-zeroops-put-your-data-and-ai-operations-on-autopilot.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Bilal Aslam, Lennart Kats, Ray Zhu, Mike Del Balso, Ori Zohar - Published: Jun 16, 2026 - Type: Announcement Genie ZeroOps is an autonomous background agent for monitoring and operating data and AI assets, including jobs, pipelines, tables, and ML models. It continuously detects visible and silent failures, uses Unity Catalog lineage and platform observability to assess root causes, generates remediation through development workflows, and verifies fixes in isolated sandboxes. These environments use shallow, zero-copy table clones, scoped permissions, and network isolation, so proposed changes run against real data without touching production or applying anything before approval. For ML workloads, the agent can diagnose degraded predictions, train a candidate on corrected features, evaluate it against the production model’s existing eval suite and criteria, and support live-traffic ramping when it is measurably better. Genie ZeroOps is entering private preview in the coming weeks, initially supporting jobs, pipelines, tables, and ML workloads; Apps and Lakebase databases are on the roadmap. ### [Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres](https://yomu.fyi/post/announcing-lakebase-search-agent-native-retrieval-built-into-lakebase.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Pranav Aurora, Zhou Sun, Jinjing Zhou - Published: Jun 16, 2026 - Type: Announcement Lakebase Search is a beta offering on AWS and Azure that adds hybrid vector and full-text retrieval to Lakebase Postgres. It uses the lakebase\_vector and lakebase\_text extensions to keep retrieval, memory, operational data, and hybrid search in one backend. lakebase\_vector retains pgvector types and operators, applies RaBitQ clustering and compression for 32x smaller indexes, and targets more than 1B vectors, while lakebase\_text replaces GIN with object-storage-optimized BM25 ranking. A tiered cache keeps hot data on NVMe and places colder data in object storage; the source reports lower memory needs, faster index builds, and cold-cache startup than standard pgvector HNSW in its LAION-100M benchmark. The extensions also combine vector similarity and keyword relevance with reciprocal rank fusion in a single SQL query, enabling joins and tenant filtering alongside transactional workflows. ### [Announcing the new Databricks Startup Program](https://yomu.fyi/post/announcing-the-new-databricks-startup-program.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Brad Van Vugt, Arjun Rajeswaran - Published: Jun 16, 2026 - Type: Announcement The Databricks Startup Program has been updated for venture-backed, early-stage startups building with data and AI. Qualifying companies can receive up to $200,000 in credits across Databricks and Neon, along with hands-on technical guidance, partner and community access, and connections to go-to-market teams and founder communities. The program is aimed especially at startups that recently raised institutional funding from pre-seed through Series A, and it is intended to provide an app backend, data, and AI stack from idea through product-market fit. Databricks and Neon are presented as providing a database and access to foundation models on day one, while Databricks supplies analytics, data warehousing, AI systems, and enterprise-grade governance as companies launch. Applications are available through the Databricks Startup Program. ### [Unifying Data and Governance in the Agentic Era: What’s New with Azure Databricks](https://yomu.fyi/post/unifying-data-and-governance-in-the-agentic-era-what-s-new-with-azure.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Isaac Gritz, Toussaint Webb, Ben Tripp, Kiriana Stukas - Published: Jun 16, 2026 - Type: Announcement At Data + AI Summit 2026, Azure Databricks announced capabilities aimed at moving enterprises from experimental AI pilots to production-grade automated workflows by unifying data, productivity tools, marketing, and governance on Azure. Its Agentic Data foundation introduces LTAP, combining analytical data, streaming pipelines, and live application transactions in one lakehouse storage copy; Lakebase adds a managed serverless Postgres engine with copy-on-write branching, while Lakehouse//RT targets millisecond responses for high-concurrency workloads. Genie integrations for Microsoft Teams, M365 Copilot, Excel, and SharePoint bring governed lakehouse intelligence and ingestion into daily work, alongside tools for agents, applications, pipelines, and autonomous operations. CustomerLake adds Profile Agents and Campaign Agents for customer profiles and personalization, while Genie Ontology and Unity AI Gateway provide semantic context, rate limits, content filtering, and spend controls. ### [Introducing Genie One, Genie Agents, and Genie Ontology](https://yomu.fyi/post/introducing-genie-one-genie-agents-and-genie-ontology.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Sydney Sundell, Ken Wong, Elise Georis - Published: Jun 16, 2026 - Type: Announcement Databricks announces Genie One, Genie Agents, and Genie Ontology to help enterprises answer business questions and act on data whose context is scattered across dashboards, queries, documents, tickets, and chats. Genie One connects data and business tools through Lakehouse federation, Lakeflow Connect, native integrations, Slack, Teams, mobile apps, schedules, alerts, document creation, custom skills, and MCP support. Genie Agents evolve Genie Spaces into domain-specific agents that can reason over structured and unstructured data, execute multi-step workflows, and be created from a prompt. Genie Ontology builds a permission-aware living graph from enterprise assets, weighting sources by authority, usage, certification, and freshness. In an internal 28-question benchmark, Genie answered 84.5% correctly on the first attempt and delivered twice the speed of the strongest coding agent. [Newer posts](https://yomu.fyi/posts/page/111.md) · [Older posts](https://yomu.fyi/posts/page/113.md)