---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Data Pipeline Best Practices: Architecture, Modern Pipelines, and Deployment](https://yomu.fyi/post/data-pipeline-best-practices-architecture-modern-pipelines-and-deploym.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 18, 2026
- Type: Tutorial

Modern data pipelines move raw data from source systems through transformation and storage to downstream consumers, with batch, streaming, or hybrid operation selected according to latency needs. The guidance places service-level agreements first, defining acceptable latency, uptime, and error rates before architecture choices, and compares ETL, ELT, and zero-ETL patterns. It recommends immutable raw landing zones, CDC for transactional sources, schema registries, event replay, incremental loads, and idempotent writes using techniques such as MERGE operations. Performance and reliability practices include partitioning, clustering, data-quality checks, observability, automated CI/CD testing, and cost controls such as autoscaling and storage tiering. Adoption guidance advises piloting a representative pipeline, measuring telemetry and cost, then iterating through regular architecture reviews.


### [What's new in Postgres Managed by ClickHouse: RBAC, Terraform, ClickPipes, extensions, and more](https://yomu.fyi/post/what-s-new-in-postgres-managed-by-clickhouse-rbac-terraform-clickpipes.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 18, 2026
- Type: Announcement

Postgres managed by ClickHouse, which entered public beta in May, received a broad update covering console operations, per-service RBAC, Terraform, Stripe provisioning, ClickPipes migrations, billing, capacity reservations, and extensions. The redesigned console adds service health, resource, and query views, while RBAC provides View and Manage roles across the console and public REST API; the alpha Terraform provider covers lifecycle operations including resizing, HA, replicas, restore, and password rotation. Postgres-to-Postgres ClickPipes now automate schema migration, usage is integrated into ClickHouse Cloud’s metering and billing, and AWS capacity reservations are allocated dynamically across availability zones. Underneath, pg\_clickhouse and pg\_stat\_ch moved to the header-only clickhouse-c library, removing a C++ exception conflict, using Postgres memory contexts, and reducing the x86-64 Linux extension binary from 4.9 MB to 1.4 MB; the update shipped less than a month after beta, with Terraform and extensions still evolving.


### [How Stagwell built privacy-safe ID matching on Databricks](https://yomu.fyi/post/how-stagwell-built-privacy-safe-id-matching-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sridhar Sundaresan, Suvan Kaul
- Published: Jun 18, 2026
- Type: Explainer

Stagwell built a Databricks Marketplace App to let brands match first-party records against its Identity Spine without sending raw customer data outside their own workspaces. It combines Databricks Clean Rooms, Unity Catalog, Jobs and Notebooks, Delta Sharing, and a React/Express interface. Workspace OAuth and On-Behalf-Of authorization enforce the user's Unity Catalog permissions, while separate service principals and a scoped personal access token support app and clean-room operations. Within the clean room, a packaged notebook joins brand data with the Identity Spine, resolves multiple identifiers, and computes match rates, coverage metrics, and household and consumer IDs; results include aggregated audience insights for activation, while raw customer data and proprietary matching code remain separated.


### [Building a SQL ETL Pipeline: The Complete Guide for Data Engineers](https://yomu.fyi/post/building-a-sql-etl-pipeline-the-complete-guide-for-data-engineers.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 18, 2026
- Type: Tutorial

SQL ETL pipelines are presented as repeatable workflows that extract data from sources, transform it, and load it into warehouses, lakes, or lakehouses for analysis and machine-learning use. The guide addresses source connectivity, extraction patterns, transformation logic, loading targets, governance, performance, testing, and operational design, while contrasting ETL with ELT and broader data pipelines. It explains that SQL can serve as the primary implementation language for transformations and load operations, with techniques including JOIN and GROUP BY, window functions, MERGE upserts, and deduplication with ROW\_NUMBER() or DISTINCT. It also covers full versus incremental extraction, batch and streaming needs, schema-on-write versus schema-on-read, and layered validation using row counts, checksums, business rules, and schema-drift monitoring.


### [What is Artificial Intelligence (AI)?](https://yomu.fyi/post/what-is-artificial-intelligence-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 18, 2026
- Type: Explainer

Artificial intelligence (AI) is a branch of computer science that enables machines to perform tasks associated with human intelligence, including learning, reasoning, pattern recognition and decision-making. Modern AI generally learns patterns from large datasets, tunes internal weights and parameters during training, evaluates outputs on held-out data, and applies the resulting model during inference to classify, predict, generate content or trigger actions. Most organizations fine-tune existing foundation models rather than train from scratch, while output quality remains dependent on the completeness, bias and quality of training data. The page separates reactive machines and limited memory from theoretical theory of mind and self-aware systems, and distinguishes today’s narrow AI from theoretical general AI and superintelligence. It also describes generative AI, common applications, risks including hallucinations, bias, privacy and security gaps, and governance, concluding that practical adoption depends on real problems, trusted data and responsible oversight.


### [Data Engineering for AI: A Practical Guide for Data Professionals](https://yomu.fyi/post/data-engineering-for-ai-a-practical-guide-for-data-professionals.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 18, 2026
- Type: Explainer

Data engineering for AI extends traditional ETL by adding requirements for model-ready data, unstructured inputs, feature engineering, governance, and production monitoring. The guide addresses data professionals building or scaling AI infrastructure, covering ingestion, architecture, data quality, privacy compliance, generative AI, and career development. It emphasizes shared feature schemas, pipeline data contracts, lineage, statistical validation, drift detection, and PII stripping or anonymization as responsibilities shared across data engineering and data science. For generative AI systems, it describes preparing RAG pipelines by ingesting and chunking documents, creating vector embeddings, and indexing them for semantic retrieval, while evaluating vector databases for latency, scale, and integration. It concludes that reliable AI depends on fresh, accurate, compliant data and ongoing operational and architectural review.


### [Data Warehouse Types: A Complete Guide to Architectures and Use Cases](https://yomu.fyi/post/data-warehouse-types-a-complete-guide-to-architectures-and-use-cases.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 18, 2026
- Type: Explainer

A data warehouse is a centralized repository for structured data, supporting complex queries, reporting, and business intelligence rather than transaction processing. The guide compares architectures by scale, latency, cost, scope, ownership, and governance. Enterprise data warehouses integrate organization-wide sources through ETL, apply cleansing and validation, and provide a governed source of truth, while data marts focus on departmental analysis and may be dependent or independent. Operational Data Stores replicate current or recent operational data for reporting refreshed from minutes to hours, whereas virtual, cloud, hybrid, and lakehouse designs trade physical consolidation, scalability, flexibility, and governance differently. The comparison also frames lakehouses as combining open-format data lake flexibility with warehouse-style governance and transactional reliability.


### [The new bottleneck](https://yomu.fyi/post/the-new-bottleneck.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Eira May
- Published: Jun 18, 2026
- Type: Problem & solution

AI coding tools have significantly lowered the cost of generating software, yet many engineering organizations fail to realize overall delivery speed improvements. Applying the Theory of Constraints reveals that eliminating code production as a bottleneck shifts inventory directly into surrounding legacy processes that remain unadjusted. New friction points consistently emerge across underspecified requirements, prolonged design handoff gates, senior engineer review capacity, and external sign-offs from legal or security. Organizations can address these blockers by interrogating legacy agile ceremonies checkpoint by checkpoint to determine if their original constraints still exist. Practical remedies include adopting real-time co-development between product and engineering, treating initial designs as fluid starting points, and restructuring processes around running rapid, high-volume experiments.


### [What is Row-Level Security?](https://yomu.fyi/post/what-is-row-level-security.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 18, 2026
- Type: Explainer

Row-level security (RLS) is a database access control that filters which table rows a user may read or change according to identity, role, or session context. It supports multi-tenant SaaS, regional segregation, departmental access, compliance, healthcare, and partner use cases, while typically layering table-level grants with column-level security or data masking. The database applies a policy or predicate automatically at query time across BI dashboards, notebooks, ad-hoc SQL, APIs, and third-party tools, with separate USING and WITH CHECK rules for reads and writes. On Databricks, Unity Catalog row filters use SQL user-defined functions and apply across Databricks SQL, notebooks, jobs, and connected BI tools. The source recommends simple predicates, indexed keying columns, identity-based testing, and explicit enforcement for both reads and writes.


### [How A2A is Building a World of Collaborative Agents](https://yomu.fyi/post/how-a2a-is-building-a-world-of-collaborative-agents.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Alan Blount, Frank Guan, Nick Losier
- Published: Jun 18, 2026
- Type: Announcement

A2A, launched one year before this June 18, 2026 update, is presented as a common language for conversational, dynamic agents that need to collaborate and hand off tasks securely rather than behave like stateless tools. The protocol uses black-box handoffs to keep sensitive data and proprietary processes encapsulated, prevent context pollution, support clarifying exchanges, and distribute specialized workloads across teams or services. Foldrun demonstrates this model by exposing protein-structure prediction as a standalone agentic interface that manages long-running tasks, adjusts parameters based on prediction confidence, and selects among AlphaFold 2, OpenFold 3, and Boltz-2 depending on the molecule. In an A2A-compliant environment, users can register Foldrun and delegate work in natural language without custom glue code, while a primary agent handles other research tasks. The post also cites applications in commerce, streaming, IT operations, and regulated telecom, while Python and Go SDKs are 1.0 GA.


### [Beyond LoRA: Can you beat the most popular fine-tuning technique?](https://yomu.fyi/post/beyond-lora-can-you-beat-the-most-popular-fine-tuning-technique.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Benjamin Bossan, Sayak Paul, Marian Tietz, Kashif Rasul
- Published: Jun 18, 2026
- Type: Explainer

Parameter-efficient fine-tuning (PEFT) reduces the memory required to adapt open models, enabling fine-tuning with a fraction of the memory and, in some cases, quantized models. LoRA is dominant because it freezes base weights and trains a small set of added parameters, but its popularity may also reflect ecosystem momentum and uneven comparisons. The post presents shared-condition benchmarks for language-model math reasoning and image generation, using identical models, datasets, code, hardware, and measures including test performance, VRAM, forgetting or drift, runtime, and checkpoint size. The results show that LoRA performs well but is not consistently best: other techniques can outperform it on one or more axes, especially in the image-generation benchmark. The recommended choice is therefore to compare techniques against a use case’s needs and constraints, including quantization support, layer coverage, adapter merging, and downstream serving compatibility.


### [Is it agentic enough? Benchmarking open models on your own tooling](https://yomu.fyi/post/is-it-agentic-enough-benchmarking-open-models-on-your-own-tooling.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Lysandre, Nathan Habib, Pedro Cuenca
- Published: Jun 18, 2026
- Type: Benchmark

Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison.


### [June 2026 newsletter](https://yomu.fyi/post/june-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 18, 2026
- Type: Announcement

ClickHouse’s June 2026 newsletter marks the project’s ten years as open source, noting more than 2,000 contributors and community projects alongside engineering updates. It profiles Vasily Chekalkin’s 26.5 improvements to WebAssembly user-defined functions, including the DETERMINISTIC keyword and visibility in system.functions, and covers native Jaeger storage, the ClickCannon benchmark tool, and Open House announcements. The 26.5 release reports Top-N optimizations that make ORDER BY … LIMIT through JOIN 20.4× faster with 175× less memory on TPC-H SF100, while GROUP BY … LIMIT is 11.9× faster with 185× less peak memory. Other featured systems include a 15-million-rows-per-hour KPI pipeline, join-engineering work yielding a 26× improvement on join-heavy TPC-H workloads, and community guidance on inserts, columnar storage, ArrowStream, and RAG ingestion.


### [Payment Fraud Detection: How Banks and Businesses Stop Fraudulent Transactions](https://yomu.fyi/post/payment-fraud-detection-how-banks-and-businesses-stop-fraudulent-trans.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 17, 2026
- Type: Explainer

Payment fraud detection is presented as a real-time practice for identifying and blocking unauthorized transactions amid evolving card-not-present, account takeover, authorised push payment, friendly, and card-testing attacks. Modern systems combine predefined rules, machine learning, behavioral analytics, device fingerprints, geolocation, transaction history, and velocity signals to calculate risk scores within milliseconds, declining or routing high-risk payments to review. Rules are fast and interpretable but require continual tuning, while machine-learning and behavioral models can identify complex or user-specific patterns that static thresholds miss. The source also emphasizes prevention controls such as MFA, tokenization, 3D Secure, and device binding, alongside unified real-time data infrastructure that supports cross-channel signals, continuous retraining, and deployment into payment authorization flows.


### [What is an AI Agent Harness?](https://yomu.fyi/post/what-is-an-ai-agent-harness.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 17, 2026
- Type: Explainer

An AI agent harness is the software infrastructure around a large language model (LLM) that lets it act on tasks rather than only answer prompts. The model reasons and chooses actions, while the harness connects tools, execution environments, memory, external systems and safety controls. Many agents follow a reason–act–observe loop: the model proposes an action, the harness executes it, captures the result and returns that result as context for the next decision. Production harnesses typically include system prompts, tool execution, sandboxes, filesystems and durable storage, context management, feedback and self-verification, guardrails, human-in-the-loop approvals, and observability; their design increasingly shapes reliability as model capabilities converge.


### [GCP Pub/Sub connector for ClickPipes is now in Private Preview](https://yomu.fyi/post/gcp-pub-sub-connector-for-clickpipes-is-now-in-private-preview.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 17, 2026
- Type: Announcement

ClickHouse has announced a GCP Pub/Sub connector for ClickPipes, now in Private Preview, to stream GCP infrastructure logs, application events, and operational signals into ClickHouse Cloud in real time. Previously, routing data from Pub/Sub into ClickHouse required GCS or a Dataflow pipeline; the connector instead manages subscription lifecycle, schema inference, and message delivery after a topic and target table are configured. It supports JSON, Avro, and Protobuf, Pub/Sub Schema Registry integration, automatic decompression, attribute-based filtering, and start positions of latest, earliest, or a specific timestamp. ClickPipes supports horizontal and vertical scaling, configurable replicas, automatic retries, built-in metrics and logs, and programmatic management through OpenAPI and the ClickHouse Terraform Provider. The connector is available upon request and free during Private Preview, with access offered through a waitlist or a ClickHouse Cloud account manager.


### [Databricks and NVIDIA: Building for the Agentic Era](https://yomu.fyi/post/databricks-and-nvidia-building-for-the-agentic-era.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Hanlin Tang, Tejas Sundaresan
- Published: Jun 17, 2026
- Type: Announcement

Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.


### [Data Warehouse Modernization: Roadmap, Architecture, and Services](https://yomu.fyi/post/data-warehouse-modernization-roadmap-architecture-and-services.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 17, 2026
- Type: Explainer

Data warehouse modernization addresses legacy systems that cannot scale efficiently with growing data volumes, real-time analytics, machine learning, and self-service access. The proposed roadmap spans two to four years for large estates, moving from assessment and architecture design through high-impact workload migration, governance embedding, and optimization rather than relying on a risky big-bang cutover. Its target architecture favors a lakehouse or enhanced cloud data warehouse, with open formats such as Apache Iceberg or Delta Lake, separate compute and storage, and Bronze, Silver, and Gold layers supporting incremental ELT and lineage. The source says modernization can reduce infrastructure maintenance costs by 30–50%, compress query latency from hours to seconds, and halve redundant ETL pipelines, while also improving governance for sensitive data and enabling BI, machine learning, and generative AI workloads on a shared foundation.


### [You're Spending Too Much on AI. You're Also Using Too Little.](https://yomu.fyi/post/you-re-spending-too-much-on-ai-you-re-also-using-too-little.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Anand Kuchibotla, Kedar Thakkar, Rahul Sengottuvelu
- Published: Jun 17, 2026
- Type: Problem & solution

The post argues that a large AI bill does not show excessive use: companies can overspend on routine work while using too little AI where advanced models could create value. It proposes measuring work in atomic tasks—such as invoices coded or pull requests reviewed—rather than tokens, with cost defined by tasks attempted and value by successful tasks. The operating model uses defaults that pair routine work with the cheapest model meeting quality benchmarks, medium reasoning, and flexible latency, while escalating ambiguous, high-stakes work to frontier models at higher effort. It also recommends attributing spend by provider, product, team, and workflow, finding concentrated costs, benchmarking repeated tasks, repricing after model releases, and centralizing controls in one gateway. The conclusion is that efficiency should be treated as an engineering achievement, so cheaper routine execution funds ambition on rare tasks where extra intelligence can materially change outcomes.


### [The open ecosystem around ClickHouse](https://yomu.fyi/post/the-open-ecosystem-around-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 17, 2026
- Type: Explainer

ClickHouse’s open ecosystem spans clients, dashboards, ingestion pipelines, projects built on its engine, and agents that query it, with 2,600+ contributors, 240,000+ commits, and hundreds of millions of downloads cited. It describes open standards including OpenTelemetry for observability, Model Context Protocol (MCP) for AI tooling, and Iceberg and Delta Lake for lakehouse data, plus Iceberg REST catalogs, Apache Polaris, and Unity Catalog. The surrounding stack includes ClickStack, chDB, Langfuse, LibreChat, official language clients, a Kubernetes operator, Postgres bridges such as PeerDB and pg\_clickhouse, and integrations with Airflow, dbt, Grafana, Fivetran, and Sigma. Specific updates include full Iceberg v2 support, Iceberg writes in 25.8, Paimon in 25.10, chDB v4’s DataStore API, and a Python client v1.0 with native async and 4x faster cold starts. The post frames this openness as supporting flexible tooling and community-led growth, reporting more than 200 integrations while noting that the ecosystem’s next direction remains unknown.


[Newer posts](https://yomu.fyi/posts/page/109.md) · [Older posts](https://yomu.fyi/posts/page/111.md)