Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Building an Enterprise Data Management Strategy
Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.
Index sharding in ClickHouse Cloud: Petabyte-scale data needs petabyte-scale indexing
In large-scale ClickHouse Cloud deployments, replicas traditionally load entire primary and secondary indexes from object storage into working memory, creating massive redundant memory overhead as fleets expand. Index sharding resolves this bottleneck by distributing the index analysis phase across available replicas using consistent hashing over data parts. Instead of every node loading full index structures, each replica loads and analyzes only its assigned fraction of primary keys, bloom filters, text indexes, or vector indexes. The query initiator merges the resulting granule match ranges, allowing parallel replicas to stream data without any single machine retaining the whole index. This approach freed working memory for query execution and accelerated index analysis by up to 7.7x in benchmark tests on a 50 billion row table.
Real-Time Decisioning for AI Agents: Why you Need a Customer Context Layer First
Real-time decisioning for AI agents depends on more than customer records: it requires a customer context layer that captures what people are doing now and connects that behavior to identity. The post positions the data platform as the center of a composable martech architecture, with agents and applications operating on shared data in open formats such as Apache Iceberg and Delta Lake. Snowplow’s described approach places structured event collection, schema validation, enrichment, and identity resolution before data reaches the platform, while operating in the customer’s cloud environment. It distinguishes historical profiles from granular behavioral event streams and argues that agent interaction outcomes should return to the foundation as first-class events. The resulting four-stage loop—collect, resolve and enrich, serve, and learn—supports simultaneous real-time and historical context, with decision quality dependent on source data coherence.
A Practical Guide to LLM Fine Tuning
The guide presents LLM fine tuning as a way to adapt a pre-trained model to specific tasks, domains, or applications without full pretraining. It explains when prompting or few-shot learning is sufficient and when fine tuning is justified by quality gaps, domain terminology, latency, cost, or behavioral control. The recommended lifecycle covers scoping, data collection and cleaning, base-model and method selection, training, evaluation, deployment, and monitoring, with production-format consistency and held-out splits treated as important safeguards. It compares supervised and full fine tuning with parameter-efficient fine tuning (PEFT), including LoRA and QLoRA, which update fewer parameters and reduce compute and memory requirements. The main risks are noisy data, overfitting, and catastrophic forgetting; the proposed default is a pilot that compares a PEFT-adapted model with its base model, while combining fine tuning with RAG and prompt engineering when appropriate.
Keeping the Cursor app stable
Cursor's desktop app faces disruptive crashes as usage and features such as subagents, instant grep, and browser use have expanded, with most failures caused by out-of-memory conditions. The team measures instability across Electron's multi-process architecture using telemetry, per-session and per-request crash rates, minidumps, stack traces, crash breadcrumbs, and dashboards that update within minutes. Its two-pronged debugging strategy combines feature-level experiments and oversize-message instrumentation with bottom-up analysis based on a crash watcher, Chrome DevTools Protocol (CDP) stacks, automated daily investigations, opt-in heap snapshots, and low-rate allocation profiling. Findings led to chunking large data, killswitches, lifecycle fixes for leaks, and process isolation for extensions, while prevention uses Bugbot rules, stress tests, garbage collection, performance tests, and automated rollbacks. OOM-per-session fell 80% from its late-February peak and OOM-per-request fell 73% since March 1.
AI Data Transformation Guide for Data Engineers and Data Scientists
AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.
Concurrency Control in DBMS: How Locking, MVCC and Optimistic Strategies Keep Data Consistent
The guide explains how database management systems coordinate simultaneous transactions to preserve consistency and serializability, preventing dirty reads, lost updates, non-repeatable reads and phantom reads. It compares lock-based control, including shared and exclusive locks and two-phase locking, with MVCC, which gives transactions consistent snapshots and lets readers and writers proceed without blocking. The discussion covers pessimistic and optimistic strategies, isolation levels, deadlocks, retries, lock-granularity trade-offs and MVCC’s storage and garbage-collection costs, including PostgreSQL’s VACUUM. For lakehouse workloads, Delta Lake uses a read-validate-commit process with optimistic concurrency, snapshot isolation and conflict detection for concurrent writes. The source presents Databricks Lakebase as a managed implementation offering optimistic concurrency control, snapshot isolation for reads and write-serializable isolation for writes without lock or deadlock management.
QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard
QIMMA is an Arabic LLM leaderboard built to address fragmented evaluation, translation artifacts, weak benchmark quality checks, limited coverage, and reproducibility gaps. It treats benchmark validation as a prerequisite for reported scores rather than an afterthought. The suite consolidates 109 subsets from 14 source benchmarks into more than 52,000 samples across seven domains, with 99% native Arabic content, Arabic-adapted HumanEval+ and MBPP+ coding tasks, and six prompt templates. Before model evaluation, Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B assess samples using a 10-point rubric, flagged cases receive native-speaker review, and code statements are refined without changing task identifiers, reference solutions, or tests. In the April 2026 rankings, Qwen/Qwen3.5-397B-A17B-FP8 ranks first, while Arabic-specialized models lead cultural and linguistic tasks and multilingual models lead coding.
LLM apps: Real-world use cases, core building blocks and risks to know
LLM applications are presented as production systems that extend beyond chatbots by connecting foundation models to external data, tools, APIs, memory and workflows. Modern categories include assistants and copilots, retrieval-augmented generation (RAG), autonomous agents, multi-agent systems and local or on-device inference, each with distinct capabilities and trade-offs. A typical RAG pipeline uses vector search over a curated knowledge base, then supplies retrieved passages to the model; chunking, source freshness and retrieval accuracy affect results. Production systems also require orchestration, evaluation, observability, governance, guardrails, audit trails and human-in-the-loop checkpoints, especially when agents can call tools or execute workflows. The source distinguishes prompt engineering, which guides behavior without changing model weights, from fine-tuning, which adapts models to domain-specific tasks, and describes a unified path combining data, models, retrieval, orchestration, evaluation and governance for production deployment.
AI and the Future of Cybersecurity: Why Openness Matters
Mythos is presented as a frontier AI system that rapidly finds and patches software vulnerabilities, with the post arguing that its surrounding system matters more than the language model alone. Its recipe combines substantial compute, software-relevant training data, vulnerability-probing and patching scaffolding, speed, and some autonomy; together, these elements can uncover vulnerabilities, find exploits, and build patches. The post argues that open code and tooling can distribute detection, verification, coordination, and patch propagation across communities, while reducing capability asymmetry between attackers and defenders. For defensive use, it recommends semi-autonomous agents with prespecified actions, human approval, auditable logs, and private deployment using open components and existing security tools. This approach is especially relevant to high-stakes organizations that need inspectable, controllable systems and protection for sensitive data.
ClickHouse expands strategic collaboration with Google Cloud, bringing deeper integration, cloud flexibility, and next-generation performance
ClickHouse announced an expanded strategic collaboration with Google Cloud introducing four core architectural milestones. First, native integration with Google Cloud Lakehouse allows direct sub-second queries across data lakes and ClickHouse deployments without ETL pipelines or data duplication. Second, the Bring Your Own Cloud deployment model lets organizations run managed ClickHouse Cloud inside their own Google Cloud Virtual Private Cloud to retain data sovereignty and network controls. Third, ClickHouse Cloud is migrating automatically to Google's custom Arm-based Axion processors, boosting query throughput and cost efficiency for analytical workloads without requiring application changes. Finally, a new ClickHouse MCP server integration connects ClickHouse with Google Antigravity to enhance developer and analyst workflows.
From Static Policies to Autonomous Insurance: How AI enables Real-Time Coverage
Insurance products often rely on annual pricing, periodic risk assessment, and coverage changes tied to major life events, even though driving behavior, digital fraud exposure, and travel risk can change in seconds. The post presents embedded insurance and usage-based insurance as a shift toward contextual, just-in-time coverage delivered through apps, checkout flows, vehicle purchases, and partner platforms. Its automotive example combines telematics, geospatial data, and real-time weather signals to update risk scores and premiums from trip behavior such as braking, following distance, mileage, location, and environmental conditions. A Databricks Lakehouse architecture registers devices, ingests telemetry through Zerobus, curates data in a governed Medallion architecture, and exposes features through Lakebase and Databricks One, with Unity Catalog supporting access controls, lineage, explainability, and audit trails. The proposed trajectory leads from embedded coverage toward autonomous insurance whose policies continuously adapt to risk.
Bridging data science and marketing: Databricks unveils Delta Sharing integration for Adobe Experience Platform and agentic marketing workflows
Databricks and Adobe announce three partnership milestones aimed at connecting governed Lakehouse data with marketing activation in Adobe Experience Platform. The changes include Delta Sharing support for zero-copy access, a Databricks Genie connection to Adobe Marketing Agent through Model Context Protocol (MCP), and an upcoming Adobe Marketing Agent beta for the Databricks Marketplace. Using Unity Catalog as the source, Experience Platform can access shared tables and views as virtual tables, while Adobe Data Distiller queries live Databricks data without copying underlying records. Through MCP, agents in both platforms can use governed datasets, metadata, models, campaign metrics, SQL Warehouses, and Model Serving endpoints with natural-language interactions. The stated outcome is a shared, real-time context for audience creation, campaign analysis, closed-loop attribution, and model improvement, while reducing data movement, duplication, latency, egress, and governance overhead.
Take Control: Customer-Managed Keys for Lakebase Postgres
Lakebase Customer Managed Keys (CMK) gives regulated enterprises control of the root of trust through customer-owned keys from AWS KMS, Azure Key Vault, or Google Cloud KMS. Because Lakebase separates persistent storage from elastic Postgres compute that can scale to zero, CMK protects both layers and their caches rather than storage alone. Its hierarchical envelope-encryption model uses the CMK as the root, transient Key Encryption Keys (KEKs) to wrap Data Encryption Keys (DEKs), and unique DEKs for each data segment, while plaintext CMKs remain in the customer’s KMS. Account Admins bind a configured key to a workspace, after which new projects inherit the CMK; rotation needs no re-encryption or downtime, revocation terminates compute instances, and audit services record cryptographic operations.
Get hands-on with agents, vibe coding and more at Data+ AI Summit
Data + AI Summit returns to San Francisco from June 14 through June 18, 2026, with Databricks Training and Certification offering more than 20 hands-on courses across AI agents, vibe coding, data engineering, and Lakebase. Training-exclusive days on Sunday and Monday are designed to provide dedicated learning time, while sessions continue during the conference week. New courses cover agentic development with Genie Code, Claude Code, Cursor, MCP servers, and structured prompting; production agent deployment with MLflow’s ResponsesAgent framework and tracing; Lakeflow Spark Declarative Pipelines; and AI/BI dashboards and Genie Spaces. Onsite certification exams cost $100 plus taxes, a 50% discount, and seven listed exams are available. The Learning Hub adds labs, skills assessments, customized learning plans, and community access.
Mercedes-Benz builds a cross-cloud data mesh with Delta Sharing and intelligent replication, cutting costs by 66%
Mercedes-Benz built a cross-cloud data mesh to share after-sales data between AWS and Azure while supporting its data-defined vehicle strategy. Existing direct queries incurred egress costs, and weekly full loads delayed updates for datasets that included vehicle over-the-air events and workshop visits; source tables were in Iceberg while many consumers expected Delta. Using Unity Catalog and Delta Sharing, the company federated AWS Glue tables, established cross-metastore shares, and used periodic serverless Sync Jobs with Delta Deep Clone to incrementally replicate selected tables into Azure object storage for local querying. This hybrid model preserves direct sharing for fresher workloads while reducing transfer for less time-sensitive ones. For the initial ten data products, egress costs fell 66%, update frequency improved from weekly to every second day, and the source reports an approximately 93% annual reduction in its 50-use-case comparison.
What Is a Transactional Database?
Transactional databases handle large volumes of short, real-time read and write operations for systems such as orders, payments, account balances, and customer records. They support online transaction processing (OLTP) by treating related steps as one logical unit: successful work commits, while failures roll back the full transaction. ACID—atomicity, consistency, isolation, and durability—protects integrity under concurrency and failures, with row-oriented storage optimized for individual-record access, locking and isolation levels coordinating concurrent work, and write-ahead logging (WAL) supporting crash recovery. The source positions these databases as foundations for banking, e-commerce, healthcare, reservations, and telecommunications, while noting that row-oriented execution is less suitable for large analytical workloads and that horizontal scaling is more complex than in distributed NoSQL systems.
Launch Week 6: Wrap Up
Launch Week 6 concluded with five feature releases and one announcement covering email automation, content creation, developer tooling, tracking, and community events. Automations orchestrates lifecycle emails from application events with delays, waits, conditions, test runs before publishing, and real-time visibility into runs. The AI-extended editor turns URLs into on-brand drafts, supports inline block editing and mistake detection, and connects to agentic stacks through the Resend MCP server, while CLI 2.0 adds React-file sending, automation and event management, webhook listening, and AI-agent skills across more than 50 API commands. Custom tracking domains support click and open tracking, React Email 6 adds an embeddable editor and extensions API, and Resend Forward was announced as a San Francisco conference scheduled for October 21.
Terraform & OpenAPI for ClickPipes is now Generally Available
ClickHouse Cloud announced the general availability of ClickPipes resources in Terraform and OpenAPI, expanding infrastructure-as-code support across its managed data ingestion platform. The release resolves previous beta gaps by adding full connector coverage, including Change Data Capture for Postgres, MySQL, and MongoDB, alongside support for BigQuery, Azure Blob Storage, and reverse private endpoints. Operators can provision new data ingestion pipelines, pause or resume sync operations, and scale resources programmatically through OpenAPI endpoints or the ClickHouse Terraform provider starting at version 3.14.0. Existing ClickPipes can also be imported into Terraform state using native import blocks and automated configuration generation. Although SSH tunneling remains under active development, nearly all UI capabilities now map cleanly to programmatic interfaces.
Resend Forward: A Conference about Craft
Resend Forward is a newly announced conference for developers who treat software as a craft. It will take place in San Francisco on October 21st and run for a full day focused on the craft behind building software. The program will feature leaders working at the edge of software and AI, with special emphasis on practical tools and ideas that attendees can use. Food, conversations, and swag are planned between sessions, while the stated goal is to connect people who care about quality, share what’s working, and leave them more excited to build. More details are expected in the coming months; tickets will go on sale after the waitlist notification.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.