Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Integrating the Rust Delta Kernel into ClickHouse
ClickHouse initially built native protocol support to read Delta Lake tables, but maintaining pace with the format's evolving specification created high complexity and fragmented features. To resolve this, engineers integrated the Rust Delta Kernel to offload protocol handling while retaining ClickHouse's optimized Parquet data reading. The Delta Kernel parses JSON transaction logs, resolves snapshots, manages transactional write metadata, and exposes schema evolution mapping alongside Change Data Feed streams. ClickHouse also contributed upstream enhancements to the Rust kernel, adding dynamic runtime logging and asynchronous metadata processing via FFI handles. While ClickHouse can attach to existing Delta tables, the integration cannot yet initialize empty Delta Lake tables.
How security teams can report cyber risk to boards
Boards are seeking visibility into cyber risk, but technical reports often fail to connect security posture with business impact or financial exposure. The post explains that compliance and cyber risk leaders can use Databricks Genie to query vulnerability posture, asset criticality, threat intelligence, control data, and historical incident costs in a governed environment. It recommends probabilistic financial modeling, including Monte Carlo simulation, to run randomized attack scenarios and produce loss distributions; Value-at-Risk framing can make those results familiar to directors. This approach replaces qualitative red/amber/green reporting with expected-loss ranges, supports investment prioritization, and enables trend analysis and board-ready answers, while the suggested cadence combines quarterly strategic briefings, monthly operational reviews, and incident-triggered updates.
The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink
Hugo evolved from a batch-oriented Spark platform into Grab’s unified, self-service data ingestion system for near real-time pipelines. The earlier expansion across Kafka Connect, Sprinkler, and Hugo fragmented onboarding, forcing users to translate configurations and manage manual schema changes across systems. A custom automation layer and Apache Flink now orchestrate one-click MySQL CDC and Kafka ingestion, retiring Kafka Connect and Sprinkler for the data lake. For CDC, Flink reads the binlog directly, uses automated schema detection and dynamic validation, and writes to queryable Hive tables through Spark compaction; for Kafka, it fetches Protobuf schemas from Confluent Schema Registry at runtime and resumes from the latest checkpoint after restart. Onboarding now takes approximately three minutes for MySQL CDC and six for Kafka, although schema updates still require manual restarts; new pipeline adoption in the last year also exceeded the previous five years combined.
Welcoming Manoel do Amaral, our new Brand Designer
Resend introduces Manoel do Amaral as its new Brand Designer, citing more than a decade of experience in product design, motion, and branding, including work for Google, Airbnb, and Accenture. He describes beginning at 13 with Photoshop and Illustrator, learning through YouTube tutorials, Abduzeedo posts, experimentation, and guidance from a designer friend before landing an agency job at 17. Manoel says Resend’s quality, simplicity, sidebar icon animations, and brand work drew him to the company after he used its email product at Savee for customers, prospects, and a weekly inspiration newsletter. Photography, visual curation, sports, and triathlon inform his interests; he names Granola and Strava as favorite tools and Yosemite as a place to recharge in his free time.
Solving the Identity Crisis for AI Agents
Uber describes its 2025 effort to give AI agents verifiable identities and preserve attribution as agents act for users across compositional, multi-step workflows. The problem is that conventional human and workload identities expose only generic service identities, dropping originating users and intermediate agents across hops and weakening audits, authorization, and incident response. The design extends Uber’s Zero Trust Architecture with an Agent Registry, an AI Agent Mesh, Security Token Service (STS), MCP Gateway, downstream systems, and an AI Gateway, while integrating identity and context propagation into the agent SDK. Workloads obtain cryptographically signed workload SVIDs from SPIRE, and STS issues short-lived, scoped JWTs for each hop; MCP Gateway authenticates and authorizes tool calls. Production metrics reported show P99 STS token-exchange latency consistently below 40 milliseconds, while the roadmap places dynamic access control and unified policy enforcement above the identity foundation.
Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use
User sequences serve as core primitives powering machine learning models across ranking, retrieval, and recommendation surfaces at Pinterest. Managing these sequences previously caused high infrastructure costs, operational fragility, and data drift between offline training pipelines and online serving systems. In response, engineering teams redesigned the platform around a single definition executed across multiple runtimes, including real-time streaming, batch backfills, and online inference. The updated architecture pairs a unified enrichment engine with columnar time-partitioned storage and standardized validation tiers. Consequently, the redesign achieved lower storage and network costs, faster signal onboarding via configuration, and improved engagement metrics across major recommendation surfaces.
Introducing Nova, our internal platform for coding agents
Dropbox developed Nova, an internal platform that runs AI coding agents in isolated cloud environments integrated with their Bazel monorepo. The platform supports both interactive developer workflows and autonomous background tasks, such as automated CI debugging, flaky test remediation, and codebase-wide migrations. By pairing code generation with automated validation and strict execution guardrails, Nova ensures generated fixes are tested and reproducible.
Transforming industries with conversational AI: Partner solutions built on Databricks Genie
Databricks presents a first cohort of partner-built industry solutions that use Genie to reduce the “Analyst Bottleneck,” where leaders wait for custom SQL or dashboard updates. Genie provides a conversational analytics layer grounded in Unity Catalog metadata and business semantics, using specialized AI agents to return governed, secure, traceable answers to natural-language questions. The featured accelerators span communications, media and entertainment, financial services, healthcare and life sciences, manufacturing and energy, public sector, and retail, with examples including churn prediction, next-best-action recommendations, billing anomaly investigation, KYC monitoring, actuarial analysis, and commercial insights. Across the examples, solutions combine domain expertise with Databricks platform capabilities such as machine learning, Agent Bricks, Lakebase, Serverless SQL, and auditable governance to support self-service analysis and operational decisions.
From emissions reporting to decarbonization decisions
Energy companies have built sophisticated infrastructure for Scope 1, 2, and 3 emissions reporting across complex asset portfolios, but that infrastructure is often oriented toward historical disclosure rather than forward-looking action. Sustainability leaders still need timely answers about which assets warrant intervention, what operational choices drive carbon intensity, and whether current performance is tracking toward commitments. Databricks Genie addresses this gap by letting users query emissions, operational, and financial data in natural language, linking results to dispatch decisions, fuel purchases, and asset utilization. It supports multi-scope analysis, traceability to source data, and scenario questions such as retiring an asset or increasing renewable PPA volume. The stated outcome is a shift from compliance reporting toward data-driven decarbonization decisions, with Genie available today for energy-sector use.
You’ve built the media products, now make them personalized
Media companies may have launched streaming services, digital editions, and mobile apps, yet still face a digital product intelligence gap: product teams must answer behavioral data questions quickly enough to personalize experiences and improve engagement. Databricks Genie gives nontechnical product leaders a conversational interface that translates natural-language questions into SQL queries, visualizations, and actionable insights over governed enterprise data, without requiring code or analyst handoffs. The agent queries governed Delta Lake tables fed by streaming clickstreams, views, and session signals, combining event-level behavior, A/B tests, audience segments, and cross-platform data in conversational answers. The source says internal benchmark accuracy improved from 32% to over 90% through multi-LLM orchestration, specialized knowledge search, and parallel reasoning, and presents Genie as reliable enough for production personalization decisions.
From "What Happened?" to "What Will Happen?"
Databricks Genie makes descriptive analytics accessible in natural language, but predictive questions still require specialized data science workflows and carefully prepared datasets. This post presents a multi-agent supervisor deployed as a Databricks App, combining Genie, TabPFN, and Agent Bricks to turn business questions into predictions. The orchestrator asks Genie to use governed Lakehouse data, schemas, relationships, and semantics to generate labeled training data through SQL, then sends it to TabPFN, which predicts in a single forward pass without feature preprocessing, model selection, or hyperparameter tuning. The resulting conversational experience supports descriptive and predictive analytics with Unity Catalog lineage and access control, while an MLflow GenAI evaluation harness monitors reliability and regressions. Its central limitation is that predictions depend on Genie producing a meaningful dataset with a clear label, so missing signals, joins, outcomes, or agent omissions can make results unreliable.
The product analytics you already have
Software engineers frequently rely on separate product analytics tools and teams to answer user behavior questions, creating disjointed data models and delayed insights. Existing debugging telemetry within tools like Sentry—specifically spans, structured logs, and application metrics—can serve as the primary source for product metrics. Developers enrich request spans with business-level attributes, record discrete business events through wide structured logs, and track KPIs with metric counters and distributions. This unified telemetry links business outcomes directly to underlying traces, errors, releases, and source code lines, enabling immediate root-cause analysis and automated alerts. While multi-session cohort and funnel aggregation remains a current tooling gap, standard telemetry solves day-to-day adoption, performance, and revenue questions.
Modal's Series C: Raising $355M at a $4.65B valuation
Modal announced a $355 million Series C at a $4.65 billion post-money valuation, led by General Catalyst and Redpoint, with participation from Menlo, Bain Capital Ventures, Accel, and all existing investors. The company grew fivefold since September and surpassed $300 million in annualized revenue. Modal is a cloud platform built for AI workloads, offering elastic inference, dynamic agent runtimes, and batch processing. Its Sandboxes feature, which provides isolated environments for untrusted code, has launched over 1 billion instances and accounts for more than a third of revenue. Modal plans to invest in low-latency inference, reinforcement learning infrastructure, and expanded agent computing capabilities.
Welcoming Michael Vaz, our new Customer Success Engineer
Resend introduces Michael Vaz as a new Customer Success Engineer based in Portugal, with experience in systems administration, networking, support engineering, cybersecurity, SaaS video, and email-related products. He began in technology by installing, configuring, and maintaining email servers, then developed broader experience at Sophos and Loom, including exposure to SaaS and startup scaling. At Resend, he says he wants to help unblock people, ensure systems work well for customers, and contribute to a product he finds easy to start with and capable of growing with users’ needs. His preferred tools include log platforms such as Datadog, which he describes as a source of truth when diagnosing failures, while his background also shapes a people-centered approach to support and problem-solving.
May 2026 newsletter
The May 2026 ClickHouse newsletter details enterprise observability migrations, ecosystem integrations, and core database enhancements. Digital banking platform Qonto migrated from Grafana Tempo to ClickHouse Cloud, compressing 231 TB of raw trace metadata to 376 GB while expanding query windows to two weeks and introducing an AI incident companion. Similarly, LINE MAN Wongnai consolidated multiple business clusters onto self-hosted ClickHouse, achieving 1.5 million rows per second peak ingest, 100% trace retention, and a 53% reduction in observability costs. Comparative log analytics benchmarks demonstrated that ClickHouse utilized 5x less disk space and achieved 4 to 6x faster cold query execution than Elasticsearch across 50 billion rows. Additional updates include the ClickHouse 26.4 release featuring improved SQL compatibility, new agent telemetry adapters in Mastra, and SQL-based alerting capabilities in ClickStack.
How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%
Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.
Safety guidelines for autonomous mobility and delivery on the Uber Platform
Uber has updated its Safety Guidelines for Autonomous Mobility & Delivery Providers after three years of real-world autonomous vehicle deployment and onboarding new autonomous modalities. The expanded framework addresses new AI standards, advances in industry safety standards, food safety for sidewalk robots, and delivery drone safety. It works alongside regulations and industry best practices, while Uber uses a partner-specific Safety Plan assessment rather than one universal metric or threshold. The AM&D Safety Program covers planning, demonstration, and operation, with reporting on safety performance and obligations for partners at each stage. Assessment considers each partner’s planned scope and comprehensive safety approach, including testing and validation, as the platform seeks consistent, transparent, and accountable deployment across mobility and delivery use cases.
clickhousectl v0.2.0: Postgres, ClickPipes and more
clickhousectl v0.2.0 expands the beta CLI with local and ClickHouse Cloud managed Postgres, ClickPipes, SQL over HTTP for Cloud services, agent-oriented output changes, and a standalone Rust client for the ClickHouse Cloud API. Local Postgres runs in Docker, is keyed by name and major version, stores isolated data under .clickhouse/servers/-pg/data/, and supports versions 16, 17, and 18. ClickPipes covers batch, streaming, CDC, and snapshot sources including object storage, Kafka-compatible systems, Kinesis, Postgres, MySQL, MongoDB, and BigQuery. Cloud queries use a provisioned or lazily enabled Query Endpoint and service-scoped API key, while the endpoint uses sql_console_admin and the key has no organization-level roles. The release also adds Markdown-style tables and credential debugging, with more than 5,000 users reported five weeks into the project.
Unlock seamless and cost-effective marketing campaigns with Lakebase
Retail marketing teams often store customer segments in OLTP databases, creating underused capacity between campaigns and synchronization work whenever new segments are requested. Lakebase, Databricks’ implementation of a lakebase architecture, separates storage from compute and uses elastic, serverless Postgres that can scale to zero when idle and up for bursty campaign traffic. The integration with the Lakehouse uses managed Synced Tables for customer segments and Lakehouse Sync for continuous CDC-based replication of operational data into Unity Catalog Delta tables. The SAP Engagement Cloud example configures a Lakebase Autoscaling project, native Postgres credentials, an ISRG Root X1 certificate, and snapshot synchronization for datasets with more than 10% updated. The post concludes that this design lowers idle and sizing costs, reduces pipeline maintenance, and supports low-latency point lookups, while warning that Lakebase is not optimized for large scans or classic OLAP.
Governing AI agents at scale with Unity Catalog
With AI agents proliferating across organizational functions, governance must answer which agents access sensitive data and how they act. Traditional approaches either scatter logs and permissions across systems or restrict deployment, while agents require controls over autonomous tool use and runtime behavior. Unity Catalog and Unity AI Gateway address this through four pillars: delegated access, data-centric governance, cost intelligence, and open interoperability. The design passes user identity through agents with on-behalf-of tokens, evaluates MCP tool calls with Service Policies, applies inline guardrails, and records model, access, trace, and usage data in lakehouse tables. It also connects data quality, classification, spend, and outcomes, while supporting multiple frameworks and model providers through shared governance.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.