Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Flexible Authentication: Reimagining authentication for millions of users at Airbnb
Airbnb overhauled its login and registration architecture by transitioning to a server-driven framework called Flexible Authentication. The new system separates user identification from verification, using a backend policy engine to dynamically serve the optimal auth challenge and fallback options based on regional and user context. This shift eliminated dead ends, reduced client bundle sizes, and dramatically sped up experimentation velocity across Web, iOS, and Android.
How a major freight railroad scaled pipeline creation with Genie Code
A major Canadian freight railroad with roughly 20,000 route miles needed to modernize a decades-old analytical estate while handling hundreds of pipelines and rising demand for real-time analytics and AI. Using Databricks Genie Code, Unity Catalog, custom Agent Skills and a Streamlit Databricks App, the team created a metadata-grounded pipeline-generation workflow driven by compact YAML prompts. The workflow discovers schemas, maps source fields, and emits six artifacts—DDL, historical and streaming ingestion, incremental merges, and automated tests—using deterministic patterns for audit columns, deduplication, change-sequence guards and soft-delete reconciliation. Human-reviewed Source-to-Target Mapping preserves business interpretation, while Genie Code handles discovery, orchestration and artifact generation within governed Databricks execution. The reported result was more than 90% automation for new table ingestion, reducing delivery from days per table to minutes and supporting single, multi-table and bulk generation.
Introducing OlmoEarth embeddings: Custom embedding exports from OlmoEarth Studio for downstream analysis
OlmoEarth Studio now supports custom computation and export of embedding vectors, compact representations of Earth-observation data generated by open-source OlmoEarth foundation models. Users can select an area, a one-to-twelve-month time span, an encoder variant, spatial resolution, and Sentinel-2 L2A or Sentinel-1 RTC imagery, then receive a Cloud-Optimized GeoTIFF with one int8 band per embedding dimension. The post demonstrates similarity search, few-shot segmentation, change detection, and PCA-based exploration; a logistic regression trained on 60 labeled pixels produced a weighted F1 of 0.84 for three land-cover classes in Ca Mau, Vietnam. Embeddings provide a fast, shareable starting point, while supervised fine-tuning can improve performance for applications needing task-specific results; cloud cover, atmospheric artifacts, and missing observations can affect quality.
How to Choose Full-Stack Observability for NVIDIA AI Factories
AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.
How iFood built its agentic security platform on ClickHouse Cloud
iFood’s security incident response team needed high-volume log ingestion, long-term retention, and rapid access to months of history, but its Databricks-based platform became costly to scale beyond 130 TB and imposed query timeouts. The team ran a production-data proof of concept and rebuilt its tooling around ClickHouse Cloud on AWS: logs land in S3 and flow through ClickPipes into ClickHouse, with Querybook preserving notebook-based investigations and ClickStack planned for operational metrics. With ClickHouse Cloud, iFood reports 9–16x faster queries, expected cost reductions of 40–50%, and data freshness improving from hourly batches to 2–10 minutes; alerts that previously ran hourly can now run about every 10 minutes. That foundation also supports agentic threat hunting: multiple sub-agents query more than 30 TB in parallel, reducing a hunt from about a week to roughly two hours, while Langfuse is used to inspect and tune agent analyses.
Running Cost-Efficient Export Workloads at Uber
Uber’s offline export workloads often retrieve a small set of records from large, date-partitioned Apache Hudi tables, but narrow historical predicates can still trigger full-table scans. Repeatedly touching files, metadata, or Parquet footers keeps Google Cloud Storage objects hot, undermining Auto-Class tiering and increasing storage, retrieval, metadata-operation, egress, and latency costs. The approach combines Hudi column stats with sorting on predicate columns: metadata-table statistics prune files without opening them, while clustered records tighten file-level min/max ranges and make pruning more selective. A Hudi secondary index was evaluated but did not reduce file scans enough because records for a subject were spread across most files. The conclusion is that a one-time rewrite and ongoing Hudi table services can turn broad historical scans into lower-touch lookups, while future work targets multi-column layouts and automated ROI-based optimization.
How We’re Building Scam Alert on WhatsApp With End-to-End Encryption and Verifiability Guarantees
WhatsApp presents an early technical overview of Scam Alert, an optional feature that uses an on-device machine learning model to identify potential scam messages from non-contacts while keeping message content on the device. The warning appears only to the user, who can block, report, continue, or mark a chat trusted; reporting and sharing require explicit user action. To measure performance, the system sends locally aggregated warning and user action counts through confidential federated analytics using Trusted Execution Environments and differential privacy, rather than transmitting raw signals. Model versions are recorded on a public transparency ledger, while published weights, client-side logs, and an expanded Bug Bounty program support independent inspection of delivery and behavior.
The Future of Data Analytics: Why AI is rewriting the Analyst’s Job Description
The post argues that AI is not eliminating data analysts; it is automating the technical work that has crowded out their business impact. Natural-language tools such as Databricks AI/BI, Genie Code, and Genie One can generate dashboards, answer ad hoc questions, and accelerate tasks such as customer segmentation, which the post says one analyst completed in half a day instead of two months. This shifts the analyst’s focus toward problem framing, context, validation, storytelling, governance, and directing AI agents, while keeping intent, accountability, and decision ownership human. For organisations, the proposed response is to hire and develop curiosity, business acumen, and communication, embed analysts near decision-makers, retain human review, and measure decisions influenced rather than dashboards delivered.
Welcoming Fabio Kubo, our new Software Engineer
Resend announces Fabio Kubo as a new software engineer joining its software engineering team. With more than 10 years of experience, Kubo has helped scale products to millions of customers at iFood and Nu, and he will bring that experience to Resend as the platform grows. He says the product, rapid growth, technical challenges, and fully public handbook attracted him, with the company’s values, culture, working style, and transparency resonating particularly strongly. Kubo also discusses customer feedback as a source of motivation, names Notion and multi-cursor editing as favorite tools, and advises ambitious engineers to learn how they learn and understand underlying concepts when navigating new domains, technologies, and tools.
Welcoming Mateus Resende, our new Software Engineer
Resend introduces Mateus Resende as a Software Engineer based in Belo Horizonte, Brazil. He began working in software in 2015, first as a Ruby and Ruby on Rails developer before moving across domains including e-commerce recommendations, vehicle tracking, fintech, and email systems. A snowstorm during his exchange at SUNY Buffalo led him to discover MIT OpenCourseWare, code for hours, and later switch from Electrical Engineering to Computer Science. At Resend, he says its remote and async culture, extensive documentation, and developer-oriented product and company shaped his decision to join. His approach to engineering emphasizes balancing quality with speed and tailoring solutions to users and reviewers, while he recommends fundamentals such as testing, Domain-Driven Design, and Evolutionary Architectures over relying on hyped LLM coding techniques.
Introducing Grok 4.6
Grok 4.6, released with SpaceXAI, is presented as a model for long-running agents and ambitious interactive, visual, coding, and knowledge-work tasks. It builds on Grok 4.5 through a longer supplemental training run using curated model-generated reasoning and technical data, an improved optimizer and training recipe, regenerated SFT trajectories, model-based filtering, and agentic reinforcement-learning tasks across several environments. The release says Grok 4.6 matches GPT-5.6 Sol on the Artificial Analysis Intelligence Index and reports stronger first passes, sustained multi-step work, self-testing, and iterative refinement for working projects. It is available in Cursor, Grok Build, the SpaceXAI API, and through OpenRouter, Vercel, and Cloudflare, with pricing from $2 per million input tokens and $6 per million output tokens, while a fast variant costs twice as much.
How we raised mobile end-to-end test stability to 98%
Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures.
Grab Bench: Evaluating AI on Grab-shaped production work
Grab Bench is a configurable evaluation harness for assessing AI systems on production-shaped work where plausible outputs can violate metric definitions, tool contracts, evidence boundaries, or hidden code invariants. It uses YAML configuration, task plugins, provider and modality checks, row-level records, model summaries, and deterministic scorers or LLM judges according to each task. Synthetic or redacted cases preserve production-relevant difficulty while protecting live data; profile reasoning requires ontology-valid JSON claims and faithful evidence IDs, while coding tasks exercise hidden tests and anti-gaming checks. The harness also records tokens, latency, skip reasons, configurations, and failure tags so teams can diagnose whether errors reflect unsupported claims, parameter drift, hidden invariants, cost, or capability gaps. It concludes that synthetic evaluations test contract adherence under controlled pressure but do not establish production uplift.
Taking AUTO CDC to the next level: Solving the hardest real-world use cases
The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.
Scaling patterns for self-organizing multi-agent clusters with Kiro
Traditional multi-agent architectures rely on a central supervisor agent to decompose tasks and aggregate results, creating single points of failure and context-window throughput bottlenecks. To address these scaling limits on decomposable workloads, kiro-flock implements self-organizing clusters of headless Kiro CLI agents on Amazon EC2 coordinated entirely through shared Amazon S3 state. Agents operate in bounded peer topologies such as logical rings, reading neighbor append-only logs and generating artifacts without direct inter-agent messaging or central brokers. Each agent starts fresh per iteration without conversational memory, reading current direction and peer logs to prevent drift and preserve steering control. This shared-state pattern enables resilient parallelism for tasks requiring diverse independent contributions, such as code reviews or codebase migrations.
Open-sourcing Metals v2: Databricks’ Java and Scala language server for multi‑million line codebases
Databricks open-sourced Metals v2, a Scala and Java language server designed to provide low-latency code intelligence for multi-million-line Bazel monorepos, including its 26M-line repository. The project extends Metals v1 by reworking three layers: a build-free, content-addressed mbt repository index; compiler-backed Scala and Java pipelines using a Metals-provided sourcepath; and metadata-first BSP integration that keeps build synchronization off the startup path. Its Java implementation uses javac APIs and a modified Turbine header compiler, which processes close to one million Java lines per second on one thread and supports interactive analysis at nearly 100,000 lines per second in benchmarks. Metals v2 is available in Cursor, VS Code, and Neovim under Apache 2.0, while Databricks reports Cursor’s share of Scala and Java file-open events rising from 40% to 78%.
NVIDIA JetPack 7.2.1 Adds Agentic Video Skills and T3000 Emulation
JetPack 7.2.1 adds support for PyNvVideoCodec 2.2, NVIDIA's Python library for hardware-accelerated video encoding and decoding on GPUs, alongside foundational agentic video skills for Jetson workflows. These skills inspect the live target, query codec capabilities, generate Video Codec SDK or PyNvVideoCodec recipes, execute runs, and return measurements, warnings, and reproducible evidence. PyNvVideoCodec keeps frames in GPU-resident device memory through DLPack and CUDA device buffers, with multi-mode frame sampling and a ThreadedDecoder that pre-decodes frames in a background thread. The release also enables emulation of recently announced Jetson T3000 performance on a Jetson T5000 module in the Jetson Thor AGX Developer Kit, while the video skills are limited to Video Codec SDK and PyNvVideoCodec stages.
What's new in pg_clickhouse v0.10.0: Subqueries, TPC-H Speedups, C Driver, and Aggregates
pg_clickhouse v0.10.0 expands pushdown for analytic workloads, with full pushdown across the TPC-H suite remaining the stated target. The release moves the scoreboard from 12 of 22 fully pushed-down queries to 16, including Q17, which falls from 32.7 seconds to 37 milliseconds when its correlated aggregate subquery is shipped to ClickHouse. SubPlan handling now turns PostgreSQL subqueries into ClickHouse subqueries, while NOT IN and related operators receive guards that preserve PostgreSQL’s three-valued NULL semantics; compatibility requires ClickHouse 25.8 or later for the correlated-subquery shape. The binary driver was rebuilt on a plain-C client library and hardened against concurrency bugs. The release also broadens pushed-down functions and aggregates, adds clickhouse_perform() and clickhouse_server_version(), and leaves six TPC-H queries plus other roadmap items unresolved.
amazon.science
A decade of mathematical certainty: Reflections on the Automated Reasoning Group
Launched in 2016, the AWS Automated Reasoning Group applies mathematical logic, formal verification, and satisfiability modulo theories solvers to prove the correctness and security of cloud systems at scale. Early research prototypes such as Tiros and Zelkova evolved into production tools like Reachability Analyzer, Amazon Inspector, S3 Block Public Access, and IAM Access Analyzer to evaluate network reachability and access policies. Internally, the group has verified foundational infrastructure, including cryptographic libraries, boot code, the Nitro Confidentiality Engine, and an authorization engine handling one billion API calls per second. Requiring precise formal specifications frequently uncovered simpler, more maintainable system designs while establishing guarantees that traditional scenario-based testing cannot provide. Recent initiatives integrate automated reasoning into generative AI workflows, powering Amazon Bedrock Guardrails, agent compliance policies, and requirement verification in Kiro.
Car Seat FAQ
Uber Car Seat rides use a Nuna RAVA Series convertible car seat and require drivers to view a Nuna installation demonstration. The RAVA is intended for children weighing 5 to 65 pounds and measuring 49 inches or less; parents, guardians, or caregivers must verify eligibility and secure the child properly. Nuna prefers vehicle-seat-belt installation because it avoids confusion about switching from lower anchors, fits more vehicle makes, and supports more seating positions, while NYC drivers attend an installation demonstration. Children above 65 pounds or 49 inches will not have an appropriate seat, and caregivers should consult the provided videos or manual for installation and securing guidance. To address allergy concerns, drivers may ask for children’s hands to be cleaned after eating, and caregivers should ask before food is consumed in the vehicle.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.