How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading visual explainer…
How Aerospike Handles High-Speed Counters
Aerospike is an in-memory database optimized for real-time counters and fraud detection at extreme scale. Learn how it replaces wide-column stores and cuts latency and cost.
Explanation
Aerospike is a purpose-built in-memory database designed for applications that need sub-millisecond latency and extreme throughput—like fraud detection systems processing tens of thousands of queries per second. Unlike traditional wide-column stores, Aerospike uses a shared-nothing architecture with atomic operations, efficient memory layout, and optional persistent storage to deliver both speed and durability. When migrating workloads like counter services, teams redesign around Aerospike's single-key atomic increments and map-based schemas rather than trying to replicate old patterns. The result is typically 50% latency reduction, smaller disk footprint, and significantly lower per-node costs while maintaining data integrity.
The posts behind it
Related reading
Stack Overflow · · ExplainerThe economics of agent scale: tokens, ROI, and building platforms for AI-first teams (Part 2)
The conversation examines the economics and platform implications of operating dozens or hundreds of AI agents rather than a small number of human-driven workflows. Andi Gutmans argues that many current models are already sufficient for automation, shifting the optimization problem to the complete system: model, harness, context, personalization, and evaluation. The goal is the least sophisticated model and smallest viable context that produce a reliable outcome at the lowest cost, with better trajectories reducing reasoning loops and token processing; Google’s stated approach is to hill climb models and data together. The discussion also addresses cost governance, search and observability for agents as a distinct persona, human-in-the-loop handling, and the continuing importance of computer-science fundamentals, agentic fluency, and business judgment.
Databricks · · ExplainerSouthern Company’s SCOUT: Completing the Storm Intelligence Story
Southern Company’s SCOUT is a real-time storm operations application designed to make outage intelligence accessible beyond control-center desks during active events. Built on the Databricks lakehouse, it combines Delta Lake storage, Databricks Lakehouse warehouses, Unity Catalog governance, and collaborative notebooks, querying a dedicated warehouse every minute for current operational data. The mobile-friendly application unifies outage, customer, GIS, weather, terrain, crew, and reliability information, helping teams assess restoration times, assign appropriately equipped crews, coordinate mutual assistance, and communicate with customers. SCOUT is used across Southern Company’s operating businesses and had 1,139 adopted employees, including more than 250 users on a peak storm day; the source also describes exploration of AI crew recommendations and autonomous-drone integration, not a completed deployment.
Fly.io · · ExplainerShared Nothing Architecture
Shared-nothing architecture is presented as a scaling model for Showcase, a Ruby on Rails ballroom-competition application that began on a Mac Mini and grew to cloud VMs serving events in multiple regions. Because each event has few geographically concentrated users, a roughly one-megabyte SQLite database, about one transaction per second, and typical response times of 100ms or less, the design keeps state separate rather than sharing databases and Redis across events. The implementation runs an event’s services on one Fly.io VM, repeats that unit across machines or regions, and can place multiple event instances on one server with Phusion Passenger, NGINX, volumes, and a shared Action Cable process. It specifies 1GB machines with 2G swap after observing about 220MB idle and 450MB active RAM, and recommends WireGuard; the conclusion is that local state and fewer network calls suit this workload and can improve throughput and reliability.
Grab · · Problem & solutionMigrating Counter Service storage: Design choices and learnings
Grab’s Counter Service supports time-windowed fraud queries at tens of thousands of QPS and about a billion requests per day, but an infrastructure review prompted migration from a wide-column database to Aerospike. The team first separated Rust business logic from storage, then added an enum-dispatched facade with Single, WithShadow, and WithSplit modes so traffic could be compared and shifted without downtime. They also redesigned the writer path around Aerospike, including a map-based schema and a single atomic increment, while testing index placement and reverting from an NVMe-backed primary index after hot-key p99 spikes. Production p99 read latency improved by roughly 50%, disk usage fell from about 3 TB to 1 TB, and per-node cost dropped 45–50%; the rollout reported zero downtime and no data-integrity issues.
More like this
More how it works explainers
How MaaS Governance Wraps a Model Endpoint on AKS
A model deployed on AKS is open by default. See how MaaS policies insert Authorino authentication and Limitador rate limits in front of vLLM, so bad keys and over-quota requests stop at the gate.
How Caddy Serves a Static Site: The Request Flow
See how Caddy handles a request for a static site, from DNS resolution through container routing to file_server, and how TLS and redirects fit into the flow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.