How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading visual explainer…
Kafka vs Temporal: When to Use Each, and When to Use Both
Kafka is a durable event stream; Temporal is a workflow orchestrator. Learn which solves your problem and why many teams use them together.
Explanation
Kafka excels at capturing, storing, and replaying sequences of events across distributed systems with durable, ordered delivery. Temporal orchestrates long-running, stateful workflows—retries, timeouts, saga compensation, and distributed coordination—without rebuilding those primitives yourself. They solve different problems: Kafka moves data reliably; Temporal manages work reliably. Real-world systems often pair them: Kafka ingests events, Temporal orchestrates responses, and together they handle both event durability and workflow resilience at scale.
The posts behind it
Related reading
Salesforce · · ExplainerHow AI-Powered Attacks Led Salesforce to Reinvent Hyperscale DDoS Defense
Salesforce built DREAM (DDoS Response and Mitigation) to defend its distributed cloud against hyperscale, machine-speed DDoS attacks, with a stated goal of responding in seconds while protecting more than 4.5 million domains and 3 million customer orgs at 99.99% availability. The platform targets rapidly evolving application-layer attacks in a shared, multi-tenant environment by combining AI-driven inference with distributed orchestration for telemetry, policy deployment, notifications, retries, and fault-tolerant execution across regions. The team selected Temporal rather than rebuilding distributed-systems primitives, and used AI coding assistants to migrate about 100,000 lines at roughly ten times prior velocity; more than 90% of the rewrite was AI-assisted, with human reviews and test coverage above 95%. Production lessons moved bulk data outside orchestration and bounded workflow history, while DREAM reduced time to first mitigation fivefold.
Netflix · · Problem & solutionA Tale of Two Flink Autoscalers
Netflix operates over 30,000 Apache Flink jobs across multiple AWS regions, ranging from managed pipelines to complex stateful DAGs. Its initial in-house autoscaler used external telemetry from Mantis and Atlas to scale TaskManagers as a single unit, cutting resource usage by 25–45% but failing on multi-operator stateful topologies. To address these limitations, Netflix adopted the Apache Flink Autoscaler library, which calculates True Processing Rate and per-vertex parallelism using internal metrics. The architecture uses Temporal workflows orchestrated within a Spring Boot application to evaluate jobs individually and prevent cross-job blast radiuses. Implementing the open-source autoscaler reduced annualized compute expenditures by 58% for client telemetry and logging, saving approximately $1.1 million.
Clickhouse · · Problem & solutionShopify powers observability for global-scale commerce with ClickHouse
Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.
Stripe · · Problem & solutionHow Stripe uses graph search and state machines to auto-remediate a global database fleet
Stripe describes how its Document Database fleet previously handled degraded MongoDB shards with single-step, hard-coded plugins that struggled with combined failures, intermediate states, and more than 40 shard layouts. The replacement models shard topology and health as a directed graph, with infrastructure operations as edges and safety invariants defining valid states. A simulator uses breadth-first search to find minimum-operation recovery paths, while an abstraction over Temporal contexts lets the same remediation logic run in memory without real infrastructure calls; later, the system also uses Dijkstra's algorithm. The system reduced pager volume by 30%—about 200 pages annually—reclaimed 12 days of fleet health each year, and handled a new layout with zero code changes, according to the post.
Cursor · · ExplainerWhat we’ve learned building cloud agents
Cloud agents have evolved from local-agent extensions into systems running on dedicated virtual machines, with isolated environments, dependencies, network access, parallel execution, and unattended long-running work. That shift makes the development environment a product: reliable execution increasingly depends on rebuilding tools, VM hibernation and resumption, image checkpointing and forking, integrations, and controlled access to secrets and networks. After a fragile work-stealing design produced roughly one 9 of reliability, the team migrated the agent loop to Temporal, whose retries and scheduling across failures helped exceed two 9s; Temporal now handles more than 50 million actions daily across over 7 million workflows. Decoupling agent-loop, machine, and conversation state enables independent pod lifecycles, asynchronous subagents, and retry-aware streaming, while shorter workflows simplify upgrades. The next goal is self-healing environments that can report and resolve missing secrets, blocked network access, or other obstacles.
More like this
More how it works explainers
How MaaS Governance Wraps a Model Endpoint on AKS
A model deployed on AKS is open by default. See how MaaS policies insert Authorino authentication and Limitador rate limits in front of vLLM, so bad keys and over-quota requests stop at the gate.
How Caddy Serves a Static Site: The Request Flow
See how Caddy handles a request for a static site, from DNS resolution through container routing to file_server, and how TLS and redirects fit into the flow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.