How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading visual explainer…
Apache Camel: From Many App Signals to Governed Actions
Apache Camel is shown as the integration path that brings application actions into one request workflow. The diagram clarifies real-time versus scheduled ingestion, mapping, merging, authorization, and audit.
Explanation
Apache Camel can act as the integration path between many application systems and a unified action workflow. In the uAct pattern, real-time fetching and scheduled offline ingestion feed application-specific mappers, after which requests are stored and merged into received and sent views. The workflow then separates action dispatch, authentication and authorization, and audit recording, helping engineers see where integration ends and governance begins.
FAQ
Where does Apache Camel sit in this workflow?
It sits on the ingestion path between application sources and the unified request workflow. In the supplied uAct example, it supports both real-time fetching and scheduled offline ingestion; mapping, storage, authorization, and auditing are shown as surrounding components.
Why combine real-time fetching with scheduled ingestion?
The combination lets a platform receive current signals while also handling offline or periodic collection from internal applications. The uAct summary describes both modes feeding a common store, where requests are merged into received and sent views.
What should be kept separate from integration?
The workflow should make action authorization and audit recording explicit rather than treating ingestion as permission to act. In uAct, the Action Dispatcher authenticates and authorizes actions, while the Audit Trail Provider records them for compliance.
The posts behind it
Related reading
Canva · · ExplainerWorker Backpressure (Part 1)
Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.
Jetbrains · · ExplainerBehind the Scenes: How the OpenTelemetry Plugin Maps Your Microservices in Real-Time
Outdated architecture diagrams and static code analysis can miss how services are connected at runtime, so the OpenTelemetry plugin builds a Service Map from live telemetry. With a lightweight local OpenTelemetry backend, the plugin supplies standard OTel environment variables, receives application spans, and continuously updates an internal architecture model that the IDE renders on demand. Its stream-processing algorithm classifies spans through semantic attributes, identifies services, matches HTTP CLIENT and SERVER spans across propagated trace context, and infers database or messaging dependencies while retaining unmatched spans in memory. Because spans can arrive out of order and traces have no definitive finish signal, the map is incrementally reconstructed rather than built from complete traces; the result is a language- and vendor-agnostic view for OpenTelemetry-instrumented applications across several JetBrains IDEs.
Uber · · ExplainerLarge-Scale Automated Dependency Analysis Across Uber's Service Mesh
Uber describes an automated method for cataloging dependency behavior across its service mesh, where nested asynchronous calls make failure attribution difficult. Because distributed traces are sampled and erroneous requests are rare, trace-based correlation can take too long; the system instead instruments yarpc middleware to observe every request and correlate inbound failures with outbound failures at service-and-endpoint granularity. An in-memory request record links each inbound request to its outbound calls, while emitted metrics capture caller and callee nodes and whether each errored. The resulting edge probability, calculated as caller failures divided by requests where the callee failed, classifies dependencies as fail-close at Pc >= 0.8, fail-open at Pc <= 0.2, or unknown between those thresholds. Uber says the initiative cataloged dependencies of critical APIs and indexed most fail-close anomalies, supporting reliability work and regression guardrails for Failover Architecture 2.0.
- Uber · · Problem & solution
uAct - Unified Action Platform
Uber built uAct to consolidate action items and approval requests generated by its many internal applications, addressing notifications that could be lost among email and follow-ups. It combines real-time fetching with scheduled offline ingestion through Apache Camel, stores requests, and merges them in received and sent views. Users can act through deep links or within uAct, where the Action Dispatcher authenticates and authorizes actions and the Audit Trail Provider records them for compliance. Its onion-architecture backend uses orchestrators, application-specific mappers, MySQL for requests and CacheFront for preferences, while parallelized paginated calls and strict timeouts support performance as integrations grow. The platform has been implemented across Uber and also provides configurable views, hiding, manager filters, observability, privacy controls, and planned Slack notifications, reporting, nudging, and additional integrations.
More like this
More how it works explainers
How MaaS Governance Wraps a Model Endpoint on AKS
A model deployed on AKS is open by default. See how MaaS policies insert Authorino authentication and Limitador rate limits in front of vLLM, so bad keys and over-quota requests stop at the gate.
How Caddy Serves a Static Site: The Request Flow
See how Caddy handles a request for a static site, from DNS resolution through container routing to file_server, and how TLS and redirects fit into the flow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.