Loading…
Topic
10 posts about Apache Flink. Every summary links to the original.
See the idea
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
Apache Flink is a distributed stream-processing framework that ingests continuous data, applies transformations in parallel, and outputs results with fault tolerance and low latency.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
IBM has released Granite Time Series PatchTST-FM-r2, a roughly 385-million-parameter foundation model for zero-shot forecasting across demand, prices, energy loads, traffic, telemetry, and other time series. The model uses conformer blocks combining multi-head self-attention with temporal convolution, 50% overlapping patches with Hamming-window weighting, overlap-and-add forecasting, contexts up to 8,192 steps, and a 99-quantile prediction head for probabilistic outputs and imputation support. On GIFT-Eval, it ranks second for both CRPS and MASE among replicable zero-shot models as of September 8, 2026, and first in that category among permissively licensed models; including pretrained models, it ranks third for CRPS and fourth for MASE. IBM provides open weights, architecture, inference code, and reproducibility code under Apache 2.0 and OpenMDW 1.0. The release also describes Python use and streaming integration through Apache Flink on Confluent Cloud.
IBM and Confluent are making IBM Granite Time Series foundation models available in Early Access on Confluent Cloud, bringing forecasting and other time-series capabilities into streaming workflows. The announcement addresses the cost and delay of bespoke models by offering models that generalize across previously unseen series and can be used for forecasting, anomaly detection, similarity search, classification, gap-filling and optimization. Native inference runs directly in Apache Flink through Confluent’s AI_FORECAST and AI_DETECT_ANOMALIES functions, with Flink maintaining keyed, fault-tolerant state for each series and results written to Kafka topics. Confluent manages serving, scaling and runtime operations while retaining schemas, lineage, access controls and replayable data within the platform. Early Access starts on Confluent Cloud on AWS at no charge, with Confluent Platform planned next for on-premises and hybrid environments.
Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.
Hugo evolved from a batch-oriented Spark platform into Grab’s unified, self-service data ingestion system for near real-time pipelines. The earlier expansion across Kafka Connect, Sprinkler, and Hugo fragmented onboarding, forcing users to translate configurations and manage manual schema changes across systems. A custom automation layer and Apache Flink now orchestrate one-click MySQL CDC and Kafka ingestion, retiring Kafka Connect and Sprinkler for the data lake. For CDC, Flink reads the binlog directly, uses automated schema detection and dynamic validation, and writes to queryable Hive tables through Spark compaction; for Kafka, it fetches Protobuf schemas from Confluent Schema Registry at runtime and resumes from the latest checkpoint after restart. Onboarding now takes approximately three minutes for MySQL CDC and six for Kafka, although schema updates still require manual restarts; new pipeline adoption in the last year also exceeded the previous five years combined.
Grab’s Coban data streaming team addresses production-only Apache Flink deployment failures, which can leave an application unavailable for about 10 minutes while the system observes it and rolls it back. Its Shadow Testing stage runs a new version in parallel with the current Main version, replaying production data without affecting Main outputs and allowing behavior and regressions to be assessed before promotion. The Shadow deployment uses a shadow Kubernetes namespace, a shadow-prefixed application name, isShadow=true, a distinct Job ID, isolated sinks, and connector-specific separation such as suffixed Kafka consumer groups and shifted Debezium Server ID ranges. By default, Shadow runs for one hour while its job-manager status is monitored; users then continue to Main deployment, which receives a 10-minute health check and automatic rollback if unhealthy. The framework isolates Kafka and S3 outputs, prefixes metrics and logs for observability, and plans more connector support.
Grab’s Customer Data Platform (CDP) introduced Scenarios to address the limits of daily batch attributes for real-time personalization across marketing and user journeys. The capability combines Scribe event triggers, geo-fencing, optional historical CDP data, and predictive models, with a self-serve GUI for configuring and validating event sequences and APIs for programmatic management. Incoming events are processed through Flink, with historical data evaluated in StarRocks and outputs written to Kafka, Amphawa, or both for downstream consumption; configured Scenarios become available within an hour and are designed for latency under 15 seconds. More than 12 implementations are live, including a Grab Unlimited abandonment intervention that notified users 15 minutes after signup drop-off and produced over a 3% uplift in subscriber conversions versus non-real-time acquisition campaigns. Planned work includes scalable Scenario sharding, multiple output destinations, and delayed triggering.
Kafka stream data can fail syntactically when actual records do not match expected schemas, or semantically when fields violate agreed patterns, ranges, or constants. The described platform addresses this gap with user-defined data contracts covering schema agreements, field-level semantic rules, stream configuration, and ownership metadata. An LLM recommends semantic rules from schemas and anonymized samples, while a transformation engine converts them into Test Runner configurations and inverse SQL queries. A long-running FlinkSQL Test Runner reads Kafka with its own consumer group, publishes issue events and stores them in AWS S3; Genchi sends Slack alerts and Coban shows violating fields and sample records. Deployed across 100+ critical topics, the solution provides real-time detection, alerting, and visibility, with future work targeting producer tracing, cross-field tests, completeness, freshness, and DTO change detection.
Grab’s Flink platform saw application count grow 2.5 times, making cost-effective, self-service CPU provisioning difficult for users who often relied on trial and error and over-provisioned resources. Reactive autoscaling with Adaptive Scheduler, Kubernetes HPA, and metrics such as CPU usage or consumer latency caused restart spikes, scaling spirals, fluctuations, and additional constraints when connectors such as Kafka reached fixed partition parallelism. Predictive Resource Advisor instead uses vertical scaling and predictive, deterministic CPU decisions, automatically collecting data, training models, and applying configuration changes. The source says applications using it experienced fewer autoscaling-related disruptions, adapted to workload growth and surges, and averaged more than 35% cloud infrastructure cost savings. It also reports simpler default-configuration deployment, while planned work includes memory tuning, improved burst handling, and more efficient model training.
Partner Gateway needed low-latency analytics for API traffic and response-time metrics across intervals from one hour to 30-day retrospectives, with a particular dataset potentially reaching approximately 6.8 billion rows in a 30-day period. Apache Pinot was selected as a distributed real-time OLAP datastore, using high-throughput ingestion and columnar, indexed, partitioned data for millisecond analytical queries. Initial aggregation queries on datasets over 150GB timed out for 20- to 30-day periods, exceeding 10 seconds and threatening a 300-millisecond SLA. The implementation routes metrics through Kafka and Apache Flink, then ingests Schema Registry-compatible messages into Pinot, while partitioning by metric name and adding aggregation-focused optimizations. For one metric, query latency fell from 5,072 to 35 milliseconds; Star-tree indexing improved speed but raised 30-day storage from 192GB to 373GB, with the additional AWS EBS cost described as minor.
The post describes replacing per-user Zeppelin notebooks with a shared FlinkSQL gateway to improve interactive stream processing over Kafka data and support production pipelines. Zeppelin notebooks were limited by support for Flink 1.17, roughly five-minute cold starts, and isolation from internal platforms, while current Flink was 1.20. The replacement uses compute, integration, and query layers: a shared FlinkSQL gateway submits SQL through its REST API to session-cluster jobs, while a custom control plane handles authentication, authorization, session management, orchestration, and paginated results for UI and headless API clients. An interactive UI uses a Hive Metastore catalog to expose Kafka topics as tables, and the platform returns results in one minute rather than five; a configuration-based deployment tool provisions resources and deploys SQL-defined Flink pipelines within ten minutes. After release, the post reports an uptick in created pipelines and interactive queries, but it does not quantify that increase.
The best engineering reads, once a week.
