Loading…
Topic
Elasticsearch
14 posts about Elasticsearch. Every summary links to the original.
- 14 posts
- 8 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Rebuilding Notion’s lexical search reindexer
Notion’s lexical search reindexer had become a multi-week, manually monitored process that produced about 90% data consistency. The old design also relied on Snowflake metadata lookups and a separate catchup pipeline. The redesign replaces ECS/Fargate workers and live Elasticsearch writes with an Apache Spark–native pipeline that routes documents to exact index-and-shard pairs and generates native snapshot files through embedded Elasticsearch nodes. Temporary indices capture live updates while the snapshot is built, after which aliases, external document versions, Elasticsearch’s _reindex, and phantom detection handle catchup, including alive-to-deleted transitions. The new system completes full reindexing in under two days, catchup in under an hour, reaches 100% consistency, removes Snowflake and ECS from the indexing path, and reduces manual intervention to under two hours per run.
Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability
Palantir’s document store needs to repair or rebuild Elasticsearch indices when mappings, settings, cluster versions, or index state change, while continuing to serve live search and writes. Its reindex machinery reconstructs schema-aware indices from the strongly consistent database, rather than relying on Elasticsearch’s _reindex API, and is designed to limit manual work and resource consumption across many deployments and schemas. For an online reindex, it creates a physical shadow index with the intended mappings and settings, backfills it from the database, dual-writes current API changes to both indices, and promotes the shadow index after completion; searches remain on the primary during the rebuild. Cross-cluster migrations use a separate machine with cluster-specific metadata, configuration, index and alias evolution, and preservation of old-cluster indices to support rollback.
May 2026 newsletter
The May 2026 ClickHouse newsletter details enterprise observability migrations, ecosystem integrations, and core database enhancements. Digital banking platform Qonto migrated from Grafana Tempo to ClickHouse Cloud, compressing 231 TB of raw trace metadata to 376 GB while expanding query windows to two weeks and introducing an AI incident companion. Similarly, LINE MAN Wongnai consolidated multiple business clusters onto self-hosted ClickHouse, achieving 1.5 million rows per second peak ingest, 100% trace retention, and a 53% reduction in observability costs. Comparative log analytics benchmarks demonstrated that ClickHouse utilized 5x less disk space and achieved 4 to 6x faster cold query execution than Elasticsearch across 50 billion rows. Additional updates include the ClickHouse 26.4 release featuring improved SQL compatibility, new agent telemetry adapters in Mastra, and SQL-based alerting capabilities in ClickStack.
Ready for your busiest day: How we scale
Fin describes how its customer service platform is designed to absorb high-volume events while preserving customer isolation, availability, and operational control. At daily peaks, it handles over 150,000 customer requests per second and more than 70,000 asynchronous requests per second; its source-of-truth data spans 128 Vitess shards, while Elasticsearch supports over 650TB, 1.7 trillion documents, and peaks above 40,000 requests per second. The architecture favors AWS primitives, Vitess managed by PlanetScale, online and incremental migrations, fair queues, feature flags, heartbeat metrics, and automatic rollback, with roughly 250 production ships per workday and merge-to-production time under 10 minutes. The stated conclusion is that scale depends less on headline capacity than on clear scaling levers, tenant fairness, customer-outcome monitoring, reversible change, and routine infrastructure improvement without planned customer-impacting downtime.
Do you still need Elasticsearch for log analytics? ClickHouse says no.
ClickHouse introduced a redesigned full-text search powered by inverted indexes that feeds matching documents directly into its vectorized analytical engine for observability and log analytics workloads. To evaluate this capability, the open-source versions of ClickHouse and Elasticsearch were benchmarked on single-node AWS EC2 instances across synthetic OpenTelemetry log datasets scaling up to 50 billion rows. The benchmark evaluated nine representative query patterns combining text search with filtering, grouping, counting, and time-series aggregations under both cold and hot cache conditions. ClickHouse achieved a roughly fivefold smaller storage footprint on disk while running cold queries four to six times faster and hot queries approximately 1.7 to 2.6 times faster than Elasticsearch. Furthermore, ClickHouse ingested the 50-billion-row dataset in under four hours out of the box, whereas the tuned Elasticsearch setup required approximately five days.
Tinder’s migration to Elasticsearch 8
Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.
Enabling conversational data discovery with LLMs at Grab
Grab faced major bottlenecks in dataset discovery across its 200,000 data lake tables due to low documentation coverage and vanilla Elasticsearch limitations in its internal tool, Hubble. To resolve this, engineers first tuned Elasticsearch by boosting frequently queried P80 tables and certified schemas while deboosting deprecated datasets. They then deployed a GPT-4 pipeline that generated documentation from table schemas and sample data, raising P80 documentation coverage from 20% to roughly 90%. Finally, the team developed HubbleIQ, an LLM-powered assistant built via Glean Apps and integrated into Slack and Hubble search to answer semantic queries. These initiatives increased search click-through rates to 94% and lifted the proportion of users finding discovery easy to 73%.
How Grab built a scalable, high-performance ad server
Grab transitioned from an off-the-shelf MVP to an in-house ad serving system to accommodate business scale, hyperlocal requirements, and machine learning personalization. The architecture orchestrates core microservices and data pipelines across sequential steps: targeting, capping, pacing, scoring, ranking, pricing, and tracking. ElasticSearch serves as the targeting ads repository, while ScyllaDB acts as the high-throughput stats store fed by Kafka streams and data pipelines. The system operates on key engineering principles including parallelization and tuned latency limits, graceful fallbacks for slow dependency calls, and a unified server serving all ad types across the superapp.
Search indexing optimisation
Grab Deliveries uses MySQL as canonical storage and Elasticsearch as derived search storage, with Food-Puxian synchronising updates through Kafka. The original incremental-sync design treated Kafka events mainly as notifications, reloaded records from MySQL, and used separate consumers and in-memory event buffering, creating heavy database traffic, data-loss gaps, coupling to table schemas, redundant Elasticsearch updates, and duplicate cascade updates. A CDC system based on MySQL row-based Binlog provides common events containing Operation, PayloadBefore, and PayloadAfter, enabling filtering of irrelevant changes, partial Elasticsearch updates, event merging in the buffer, and deduplication of cascade updates through a dedicated Kafka stream. The reported results were an 80% reduction in database load from the first optimisation and an 80% reduction in cascade-update events. Overall, DSP saved about 91% of database reads and 90% of Elasticsearch reads, while stream-consumer throughput rose from 200 to 800 QPS, with peaks above 1000 and lower synchronisation latency.
Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions
Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.
Fly Behind The Scenes: Fresh Logging
Fly replaced its centralized Graylog logging platform after seven servers could not process peak volumes of 20,000–30,000 logs per second, causing drops. Vector, written in Rust, now runs on every server, parses and transforms logs from journald and other applications, and sends them directly to Elasticsearch, removing Graylog as a middleman. The pipeline adopts the Elasticsearch Common Schema (ECS), with Fly-specific fields for app names, allocation IDs, and regions, while Vector configurations are distributed through the existing configuration management system. The migration took about two weeks alongside other work; logs are now available quickly enough to support failed-allocation queries that previously often raced processing, and future sinks may include Papertrail, Honeycomb, or Kafka.
How We’re Solving Data Discovery Challenges at Shopify
Shopify describes data discovery as a response to growing numbers of data assets and the difficulty of finding, governing, and using them across the data lifecycle. At Shopify, pre-Artifact discovery relied on people, Slack, GitHub, and job logs, while teams lacked granular ownership, change-impact visibility, and documented reporting context. They built Artifact, a search-and-browse tool backed by a centralized metadata model, ingestion jobs, Elasticsearch indexes, GraphQL APIs, and a graph database for filterable upstream and downstream lineage. Its landing page supports browsing and plain-English search, while details pages expose documentation, schema, descriptions, usage, ownership, and lineage; initial scope prioritized frequently used, high-impact, queryable assets and downstream reports and dashboards. Since launch in early 2020, weekly Data team usage exceeded 30% and monthly retention exceeded 50%, above expectations, with future work planned for additional assets, integrations, stewardship, change management, notifications, and metadata APIs.
How We Built a Logging Stack at Grab
Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.
Structured Logging: The Best Friend You’ll Want When Things Go Wrong
Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.