Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Grab ·
Returning 575 Terabytes of Storage Space to Our Users
Android Vitals data revealed that 15.7% of Grab users had less than 1GB of free device storage and uninstalled the app at 1.2 times the normal rate. To understand on-device storage consumption, the team instrumented session launches using the Android StorageManager API to collect binary size, cache folder size, and total footprint metrics. Analysis showed unusually large cache sizes driven by orphaned cache folders from discontinued third-party libraries, including an image library replacement of Picasso by Glide. An automated cleanup routine deployed in app updates purged legacy cache directories upon launch. This mechanism reclaimed 575 terabytes of junk data across more than 13 million devices, averaging 40MB per user.
Lucas NelaupeGrab ·
Grab-Posisi - Southeast Asia’s First Comprehensive GPS Trajectory Dataset
Grab released Grab-Posisi, a GPS trajectory dataset covering Singapore and Jakarta designed to overcome the poor coverage, low sampling rates, and missing contextual metadata common in existing public datasets. Captured in April 2019 from drivers in transit, the dataset comprises 84,000 trajectories, over 80 million GPS pings, and more than 1 million kilometres recorded at a one-second sampling frequency. Each entry in the 2 GB Apache Parquet dataset records geographic coordinates, timestamp, accuracy radius, bearing, speed, mobile operating system, and vehicle mode. To protect privacy, driver personal information is encrypted and trip start and end locations are removed. The data enables applications including automated road network reconstruction, map matching, real-time traffic forecasting, and municipal infrastructure planning.
Zhengmin XuZoom ·
New API Rate Limiter
Zoom announced an updated V2 REST API Rate Limiter launching in March 2020 to replace its previous one-size-fits-all throttling system. Rather than applying uniform constraints across all subscription tiers, the new framework cross-references customer Account Types against API Request Groups categorized by infrastructure importance and load intensity. While most API rate limits are increasing, resource-intensive Dashboard APIs are reduced from 1 request per second to 6 or 12 requests per minute depending on account level. Throttled HTTP 429 responses will return X-RateLimit-Reset timestamps for per-minute limits and ISO8601 Retry-After timestamps for per-day limits. Additionally, the system introduces lock-key restrictions on specific user identifier path variables to restrict concurrency and enforce daily meeting creation caps.
Benjamin DeanZoom ·
Developing Zoom Marketplace Apps w/ ngrok
Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.
Tim SlagleZoom ·
What’s New and What’s Coming for Zoom Marketplace in 2020?
Zoom announced several updates to its App Marketplace alongside roadmap deadlines scheduled for early 2020. Marketplace enhancements include functional app categories, additional URL fields for Terms of Use and documentation, and support for defining multiple webhook event subscriptions within a single application. Administrators and developers gained webhook logs, enhanced pre-approval permission settings, and the ability to test draft applications directly in the installed apps interface for Data Compliance API compliance. To accommodate surging traffic, Zoom introduced revamped V2 REST API rate limits and breaking behavioral changes to the List Users API. Crucially, the platform set an absolute end-of-life date of April 26, 2020, for all legacy V1 APIs, requiring immediate migration to V2 endpoints.
Benjamin Deanhuggingface.co ·
How to train a new language model from scratch using Transformers and Tokenizers
Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.
Julien ChaumondZoom ·
Investing in Developers — Zoom Developer Summit and Competition 2020
Zoom launched the Zoom Marketplace Competition for developers creating applications on the Zoom platform. Open to for-profit organizations with less than $10M in prior funding, the competition offers a selected winner up to $2 million in potential investment. Finalists pitch their proposals to venture partners from Emergence, Horizons Ventures, Maven Ventures, and Sequoia. In addition to capital, the winner receives advisory sessions with Zoom product experts, priority platform support, a DTEN D7 whiteboard, and multiple three-year software licenses. Submissions require explicit details regarding problem definition, customer demographics, traction, revenue models, competitive analysis, and team credentials ahead of the March 31st, 2020 deadline.
Aleks SwerdlowZoom ·
Zoom Developer Relations — Meet the Team
Zoom's Developer Relations team supports third-party developers, manages the App Marketplace, and maintains platform tools including APIs and Web SDKs. The team comprises technical writers, developer advocates, developer educators, security engineers, and marketplace specialists. Specific responsibilities include authoring integration documentation, reviewing early API designs, testing webhooks, scheduling security evaluations for Marketplace applications, and building sample code. Developer interactions take place across multiple channels, including the Zoom Developer Forum, Zendesk support tickets, Zoom meetings, app submission review notes, and partnership calls. Specialists also guide creators through the formal app submission lifecycle, reviewing functional workflows, privacy statements, and regional ISV opportunities across global markets like APAC.
Abraham QueenZoom ·
Behind the Curtain: App Publishing Requests
Zoom App Marketplace reviews App Publishing Requests to enable broad distribution of applications across all Zoom accounts. During the initial review stage, marketplace specialists evaluate submission metadata, including descriptive text, media assets, functional documentation, and requested scopes to verify that applications do not access unnecessary user information. Reviewers then perform a functional assessment by executing the developer-provided test plan using supplied test accounts and credentials. When issues occur, reviewers deliver feedback through review notes, requiring developers to resolve the problems and resubmit their app to trigger a re-review cycle. Developers can validate production readiness prior to public submission through account-level local testing or by creating a Publishable URL sharing request for external testing.
Abraham QueenGrab ·
How We Prevented App Performance Degradation from Sudden Ride Demand Spikes
Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.
Corey ScottGrab ·
Plumbing At Scale
Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.
Karan KamathGrab ·
Journey to a Faster Everyday Superapp Where Every Millisecond Counts
Grab undertook an initiative to reduce startup time and improve time to interactive (TTI) on its passenger mobile app. Because local benchmarks failed to simulate real device and network conditions, the team instrumented code in production across 8–9 million daily users to capture p50 and p95 metrics. Initial gains came from caching service tiles between sessions and removing a startup animation, saving four seconds. Architectural changes followed, including converting iOS dynamic frameworks to static linking and merging others, while Android initialisation was refactored with Kotlin coroutines. Replacing a heavy third-party analytics library with an internal experimentation platform yielded further startup reductions.
Renu YadavGrab ·
Marionette - Enabling E2E User-scenario Simulation
Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs.
Anish JhaGrab ·
How We Implemented Domain-Driven Development in Golang
Building GrabPlatform's partner integration self-service portal initially resulted in an unstructured codebase where individual files exceeded 500 lines and lacked proper segregation. Modifying existing functions carried high risks of breaking functionality across imported source collections. To resolve this, the team restructured the Go application using Domain-Driven Design principles in coordination with product domain experts. They mapped business rules into bounded contexts, identified entities and aggregate roots, introduced repository interfaces, and utilized domain events for cross-context communication. The refactoring distributed core functionality evenly, simplified onboarding, and aligned technical terminology with business concepts.
Kapil ChaurasiaGrab ·
Driving Southeast Asia Forward Through People-Focused Design
Designing digital products for Southeast Asia requires tailoring user experiences to unique regional constraints and consumer behaviors across diverse populations. Users in the region often operate low-end mobile hardware on congested networks while carefully rationing prepaid mobile data. Grab addresses these challenges by designing comprehensively for non-ideal UI stacks, implementing loading skeletons, and replacing heavy video tutorials with lightweight SVG animations to minimize bandwidth consumption. Furthermore, product teams adapt to mobile-only environments by prioritizing phone number and one-time-password registrations while avoiding legacy desktop-era iconography. Visual accessibility is validated by testing UI readability on dimmed, low-resolution screens under bright ambient sunlight.
Philip MadeleyGrab ·
Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily
Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.
Muqi LiGrab ·
Using Grab’s Trust Counter Service to Detect Fraud Successfully
Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs.
Chao WangGrab ·
Being a Principal Engineer at Grab
Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people.
Roman AtachiantsGrab ·
Data First, SLA Always
Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.
Johan KokGrab ·
Save Your Place with Grab!
Grab introduced Saved Places across Southeast Asia to eliminate the friction of repeatedly typing addresses and prevent selection errors between similarly named locations. Data analysis across transport and food orders revealed that consumers consistently visit only five to seven unique locations and order food to one or two addresses. To streamline repeat bookings, cross-functional teams designed a feature allowing users to bookmark locations under custom labels such as Home and Work. Following usability testing and release, more than 14 million users stored nearly 45 million addresses across the platform. Platform metrics also showed that while office destinations clustered in central districts across major cities, residential distributions varied significantly between markets like Singapore and Jakarta.
Summit Saurav
