--- title: "Latest reads" description: "The engineering internet, summarised so you can actually read it." --- # Latest reads > The engineering internet, summarised so you can actually read it. ## Articles ### [Should I Open Source my Company?](https://yomu.fyi/post/should-i-open-source-my-company.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Ant Wilson - Published: Mar 25, 2022 Software founders often hesitate to make their companies open source due to concerns regarding code scrutiny, security exploits, and intellectual property theft. Supabase operated as an open-source business from its inception, finding that community scrutiny actually accelerates code refactoring and surfaces security vulnerabilities early through public reporting channels. In addition, rapid execution and developer experience provide stronger competitive defenses against larger competitors than proprietary code obfuscation or software patents. Open repos also transformed the company's hiring strategy by replacing traditional take-home tests and LeetCode interviews with evaluation of real-world pull requests and asynchronous discussions. Finally, external contributors expand product capability by independently developing niche integrations and platform adapters that core startup teams lack the bandwidth to build. ### [Supabase Launch Week 4](https://yomu.fyi/post/supabase-launch-week-4.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Paul Copplestone - Published: Mar 25, 2022 Supabase announced the schedule and lineup for Launch Week 4, detailing daily product releases and events starting on March 28, 2022. The week initiated with Community Day to highlight community contributions and open-source tooling utilized across the platform. Technical additions across the week included the introduction of GraphQL data fetching for PostgreSQL databases, the rollout of Supabase Enterprise, and the launch of Edge Functions. Final announcements on April 1 revealed Supabrew alongside Supabase Realtime with multiplayer features, concluding with the start of a ten-day virtual hackathon running through April 9. For the hackathon, titled Bring the Func, Supabase partnered with the Future Forest Company to plant a tree for every submitted project on madewithsupabase.com. ### [How telematics helps Grab to improve safety](https://yomu.fyi/post/how-telematics-helps-grab-to-improve-safety.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Wilson Burhan - Published: Mar 24, 2022 Grab developed an in-house telematics engine to monitor driver-partner habits, detect crashes, and improve ride safety beyond passenger feedback. The mobile telematics SDK collects accelerometer, gyroscope, and GPS readings, running on-device processing for immediate accident detection while batching ride data for post-trip safety reports. Because triaxial sensor sampling rates vary independently across diverse devices, the SDK synchronizes data through interpolation to a uniform time grid followed by decimation to an output data rate. Telematics signals are transformed to the frequency domain using Fourier Transform for local compression before backend upload. Grab also combines telemetry with spatial data to identify dangerous road zones and plans to expand on-device event detection across all platform verticals. ### [Machine Learning Experts - Margaret Mitchell](https://yomu.fyi/post/machine-learning-experts-margaret-mitchell.md) - Company: huggingface.co - Author: Britney Muller - Published: Mar 23, 2022 Machine learning models often internalize harmful dataset skews and fail to evaluate real-world contexts accurately. During work on vision-to-language generation, visual recognition systems learned to misinterpret destructive disasters as positive scenes because training images predominantly featured sunsets and fireworks. Addressing these structural flaws requires moving beyond benchmark optimization toward critical data analysis, dataset genealogies, and standardized ethical AI protocols. ML organizations must establish shared vocabularies for power differentials and dismantle competitive cultural norms that marginalize underrepresented contributors. Furthermore, lowering technical barriers—such as enabling non-engineers to inspect data without SQL—allows diverse stakeholders to evaluate models and prevent widening societal power divides. ### [Announcing the 🤗 AI Research Residency Program](https://yomu.fyi/post/announcing-the-ai-research-residency-program.md) - Company: huggingface.co - Author: Douwe Kiela - Published: Mar 22, 2022 Hugging Face launched a nine-month Research Residency Program designed to train participants into impactful machine learning researchers. Residents collaborate directly with researchers from the Science Team to formulate research questions, develop novel machine learning techniques, and openly publish their findings. The initiative specifically encourages proposals from marginalized groups—including women, LGBTQ+ individuals, people of color, and working-class communities—to address disparities exacerbated by artificial intelligence progress. Applicants must demonstrate mathematical and programming abilities through coursework, open-source projects, and end-to-end proposals targeting positive societal impact. The full-time role is fully remote, provides benefits including medical coverage depending on location, and excludes concurrent student enrollment or employment. ### [Fine-Tune a Semantic Segmentation Model with a Custom Dataset](https://yomu.fyi/post/fine-tune-a-semantic-segmentation-model-with-a-custom-dataset.md) - Company: huggingface.co - Author: Tobias Cornille, Niels Rogge - Published: Mar 17, 2022 Fine-tuning a semantic segmentation model requires domain-appropriate training data and an efficient pipeline. Existing autonomous driving datasets feature roadway imagery captured by cars, creating a distribution mismatch for sidewalk-based delivery robots. To resolve this discrepancy, a dedicated dataset of sidewalk imagery is loaded from the Hugging Face Hub, split into training and test sets, and augmented on-the-fly using SegformerImageProcessor and torchvision. The smallest SegFormer architecture, B0, is fine-tuned using Hugging Face's Trainer API with mean Intersection over Union evaluation metrics. The final pipeline pushes the fine-tuned model to the Hub and executes inference by upsampling output logits to original image dimensions. ### [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://yomu.fyi/post/accelerate-bert-inference-with-hugging-face-transformers-and-aws-infer.md) - Company: huggingface.co - Author: Philipp Schmid - Published: Mar 16, 2022 Production deployments of BERT and Transformer architectures often face cost and latency challenges because these models are significantly larger and more computationally intensive than traditional algorithms. To optimize text classification workloads, developers can compile vanilla PyTorch models for AWS Inferentia using the AWS Neuron SDK and its tracing utilities. Because the Neuron SDK requires static tensor dimensions, the model is traced with fixed input lengths and packaged alongside a custom inference script configuring one Neuron Core per worker. Deploying the resulting artifacts to an Amazon SageMaker ml.inf1.xlarge endpoint yields an average latency of 5 to 6 milliseconds for a sequence length of 128 across 10,000 synchronous evaluation requests. ### [Image search with 🤗 datasets](https://yomu.fyi/post/image-search-with-datasets.md) - Company: huggingface.co - Author: Daniel van Strien - Published: Mar 16, 2022 Hugging Face datasets expanded its capabilities by introducing an Image feature type, enabling image processing and integration with vector indexing tools. The library was applied to a sample of historical book embellishments extracted via OCR from the British Library. Images were loaded using the ImageFolder loader, enriched with filename metadata, and pushed to the Hugging Face Hub. A FAISS index and CLIP embeddings were used to retrieve images matching natural language queries, such as categories, specific objects, and boolean operators. While the retrieval system demonstrated reasonable semantic search results across various prompts, full public deployment was avoided due to CLIP model card restrictions and potential bias in the historical dataset. ### [Real-time data ingestion in Grab](https://yomu.fyi/post/real-time-data-ingestion-in-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuguang Xiang - Published: Mar 14, 2022 Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails. ### [Guiding Text Generation with Constrained Beam Search in 🤗 Transformers](https://yomu.fyi/post/guiding-text-generation-with-constrained-beam-search-in-transformers.md) - Company: huggingface.co - Author: Chan Woo Kim - Published: Mar 11, 2022 Constrained beam search introduces direct control over generated text in Hugging Face Transformers. Standard beam search operates token-by-token without knowing the optimal step to force specific words or phrases, making it difficult to enforce mandatory vocabulary or choose between alternative expressions. To resolve this, the generate interface accepts constraints through arguments such as force\_words\_ids and a list of Constraint subclasses like PhrasalConstraint. This mechanism enables disjunctive constraints, where generation must include at least one phrase from a provided set, alongside strictly required sequences. Consequently, practitioners can inject prior knowledge or formatting requirements directly at generation time rather than filtering candidate outputs afterward. ### [Postgres Auditing in 150 lines of SQL](https://yomu.fyi/post/postgres-auditing-in-150-lines-of-sql.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Oliver Rice - Published: Mar 8, 2022 Traditional PostgreSQL auditing methods duplicate source table structures, creating maintenance overhead when schemas change. To solve this, a compact SQL implementation stores insert, update, and delete events across multiple tables in a single audit table using JSONB columns. Query performance is maintained by using a BRIN index on naturally ordered insertion timestamps, a B-tree index on internal table OIDs, and UUIDv5 identifiers hashed from primary keys. Row-level PL/pgSQL triggers automatically populate the audit log and can be enabled or disabled dynamically with dedicated tracking functions. While trigger-based auditing introduces minimal overhead at rates below 1,000 writes per second, write-heavy workloads may benefit from logging changes outside SQL with tools like pgAudit. ### [BERT 101 - State Of The Art NLP Model Explained](https://yomu.fyi/post/bert-101-state-of-the-art-nlp-model-explained.md) - Company: huggingface.co - Author: Britney Muller - Published: Mar 2, 2022 Developed in 2018 by Google AI Language, Bidirectional Encoder Representations from Transformers addresses the historical challenge of machines lacking contextual understanding of human language. The model relies on an encoder-only Transformer architecture pre-trained on a 3.3-billion-word corpus consisting of Wikipedia and Google BooksCorpus. Training simultaneously combines masked language modeling, which hides 15% of tokenized words to enforce bidirectional context learning, with next sentence prediction across balanced sentence pairs. Pre-trained on Cloud TPUs over four days, BERT unifies solutions for more than eleven common NLP tasks and can be fine-tuned on task-specific annotated data within minutes. Unmasking experiments demonstrate that the model can also inherit distinct societal and gender biases from its underlying training corpora when predicting professions. ### [Abacus - Issuing points for multiple sources](https://yomu.fyi/post/abacus-issuing-points-for-multiple-sources.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chandrakanth - Published: Mar 1, 2022 Grab needed a centralised points management architecture to issue loyalty points across a growing catalog of products, membership tiers, and external partner exchanges. To address this, the engineering team built Abacus, an issuance platform designed to process millions of daily transactions with high availability. The system ingests completed transaction streams or API calls, dynamically computes points via configured multipliers, and passes calculations through Amazon Simple Queue Service queues. Once the Point Awarding module updates a persistent ledger, Abacus notifies consumers, emits events to Kafka for downstream consumers, and recalculates rolling point expiration dates. ### [Supabase Beta January 2022](https://yomu.fyi/post/supabase-beta-january-2022.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Paul Copplestone - Published: Feb 22, 2022 Supabase announced product updates, platform improvements, and community resources released during January 2022. The platform expanded authentication options by adding Notion and LinkedIn OAuth support alongside Vonage and Textlocal SMS providers for one-time passwords. Observability tooling gained SQL querying capabilities for logs, timestamp filtering, and expanded time spans across project usage charts to accelerate issue diagnosis. Database infrastructure advanced with the release of pg\_graphql version 0.1.0 introducing SQL Comment Directives, while PostgREST replaced pg\_listen with built-in schema reloading and automated compute-based connection pool scaling. To streamline customer assistance, a priority selector was added to dashboard support forms with urgent tiers for paid projects. ### [Exposing a Kafka Cluster via a VPC Endpoint Service](https://yomu.fyi/post/exposing-a-kafka-cluster-via-a-vpc-endpoint-service.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Fabrice Harbulot - Published: Feb 18, 2022 To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces. ### [How Grab built a scalable, high-performance ad server](https://yomu.fyi/post/how-grab-built-a-scalable-high-performance-ad-server.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Anthony McCallum - Published: Feb 11, 2022 Grab transitioned from an off-the-shelf MVP to an in-house ad serving system to accommodate business scale, hyperlocal requirements, and machine learning personalization. The architecture orchestrates core microservices and data pipelines across sequential steps: targeting, capping, pacing, scoring, ranking, pricing, and tracking. ElasticSearch serves as the targeting ads repository, while ScyllaDB acts as the high-throughput stats store fed by Kafka streams and data pipelines. The system operates on key engineering principles including parallelization and tuned latency limits, graceful fallbacks for slow dependency calls, and a unified server serving all ad types across the superapp. ### [Fine-Tune ViT for Image Classification with 🤗 Transformers](https://yomu.fyi/post/fine-tune-vit-for-image-classification-with-transformers.md) - Company: huggingface.co - Author: Nate Raw - Published: Feb 11, 2022 Vision Transformer models bring transformer architectures to computer vision by splitting images into grids of sub-image patches and projecting them into token sequences. To classify healthy and diseased leaves using the beans dataset, the Hugging Face datasets and transformers libraries enable streamlined data ingestion and model fine-tuning. Preprocessing relies on ViTImageProcessor paired with lazy on-the-fly dataset transforms to dynamically generate normalized pixel tensors. Training leverages ViTForImageClassification with the Trainer API, requiring remove\_unused\_columns set to False so raw image data is preserved for batch collation. Over four training epochs, fine-tuning the google/vit-base-patch16-224-in21k checkpoint achieves an evaluation accuracy of 98.5% alongside an evaluation loss of 0.0637. ### [Getting Started with Sentiment Analysis using Python](https://yomu.fyi/post/getting-started-with-sentiment-analysis-using-python.md) - Company: huggingface.co - Author: Federico Pascual - Published: Feb 2, 2022 Modern natural language processing tools allow developers to perform sentiment analysis without deep machine learning expertise. Sentiment analysis categorizes text polarity into positive, negative, or neutral labels to extract insights from large volumes of social posts, reviews, and support tickets. Using the Hugging Face Transformers library and the Hub, practitioners can quickly run pre-trained transformer pipelines or target specific multilingual and emotion-detection models with minimal Python code. For custom requirements, developers can fine-tune models like DistilBERT using the IMDB dataset via the Trainer API or train models automatically with AutoNLP. Applying these workflows to social media data enables automated sentiment distribution analysis and visualization using pandas, matplotlib, and word clouds. ### [Making automatic speech recognition work on large files with Wav2Vec2 in 🤗 Transformers](https://yomu.fyi/post/making-automatic-speech-recognition-work-on-large-files-with-wav2vec2.md) - Company: huggingface.co - Author: Nicolas Patry - Published: Feb 1, 2022 Transformer-based automatic speech recognition models like Wav2Vec2 crash on long audio inputs because the O(n²) attention mechanism rapidly exhausts GPU memory. Simple audio chunking avoids out-of-memory errors but causes poor transcription accuracy at chunk boundaries due to a lack of surrounding context. To resolve this, Hugging Face Transformers leverages the Connectionist Temporal Classification architecture to process overlapping audio chunks with configurable strides. The pipeline discards low-quality logits at chunk edges and chains the remaining central predictions together to reconstruct a seamless transcript. This striding technique operates out of the box with language model-augmented pipelines and adapts directly to low-latency live audio streaming. ### [Supercharged Searching on the 🤗 Hub](https://yomu.fyi/post/supercharged-searching-on-the-hub.md) - Company: huggingface.co - Author: Zachary Mueller - Published: Jan 25, 2022 Programmatically searching the Hugging Face Hub previously required navigating web browser widgets or enduring trial and error to guess exact query string formats. The huggingface\_hub library solves this issue by introducing helper utilities such as ModelSearchArguments, DatasetSearchArguments, and ModelFilter alongside HfApi. These namespace helpers translate accessible Python attributes into the formatted parameters expected by the backend API, covering datasets, tasks, and libraries. For complex queries across multiple tasks, frameworks, and datasets, developers can combine criteria inside ModelFilter instances and retrieve matching model metadata via api.list\_models. Under the hood, the library uses AttributeDictionary, a data structure inspired by fastcore that enables nested tab-completion while supporting standard dictionary key indexing for special characters. [Newer posts](https://yomu.fyi/page/45.md) · [Older posts](https://yomu.fyi/page/47.md)