---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [Scaling Large Language Models for e-Commerce: The Development of a Llama-Based Customized LLM](https://yomu.fyi/post/scaling-large-language-models-for-e-commerce-the-development-of-a-llam.md)
- Company: [Ebay](https://yomu.fyi/company/ebay.md)
- Author: Christian Herold and Shahram Khadivi
- Published: Jan 17, 2025
- Type: Problem & solution

eBay developed e-Llama, customized 8B and 70B language models, by adapting Meta’s Llama 3.1 models to e-commerce rather than training entirely from scratch. The approach uses continued pretraining on filtered public listings, product reviews, classifier-selected open-source examples, replay data, and 10% non-English general-domain data to add domain knowledge while limiting catastrophic forgetting. Training ran on 480 NVIDIA H100 GPUs using Megatron-LM, 3D parallelism, flash-attention-2, a 1:1 general-to-e-commerce sampling ratio, a learning-rate maximum set to 10% of the original, and one trillion tokens. The 70B model required about one month and 340,000 GPU-hours. E-commerce benchmarks improved approximately 25% in English and 30% in non-English, while general-domain NLU degraded 1% for e-Llama 70B; subsequent instruction tuning added safety and instruction-following guardrails.


### [A new Tab model](https://yomu.fyi/post/a-new-tab-model.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Phillip Kravtsov & Jacob Jackson
- Published: Jan 13, 2025
- Type: Announcement

Cursor announces Fusion, a next-generation model for Cursor Tab, which predicts edits near the cursor and “jumps” to likely next editing locations. The post positions this as a response to the limits of cursor-only text insertion, with a broader goal of reducing code-editing tedium and moving toward in-flow Next Action Prediction. Compared with the March 2024 model, Fusion predicts over 25% more difficult edits per line, suggests over 10x longer change stretches, cuts median server latency from 475ms to 260ms, expands context from 5,500 to 13,000 tokens, and adds instant, accurate cursor jumps. These gains are attributed to cleaner and larger datasets, longer editor-state prompts, training for larger edits, synthetic instruction-following data, and inference and performance improvements. Fusion is rolling out to all users in client release 0.45.0.


### [Character Prefix Conditioning](https://yomu.fyi/post/character-prefix-conditioning.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Jacob Jackson
- Published: Jan 6, 2025
- Type: Problem & solution

The post poses a problem in code completion: generating a language-model completion that begins with the characters already typed by the user. Because modern models process token sequences rather than characters, naively tokenizing the input can produce the wrong result when the cursor falls inside a token. It introduces character prefix conditioning, defined as sampling tokens from an autoregressive distribution while constraining their decoded character string to start with the given prefix. The challenge is to construct an efficient autoregressive algorithm for this conditional distribution while minimizing calls to the original language model; the post requests either an algorithm description or an implementation.


### [Supercharging LLM application development with LLM-Kit](https://yomu.fyi/post/supercharging-llm-application-development-with-llm-kit.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Boon Zhan Chew
- Published: Nov 29, 2024
- Type: Problem & solution

Grab developed LLM-Kit, a comprehensive framework for setting up production-ready Generative AI applications while addressing security, scalability, integration, monitoring, and standardisation concerns. Its pre-configured structure includes an API server, configuration management, a sample LLM Agent, tests, and an integrated stack spanning Poetry, Gunicorn, FastAPI, LangChain, LangSmith, HashiCorp Vault, Amazon EKS, and GitLab CI. Engineers submit application details through a form, which triggers GitLab project and code-scaffold generation, CI/CD setup, Terraform-based staging and production infrastructure provisioning, and secure GPT-token storage in Vault. The scaffold also includes authentication, API documentation, observability, Redis and vector-database integrations, deployment pipelines, and a cookbook with organisational examples; the post reports that it has onboarded hundreds of GenAI applications and saves an estimated 1.5 weeks before teams begin their first feature.


### [Metasense V2: Enhancing, improving and productionisation of LLM powered data governance](https://yomu.fyi/post/metasense-v2-enhancing-improving-and-productionisation-of-llm-powered.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nick Buhrer
- Published: Nov 14, 2024
- Type: Problem & solution

Metasense V2 addresses automated data governance by improving an LLM-powered metadata generation and classification system. It builds on an initial rollout that scanned more than 20,000 data entries at 300–400 entities per day, but retained human verification to catch misclassifications. Post-rollout owner feedback and manual classifications from the Data Governance Office supplied training and testing data, exposing failures involving PII in business email columns, nested JSON, and passenger communications. The revised approach splits PII and non-PII tagging, reduces the PII tag set from 21 to 8, shortens the prompt from 1,254 to 737 words, and divides tables exceeding 150 columns. LangChain and LangSmith support prompt development, evaluation across accuracy, latency, and error rate, deployment, and monitoring, while threshold-based alerts trigger a model-improvement protocol when misclassification rates rise.


### [Adding payments to your LLM agentic workflows](https://yomu.fyi/post/adding-payments-to-your-llm-agentic-workflows.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Steve Kaliski
- Published: Nov 14, 2024
- Type: Explainer

Agentic workflows combine large language models with function calling to turn prompts into sequences of actions, but connecting those actions to financial services requires controlled API access. The Stripe agent toolkit integrates Stripe with Vercel’s AI SDK, LangChain, and CrewAI, supports function-calling LLM providers, and is built on Stripe’s Node.js and Python SDKs. It exposes configurable tools for tasks such as invoicing, creating and sharing payment links, and using Issuing virtual cards with spending limits, authorization checks, and deactivation after purchase. Middleware can record input and completion token counts as billing events for usage-based customer billing. Because agent behavior is non-deterministic, the post recommends test mode, evaluations, restricted keys, limited tool sets, and reduced request and response shapes to improve focus and constrain failures.


### [LLM-assisted vector similarity search](https://yomu.fyi/post/llm-assisted-vector-similarity-search.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Oct 23, 2024
- Type: Benchmark

Vector similarity search can struggle with queries containing negation, constraints, or conceptual requirements, even though it efficiently retrieves semantically similar records. The proposed method uses FAISS and OpenAI text-embedding-ada-002 embeddings to shortlist 15 candidates, then asks GPT-4o to rank the candidates against the original query and return the top three. On two GPT-4o-curated synthetic datasets—100 food dishes and 100 Asian tourist spots—the assisted method matched raw search on straightforward queries but produced more relevant results for complex ones, including excluding fish or shrimp and identifying wildlife-related places. The authors note limits from the small datasets and query set, report similar relevance improvements in an internal deployment involving more than 4,500 relational-database rows, and identify shortlist size and added LLM latency as areas for further evaluation.


### [Leveraging RAG-powered LLMs for analytical tasks](https://yomu.fyi/post/leveraging-rag-powered-llms-for-analytical-tasks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Edmund Hong
- Published: Oct 9, 2024
- Type: Problem & solution

Data analysts at Grab faced an increasing volume of stakeholder data requests that required manually writing and executing repetitive SQL queries with minor parameter changes. To automate routine tasks like metric reporting and fraud investigations, the Integrity Analytics team paired an internal prompt-management platform, Spellvault, with Data-Arks, an in-house Python-based middleware that packages SQL queries and Python functions into APIs. A scheduler triggers automated report summarization, while an analytical bot named A\* bot executes relevant investigation queries via RAG and returns summarized findings in Slack. Automated report generation saves an estimated three to four hours per report and shortens multi-query fraud investigations to minutes. Grab favored RAG over fine-tuning because it avoids continuous retraining costs, scales faster across diverse use cases, and surfaces current production data.


### [Enabling conversational data discovery with LLMs at Grab](https://yomu.fyi/post/enabling-conversational-data-discovery-with-llms-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shreyas Parbat
- Published: Sep 26, 2024
- Type: Problem & solution

Grab faced major bottlenecks in dataset discovery across its 200,000 data lake tables due to low documentation coverage and vanilla Elasticsearch limitations in its internal tool, Hubble. To resolve this, engineers first tuned Elasticsearch by boosting frequently queried P80 tables and certified schemas while deboosting deprecated datasets. They then deployed a GPT-4 pipeline that generated documentation from table schemas and sample data, raising P80 documentation coverage from 20% to roughly 90%. Finally, the team developed HubbleIQ, an LLM-powered assistant built via Glean Apps and integrated into Slack and Hubble search to answer semantic queries. These initiatives increased search click-through rates to 94% and lifted the proportion of users finding discovery easy to 73%.


### [Chimera Sandbox: A scalable experimentation and development platform for Notebook services](https://yomu.fyi/post/chimera-sandbox-a-scalable-experimentation-and-development-platform-fo.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Arkar Min Aung
- Published: Aug 27, 2024
- Type: Announcement

Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution.


### [How we improved translation experience with cost efficiency](https://yomu.fyi/post/how-we-improved-translation-experience-with-cost-efficiency.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jie Zhang
- Published: Aug 5, 2024
- Type: Problem & solution

Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution.


### [LLM-powered data classification for data entities at scale](https://yomu.fyi/post/llm-powered-data-classification-for-data-entities-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hualin Liu
- Published: Jul 15, 2024
- Type: Problem & solution

Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.


### [More problems](https://yomu.fyi/post/more-problems.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Aman Sanger
- Published: May 25, 2024
- Type: Explainer

The post outlines research problems for the next phase of AI programming, centered on making coding systems predict actions, perform larger edits, understand codebases, and debug software. Proposed directions include low-latency next-action models for keystrokes, locations, files, terminal commands, and edits; background execution for “perfect edits” and pseudocode compilation; and multi-file changes supported by stronger tool use, reasoning, and agent trajectories. For context, retrieval alone is described as insufficient for codebase questions requiring multiple hops, while useful context may span documentation, source, history, UI pixels, logs, and messages. The post recommends combining retrieval, recurrence, and long-context attention, alongside specialized rerankers, multi-hop embedders, caching, and codebase-level retrieval. For reliability, it proposes calibrated bug detection, always-on AI linting, runtime-state tracking through cursor/debug, and improved debugger tool use, while noting trade-offs involving latency, false positives, infrastructure, and user experience.


### [Editing Files at 1000 Tokens per Second](https://yomu.fyi/post/editing-files-at-1000-tokens-per-second.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Aman Sanger
- Published: May 14, 2024
- Type: Explainer

Large code edits expose coding agents to laziness, inaccuracy, and latency, especially when frontier models must rewrite hundreds of lines. The post presents fast apply, a specialized full-file code-edit model that separates planning from applying: Cursor's frontier model plans the change, while the apply model rewrites the current file. It evaluates roughly 450 edits on files under 400 lines, using Claude-3 Opus as grader, and finds llama-3-70b-ft nearly matches claude-3-opus-diff while outperforming GPT-4-Turbo and GPT-4o. Rather than generate conventional diffs, it uses full-file rewrites and search/replace hunks, which avoid line-number errors and give models more output tokens for reasoning. A custom speculative decoding method, deployed with Fireworks, reaches approximately 1,000 tokens per second on a 70b model, about 13x faster than vanilla Llama-3-70b inference and 9x faster than the previous GPT-4 deployment.


### [Picture This: Open Source AI for Image Description](https://yomu.fyi/post/picture-this-open-source-ai-for-image-description.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 9, 2024
- Type: Tutorial

Nolan Darilek presents a self-hosted image-description service intended to help blind screen-reader users understand photos, diagrams, menus, and other visual material. Humans often provide inadequate alt text, and the post contrasts that history with increasingly capable LLM-generated descriptions from apps and browser add-ons. The implementation combines Ollama, running the Apache-licensed LLaVA 34B multimodal model, with a PocketBase API over SQLite and a small Python client; collection hooks send images and follow-up questions, retain chat history, and enforce per-user access rules. In testing, a cold request on an A100-40GB Fly Machine took about 45 seconds, while stopping idle GPU Machines reduced resource use at the cost of startup delay. The example produced useful scene descriptions but could not reliably identify leafless tree species, illustrating both the prototype’s value and its limitations.


### [NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More](https://yomu.fyi/post/neurips-2023-our-favorite-papers-on-llms-statistical-learning-and-more.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Mar 21, 2024
- Type: Explainer

Researchers reviewed prominent machine learning papers presented at NeurIPS 2023 covering large language models and statistical learning theory. One investigation showed that claimed emergent abilities in models such as GPT-3 often result from nonlinear evaluation metrics rather than fundamental shifts in model capability. To reduce the computational burden of model adaptation, QLoRA enables 65-billion-parameter model fine-tuning on a single 48-gigabyte GPU via 4-bit NormalFloat quantization and paged optimization. Direct Preference Optimization eliminates complex reward modeling by casting reinforcement learning from human feedback into a preference classification task. Additional work resolved statistical anomalies like double descent using effective parameter counts and introduced stochastic gradient approximations for Gaussian processes.


### [Cutting Through the Noise: Three Things We've Learned About Generative AI and Developer Productivity](https://yomu.fyi/post/cutting-through-the-noise-three-things-we-ve-learned-about-generative.md)
- Company: [Ebay](https://yomu.fyi/company/ebay.md)
- Author: Senthil Padmanabhan
- Published: Feb 29, 2024
- Type: Explainer

eBay describes three tracks for generative AI developer productivity: adopting a commercial offering, fine-tuning an open-source LLM on internal data, and building a knowledge base. Because no single metric is reliable, the exercise combines surveys with Git, DORA, Flow, code-quality, and performance measures. In a 300-developer Copilot A/B test, the treatment group had 27% code acceptance, 17% faster pull-request creation-to-merge time, and 12% lower Lead Time for Change; Sonar quality was unchanged. eBayCoder, based on Code Llama 13B and trained on eBay code and documentation, is presented as useful for software upkeep and broader context. Its internal GPT uses retrieval-augmented generation to answer questions from distributed sources, but early results remain preliminary and nonsensical answers require feedback and RLHF.


### [How Yoko Li makes towns, tamagoes, and tools for local AI](https://yomu.fyi/post/how-yoko-li-makes-towns-tamagoes-and-tools-for-local-ai.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jan 8, 2024
- Type: Explainer

Yoko Li’s local AI work spans playful agent experiments and practical tools for running language-model workflows on personal hardware. AI Town uses conversations among AI agents to produce changing relationships and emergent behavior, while AI Tamago feeds game state and user inputs to an unmodified LLaMA 2 7B model and uses Ollama’s JSON mode to generate pet actions. The Local AI Starter Kit packages document ingestion, vector storage, and language-model responses into a private, local “chat with a knowledge base” pipeline, avoiding setup across at least four API providers. The post also describes unreliable JSON generation, hallucinations, and grammar-guided workarounds, concluding that randomness is a liability for factual outputs but useful for game behavior and creative experimentation.


### [Scaling Large Language Models to zero with Ollama](https://yomu.fyi/post/scaling-large-language-models-to-zero-with-ollama.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 6, 2023
- Type: Tutorial

Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows.


### [Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling](https://yomu.fyi/post/let-ai-entertain-you-increasing-user-engagement-with-generative-ai-and.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Jaewon Yang
- Published: Oct 16, 2023
- Type: Problem & solution

Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.


[Newer posts](https://yomu.fyi/topic/llm/page/31.md) · [Older posts](https://yomu.fyi/topic/llm/page/33.md)