---
title: "Cutting Through the Noise: Three Things We've Learned About Generative AI and Developer Productivity"
description: "eBay describes three tracks for generative AI developer productivity: adopting a commercial offering, fine-tuning an open-source LLM on internal data, and building a knowledge base. Because no single metric is reliable, the exercise combines surveys with Git, DORA, Flow, code-quality, and performance measures. In a 300-developer Copilot A/B test, the treatment group had 27% code acceptance, 17% faster pull-request creation-to-merge time, and 12% lower Lead Time for Change; Sonar quality was unchanged. eBayCoder, based on Code Llama 13B and trained on eBay code and documentation, is presented as useful for software upkeep and broader context. Its internal GPT uses retrieval-augmented generation to answer questions from distributed sources, but early results remain preliminary and nonsensical answers require feedback and RLHF."
---

# Cutting Through the Noise: Three Things We've Learned About Generative AI and Developer Productivity

[Ebay](https://yomu.fyi/company/ebay) · Senthil Padmanabhan · Feb 29, 2024

**Type:** Explainer

## Summary

eBay describes three tracks for generative AI developer productivity: adopting a commercial offering, fine-tuning an open-source LLM on internal data, and building a knowledge base. Because no single metric is reliable, the exercise combines surveys with Git, DORA, Flow, code-quality, and performance measures. In a 300-developer Copilot A/B test, the treatment group had 27% code acceptance, 17% faster pull-request creation-to-merge time, and 12% lower Lead Time for Change; Sonar quality was unchanged. eBayCoder, based on Code Llama 13B and trained on eBay code and documentation, is presented as useful for software upkeep and broader context. Its internal GPT uses retrieval-augmented generation to answer questions from distributed sources, but early results remain preliminary and nonsensical answers require feedback and RLHF.

## Context

Developer productivity is complex and has no reliable single metric. The work also addresses large-scale software upkeep, limited context in commercial LLM offerings, and the difficulty of finding information distributed across enterprise GitHub Markdowns, Google Docs, Jira, Slack, and Wikis.

## Approach / What changed

The work evaluates three tracks: expanding GitHub Copilot, post-training and fine-tuning Code Llama 13B on eBay’s codebase and documentation, and operating an internal GPT with retrieval-augmented generation. The knowledge system creates embeddings for ingested content, retrieves similar context with vector similarity, and sends the query and context to private commercial and open-source LLMs.

## Takeaways

- The Copilot experiment involved 300 developers split between a treatment group and a comparable control group; the Copilot group had a 17% decrease in pull-request creation-to-merge time and a 12% decrease in Lead Time for Change.
- eBayCoder was built from Code Llama 13B and eBay’s codebase and documentation, with proposed uses including software upkeep and reducing duplication through access to broader internal context.
- The internal GPT retrieves context from enterprise sources using embeddings and vector similarity, but it can still produce nonsensical answers; employee feedback is incorporated through RLHF.

**Tags:** [Generative AI](https://yomu.fyi/topic/generative-ai), [GitHub Copilot](https://yomu.fyi/topic/github-copilot), [LLMs](https://yomu.fyi/topic/llm), [Retrieval-Augmented Generation](https://yomu.fyi/topic/retrieval-augmented-generation)

- Source: [Ebay](https://innovation.ebayinc.com/stories/cutting-through-the-noise-three-things-weve-learned-about-generative-ai-and-developer-productivity)
- Source URL: https://innovation.ebayinc.com/stories/cutting-through-the-noise-three-things-weve-learned-about-generative-ai-and-developer-productivity
- Ingested by Yomu: 2026-09-09T16:03:19.340Z

[Read original post](https://innovation.ebayinc.com/stories/cutting-through-the-noise-three-things-weve-learned-about-generative-ai-and-developer-productivity)