Loading post…
EbaySenthil PadmanabhanExplainer
Cutting Through the Noise: Three Things We've Learned About Generative AI and Developer Productivity
Summary
eBay describes three tracks for generative AI developer productivity: adopting a commercial offering, fine-tuning an open-source LLM on internal data, and building a knowledge base. Because no single metric is reliable, the exercise combines surveys with Git, DORA, Flow, code-quality, and performance measures. In a 300-developer Copilot A/B test, the treatment group had 27% code acceptance, 17% faster pull-request creation-to-merge time, and 12% lower Lead Time for Change; Sonar quality was unchanged. eBayCoder, based on Code Llama 13B and trained on eBay code and documentation, is presented as useful for software upkeep and broader context. Its internal GPT uses retrieval-augmented generation to answer questions from distributed sources, but early results remain preliminary and nonsensical answers require feedback and RLHF.
Context
Developer productivity is complex and has no reliable single metric. The work also addresses large-scale software upkeep, limited context in commercial LLM offerings, and the difficulty of finding information distributed across enterprise GitHub Markdowns, Google Docs, Jira, Slack, and Wikis.
Approach / What changed
The work evaluates three tracks: expanding GitHub Copilot, post-training and fine-tuning Code Llama 13B on eBay’s codebase and documentation, and operating an internal GPT with retrieval-augmented generation. The knowledge system creates embeddings for ingested content, retrieves similar context with vector similarity, and sends the query and context to private commercial and open-source LLMs.
Takeaways
The Copilot experiment involved 300 developers split between a treatment group and a comparable control group; the Copilot group had a 17% decrease in pull-request creation-to-merge time and a 12% decrease in Lead Time for Change.
eBayCoder was built from Code Llama 13B and eBay’s codebase and documentation, with proposed uses including software upkeep and reducing duplication through access to broader internal context.
The internal GPT retrieves context from enterprise sources using embeddings and vector similarity, but it can still produce nonsensical answers; employee feedback is incorporated through RLHF.
Source and freshness
- Source
- innovation.ebayinc.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.