# Enabling conversational data discovery with LLMs at Grab

[Grab](https://yomu.fyi/company/grab) · Shreyas Parbat · Sep 26, 2024

**Type:** Problem & solution

## Summary

Grab faced major bottlenecks in dataset discovery across its 200,000 data lake tables due to low documentation coverage and vanilla Elasticsearch limitations in its internal tool, Hubble. To resolve this, engineers first tuned Elasticsearch by boosting frequently queried P80 tables and certified schemas while deboosting deprecated datasets. They then deployed a GPT-4 pipeline that generated documentation from table schemas and sample data, raising P80 documentation coverage from 20% to roughly 90%. Finally, the team developed HubbleIQ, an LLM-powered assistant built via Glean Apps and integrated into Slack and Hubble search to answer semantic queries. These initiatives increased search click-through rates to 94% and lifted the proportion of users finding discovery easy to 73%.

## Context

At Grab, data consumers struggled to find relevant datasets across more than 200,000 data lake tables because the internal tool, Hubble, relied on keyword-based Elasticsearch that lacked semantic understanding. Additionally, only 20% of the most frequently queried tables had documentation, forcing 51% of data consumers to spend multiple days locating datasets via Slack.

## Approach / What changed

Grab tuned Elasticsearch by boosting high-value schemas and certified tables while deboosting deprecated assets, automated documentation creation using a GPT-4 pipeline over table schemas and sample data, and built HubbleIQ—an LLM bot via Glean Apps integrated into both Hubble search and Slack channels.

## Takeaways

- Tuning Elasticsearch parameters by boosting certified, frequently queried P80 tables and deboosting deprecated assets raised search click-through rates from 82% to 94%.
- Generating automated documentation with GPT-4 using table schemas and sample data increased documentation coverage for P80 tables from 20% to approximately 90%.
- Deploying the LLM-powered HubbleIQ assistant to enterprise search and Slack ask channels allowed data consumers to resolve contextual, semantic queries without manual intervention from data producers.

**Tags:** [Developer Experience](https://yomu.fyi/topic/developer-experience), [Elasticsearch](https://yomu.fyi/topic/elasticsearch), [LLMs](https://yomu.fyi/topic/llm), [Search](https://yomu.fyi/topic/search)

[Read original post](https://engineering.grab.com/hubble-data-discovery)