Metasense V2: Enhancing, improving and productionisation of LLM powered data governance

Summary

Metasense V2 addresses automated data governance by improving an LLM-powered metadata generation and classification system. It builds on an initial rollout that scanned more than 20,000 data entries at 300–400 entities per day, but retained human verification to catch misclassifications. Post-rollout owner feedback and manual classifications from the Data Governance Office supplied training and testing data, exposing failures involving PII in business email columns, nested JSON, and passenger communications. The revised approach splits PII and non-PII tagging, reduces the PII tag set from 21 to 8, shortens the prompt from 1,254 to 737 words, and divides tables exceeding 150 columns. LangChain and LangSmith support prompt development, evaluation across accuracy, latency, and error rate, deployment, and monitoring, while threshold-based alerts trigger a model-improvement protocol when misclassification rates rise.

Takeaways

1. 1

   The second iteration uses post-rollout table-owner feedback and manual classifications from the Data Governance Office as training and testing datasets.

2. 2

   Tables with more than 150 columns are divided into smaller tables so the LLM can focus on classifying individual columns.

3. 3

   LangSmith evaluations expose prompt performance across accuracy, latency, and error rate, while threshold alerts support ongoing quality control.