---
title: "Introducing the Data Measurements Tool: an Interactive Tool for Looking at Datasets"
description: "Hugging Face released an open-source Python library and no-code interface named the Data Measurements Tool to calculate metrics for responsible dataset curation. In AI development, scraped data often enters training pipelines without detailed analysis of representation or downstream model influence, partly because querying raw datasets typically requires complex coding. Built using Hugging Face Dataset and Spaces Hubs alongside Streamlit, the tool provides descriptive, distributional, and comparison statistics. It computes missing values, vocabulary sizes, duplicate counts, label distributions, and Zipf's law adherence to identify unnatural language patterns or markup artifacts. Additionally, it offers embedding clusters via a Sentence-Transformer model and normalized pointwise mutual information metrics across gender and sexual orientation identity terms."
---

# Introducing the Data Measurements Tool: an Interactive Tool for Looking at Datasets

huggingface.co · Sasha Luccioni, Yacine Jernite, Margaret Mitchell · Nov 29, 2021

**Type:** Announcement

## Summary

Hugging Face released an open-source Python library and no-code interface named the Data Measurements Tool to calculate metrics for responsible dataset curation. In AI development, scraped data often enters training pipelines without detailed analysis of representation or downstream model influence, partly because querying raw datasets typically requires complex coding. Built using Hugging Face Dataset and Spaces Hubs alongside Streamlit, the tool provides descriptive, distributional, and comparison statistics. It computes missing values, vocabulary sizes, duplicate counts, label distributions, and Zipf's law adherence to identify unnatural language patterns or markup artifacts. Additionally, it offers embedding clusters via a Sentence-Transformer model and normalized pointwise mutual information metrics across gender and sexual orientation identity terms.

## Context

AI development often overlooks dataset curation, relying on scraped web data without measuring representation or downstream model impacts, while querying raw data frequently creates bottlenecks for non-technical disciplines.

## Approach / What changed

Hugging Face built the open-source Data Measurements Tool, a Python library and Streamlit-powered interface on Hugging Face Spaces Hub, to compute descriptive, distributional, and comparison metrics for NLP datasets.

## Takeaways

- The tool calculates Zipf's law adherence, where an alpha value greater than 2 or a minimum rank above 10 can signal unnatural language distributions such as HTML artifacts.
- Hierarchical embedding clustering of text fields uses a Sentence-Transformer model paired with a maximum dot product single-linkage criterion to inspect data clusters.
- Normalized pointwise mutual information (nPMI) between word pairs helps identify stereotypes and prejudices across gender and sexual orientation axes.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

- Source: [huggingface.co](https://huggingface.co/blog/data-measurements-tool)
- Source URL: https://huggingface.co/blog/data-measurements-tool
- Ingested by Yomu: 2026-08-27T15:07:33.216Z

[Read original post](https://huggingface.co/blog/data-measurements-tool)