Opinion Classification with Kili and HuggingFace AutoTrain

Summary

Understanding mobile application user feedback often requires expensive and labor-intensive manual analysis. To streamline this process, an active learning pipeline was constructed using Kili for data annotation alongside HuggingFace AutoTrain and standard transformers for text classification. The workflow utilized approximately 40,000 Medium reviews collected from the Google Play Store, classifying content into categories such as Subscription, Content, Interface, and User Experience. Hyperparameter searches with Ray backend were compared against AutoTrain, which automated training within 30 minutes. Subsequent sentiment analysis of the categorized reviews revealed that while overall sentiment remained positive, subscription and interface categories contained predominantly negative feedback, especially in application version 4.5.

Context

Extracting insights from large volumes of user feedback manually is expensive and labor-intensive for user-facing applications.

Approach / What changed

Using Kili's collaborative annotation platform and Python API, the author labeled Google Play Store reviews for Medium across defined categories, then trained classification models using HuggingFace AutoTrain and standard HuggingFace transformers with Ray hyperparameter optimization before applying sentiment analysis.

Takeaways

• Kili provides APIs and web interfaces to set up multi-class text classification and named entity recognition labeling jobs.
• AutoTrain automated data cleaning, model selection, and hyperparameter optimization, completing model training in 30 minutes.
• Categorized sentiment analysis revealed that most negative reviews concentrated specifically around subscription features and interface issues.

Related reading

• huggingface.co · Nov 2, 2020

  Hyperparameter Search with Transformers and Ray Tune

  Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.

  system

• huggingface.co · Feb 14, 2020

  How to train a new language model from scratch using Transformers and Tokenizers

  Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.