# SyGra: The One-Stop Framework for Building Data for LLMs and SLMs

[Hugging Face](https://yomu.fyi/company/hugging-face) · Bidyapati Pradhan, Vipul Mittal, Amit Kumar Saha, Surajit Dasgupta · Sep 22, 2025

**Type:** Problem & solution

## Summary

Training and aligning large language models and small language models requires datasets tailored for complex reasoning, preference tuning, multi-turn questioning, and format conversion. To eliminate the need for bespoke data preparation scripts, SyGra provides a low-code and no-code Python framework for automated dataset creation, transformation, and alignment. The library integrates with diverse inference backends, including vLLM, Hugging Face TGI, Triton, and Ollama, allowing teams to focus on prompt engineering rather than pipeline infrastructure. It supports data workflows ranging from converting PDFs and knowledge bases into question-answering pairs to filtering low-quality samples and generating Direct Preference Optimization datasets. These plug-and-play workflows reduce manual curation effort while accelerating model fine-tuning and retrieval-augmented generation pipelines.

## Context

Training and aligning LLMs and SLMs requires structured, high-quality, and task-specific datasets, such as reasoning chains, Direct Preference Optimization pairs, and multi-turn questions. Raw or open data rarely matches required formats, forcing practitioners to manually filter samples, expand contexts for RAG, translate across languages, or extract structured Q&A data from PDFs and knowledge bases.

## Approach / What changed

SyGra is a low-code/no-code Python library and framework that automates dataset creation, transformation, quality filtering, and alignment. It delegates heavy pipeline engineering so users can focus on prompt engineering, offering native support for inference backends like vLLM, Hugging Face TGI, Triton, and Ollama.

## Takeaways

- SyGra provides a low-code and no-code Python framework that lets developers build complex datasets by focusing on prompt engineering rather than custom pipeline scripts.
- The framework natively integrates with multiple inference backends, including vLLM, Hugging Face TGI, Triton, and Ollama.
- Supported dataset tasks include turning knowledge bases or PDFs into Q&A formats, generating DPO preference pairs from SFT data, quality filtering, and expanding contexts for RAG pipelines.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/ServiceNow-AI/sygra-data-gen-framework)