GigaLekh: A Large Hindi Text Corpus with Educational and Toxicity Annotations

Dataset Summary

This repository contains a large corpus of Hindi text, which has been filtered and annotated using classifiers for educational content and toxicity. The dataset is intended for training language models and other NLP applications in Hindi.

Supported Tasks and Leaderboards

This dataset can be utilized for tasks involving language modeling.

Languages

Hindi.

Dataset Structure

Data Instances

The dataset consists of the following features:

• text: a string of text in Hindi.
• source: the source where that string originated.
• subset: a short string indicating the name of the subset (referring to the original dataset or crawl).
• id: a unique identifier for each sample (md5 hash).
• token_count: number of tokens in the text (Polygl0t/LilMoo tokenizer).
• edu_score: a float representing the educational quality score predicted by the Polygl0t/hindi-roberta-edu-classifier (1 = non-educational, 5 = highly educational).
• edu_int_score: an integer-rounded version of the edu_score.
• toxic_score: a float representing the toxicity score predicted by the Polygl0t/hindi-roberta-toxicity-classifier (1 = non-toxic, 5 = highly toxic).
• toxic_int_score: an integer-rounded version of the toxic_score.

Data Fields

{
  "text": "चाणक्य की गिनती भारत के श्रेष्ठ विद्वानों में की जाती है|चाणक्य को राजनीति शास्त्र, समाज शास्त्र, कूटनीति शास्त्र, सैन्य शास्त्र के साथ अर्थशास्त्र का भी गहरा ज्ञान था|",
  "source": "https://huggingface.co/datasets/HuggingFaceFW/fineweb-2",
  "subset": "finweb_2_hi",
  "id": "dc0ec0a28936637fd9ec5be2b3994eb8",
  "token_count": 410,
  "edu_score": 3.87568,
  "edu_int_score": 4,
  "toxic_score": 2.047186,
  "toxic_int_score": 2
}

Subsets and Splits

After applying the complete filtering pipeline—including text extraction, language identification, heuristic quality filtering, deduplication, and learned filtering using our trained classifiers, we further (1) removed documents shorter than 50 tokens, and (2) removed documents with a toxicity score > 3. Given that toxicity is a particularly challenging issue in Hindi web data, we opted for preserving the filtered out documents as a separate subset of our corpus, which can be useful for future research on toxicity detection and mitigation in Hindi NLP.

from datasets import load_dataset

# Load the main dataset
ds = load_dataset("Polygl0t/hindi-corpus", "default", split="train")

# Load the excluded subset
excluded_ds = load_dataset("Polygl0t/hindi-corpus", "excluded", split="train")

# If you don't want to download the entire dataset, set streaming to `True`
ds = load_dataset("Polygl0t/hindi-corpus", "default", split="train", streaming=True)

Statistics

Dataset Creation

Curation Rationale

To curate this dataset, we developed an implementaton of the FineWeb2 filtering pipeline, while also adding a learned-filter (LLM-as-a-Judge) approach based on works like FineWeb-Edu, the original Phi paper, and other works that document a similar approach.

Text Extraction and Language Identification

For web-crawled data sourced from CC WARC files, we begin by extracting text content using the Trafilatura library, configured to favor precision over recall. For datasets sourced from Hugging Face, which often include pre-cleaned text, this extraction step is bypassed, allowing direct input into the subsequent filtering stages.

To ensure relevance and a minimal level of filtering at this early stage, we apply an initial URL Filter to remove documents from undesirable sources, leveraging blocklists (Maravento's blackweb) to exclude low-quality or inappropriate websites. This is followed by a language identification step using the FastText FT176 model. To enhance language identification accuracy, we employ a second round of language identification using GlotLID as backend.

Quality Filtering and Formatting

For the purpose of building a robust Hindi corpus, we define quality based on several heuristic criteria that reflect linguistic coherence and structural integrity. In short, we import the filters developed during the creation of FineWeb-2 and MassiveText, tunning them to be more sensitive to Hindi's linguistic characteristics. Post-filtering, we apply formatting steps to corrects encoding issues, remove personally identifiable information, and eliminate/replace undesirable patterns (e.g., excessive symbols). For all of this, we used the standard implementation from the Datatrove library to implement these filters.

Deduplication

Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. Hence, to address redundancy, we implement a deduplication pipeline using the MinHash algorithm. Following FineWeb-2 implementation, we use 14 buckets, 8 hashes per bucket, and 5-grams, employing the xxHash function for hashing.

Learned Filtering with LLM-as-a-Judge

Learned filters, often based on large language models, can evaluate text quality more holistically, considering context, coherence, and other nuanced factors that heuristic methods might overlook. Hence, we implemented an LLM-as-a-Judge filtering approach to further enrich our dataset with annotations. Specifically, we used Qwen/Qwen2.5-32B-Instruct to evaluate the quality of documents that passed through the initial heuristic filters. This annotation process involved prompting the model to assess documents based on two criteria: educational quality and toxicity. The result was the following datasets:

• Polygl0t/hindi-edu-qwen-annotations
• Polygl0t/hindi-toxicity-qwen-annotations

With these annotations, we trained two separate classification models to automate the filtering process for our corpus curation:

• Polygl0t/hindi-roberta-edu-classifier (1 = non-educational, 5 = highly educational)
• Polygl0t/hindi-roberta-toxicity-classifier (1 = non-toxic, 5 = highly toxic)

The columns edu_score and toxic_score in this dataset correspond to the predicted scores from these classifiers, while edu_int_score and toxic_int_score are the integer-rounded versions of these scores.

Source Data

We sourced data from a variety of existing datasets available on Hugging Face. These datasets offer a mix of curated and semi-curated content, providing a solid foundation for our corpus to start (see the table below). Unlike web crawls, these datasets often come with some level of quality control, allowing us to bypass some of the initial cleaning steps (e.g., cleaning of HTML tags). However, we also recognized the need to supplement these datasets with more extensive web-crawled data to ensure sufficient coverage and diversity, especially when it comes to adding timely data from recent snapshots of CommonCrawl.

• Cutoff Date: December 2025

Who are the source language producers?

All text samples are native to Hindi or translated from other languages to Hindi (slight contamination of different languages should also be expected).

Annotations

Annotation process

Annotations were created using a learned-filtering approach based on large language models (LLM-as-a-Judge). Specifically, we used Qwen/Qwen2.5-32B-Instruct to evaluate the quality of documents that passed through the initial heuristic filters. This annotation process involved prompting the model to assess documents based on two criteria: educational quality and toxicity. With these annotations, we trained two separate classification models to automate the filtering process for our corpus curation.

⭐ GigaLekh Ablations: The Impact of 60B Tokens of Educational Data ⭐

We have conducted a small-scale ablation study to analyze the impact of our filtering pipeline on the performance of a language models trained on GigaLekh. To conduct this ablation we trained two models, GigaLekh-ablation-EDU-1.5B and GigaLekh-ablation-NonEDU-1.5B. Both models were trained with ~60 billion tokens from GigaLekh. GigaLekh-ablation-EDU-1.5B was trained on the educational portion of GigaLekh (i.e., samples with an Edu Score >= 3) and GigaLekh-ablation-NonEDU-1.5B was trained on non-educational part (i.e., samples with an Edu Score < 3). GigaLekh-Edu outperforms GigaLekh-NonEdu on 5 of 6 benchmarks. These results suggest that training on educationally curated content consistently yields stronger language understanding.

Both models their checkpoints and benchmarks results can be found in their respective repositories: GigaLekh-ablation-EDU-1.5B and GigaLekh-ablation-NonEDU-1.5B.

Personal and Sensitive Information

This dataset was filtered to remove personally identifiable information (PII) using standard PII detection methods as part of the formatting step in the filtering pipeline. Moreover, users should be aware that the excluded subset contains documents that were filtered out due to high toxicity scores, which may include offensive or sensitive content.

Considerations for Using the Data

Social Impact of Dataset

The creation of a large Hindi text corpus has the potential to significantly advance NLP research and applications for Hindi speakers, who represent a substantial portion of the global population. By providing high-quality training data, this dataset can facilitate the development of more accurate and effective language models, which can be used in various applications such as machine translation, sentiment analysis, and information retrieval.

Additional Information

Dataset Maintainers

• Shiza Fatimah.
• Nicholas Kluge Corrêa.
• Aniket Sen.

Licensing Information

Please refer to the individual licenses of the source datasets used to create this corpus, as listed in the "Source Data" section above. The combined dataset does not have a single unified license, and users should ensure compliance with the terms of each source dataset when utilizing this corpus.

Citation Information

@misc{shiza2026lilmoo,
      title={{Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi}}, 
      author={Shiza Fatimah and Aniket Sen and Sophia Falk and Florian Mai and Lucie Flek and Nicholas Kluge Corr{\^e}a},
      year={2026},
      eprint={2603.03508},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.03508}, 
}

Acknowledgments

Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.

We also gratefully acknowledge the granted access to the Marvin cluster hosted by University of Bonn along with the support provided by its High Performance Computing & Analytics Lab.

Contributions

If you want to contribute, contact us at polyglot@uni-bonn.de!