hiTenTen: Corpus of the Hindi Web

The Hindi Web Corpus (hiTenTen) is a Hindi corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family which is a set of web corpora built using the same method with a target size 10+ billion words. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.

For detailed information about TenTen corpora, see  Common TenTen corpora attributes.

The current version of hiTenTen Hindi corpus 2021 consists of 790 million words. The texts were downloaded between March and April 2021 from the web and in December 2020 from Hindi Wikipedia.

Part-of-speech tagsets in hiTenTen corpus

This Hindi Web corpus was part-of-speech tagged by Siva Reddy’s tagger using the IIIT tagset for Indian languages.

Overview of Hindi TenTen corpora

These web corpora were crawled and processed repeatedly during the years:

  • Hindi Web corpus 2021 (hiTenTen21) – 790 million words
  • Hindi Web corpus 2017 (hiTenTen17) – 1.2 billion words
  • Hindi Web corpus 2013 (hiTenTen13) – 350 million words

Tools to work with the Hindi corpus

A complete set of Sketch Engine tools is available to work with this Hindi corpus to generate:

  • word sketch – Hindi collocations categorized by grammatical relations
  • thesaurus – synonyms and similar words for every word
  • keywords – terminology extraction of one-word units
  • word lists – lists of Hindi words organized by frequency
  • n-grams – frequency list of multi-word units
  • concordance – examples in context
  • text type analysis – statistics of metadata in the corpus

hiTenTen21 (April 2021)

  • initial version – 1.907 billion tokens (901 million tokens after deduplication)
  • part-of-speech tagged and lemmatized by Siva Reddy’s tagger using the IIIT tagset
  • basic word sketches available
  • crawled by SpiderLing in March and April 2021 & Hindi Wikipedia 2020

hiTenTen17 (March 2019)

  • 1.375 billion tokens
  • part-of-speech tagged and lemmatized by Siva Reddy’s tagger using the IIIT tagset
  • basic word sketches available
  • crawled by SpiderLing between June and September 2017

hiTenTen13 (4 December 2013)

  • initial version – 405 million tokens
  • no tagging and lemmatization
  • crawled by SpiderLing in November 2013

TenTen corpora

Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).

Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).

Search the Hindi corpus

Sketch Engine offers a range of tools to work with this Hindi corpus.

Other text corpora

Sketch Engine offers 800+ corpora.

Use Sketch Engine in minutes

Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.