hiTenTen: Corpus of the Hindi Web
The Hindi Web Corpus (hiTenTen) is a Hindi corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family which is a set of web corpora built using the same method with a target size 10+ billion words. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.
For detailed information about TenTen corpora, see Common TenTen corpora attributes.
The current version of hiTenTen Hindi corpus 2021 consists of 790 million words. The texts were downloaded between March and April 2021 from the web and in December 2020 from Hindi Wikipedia.
Part-of-speech tagsets in hiTenTen corpus
This Hindi Web corpus was part-of-speech tagged by Siva Reddy’s tagger using the IIIT tagset for Indian languages.
Overview of Hindi TenTen corpora
These web corpora were crawled and processed repeatedly during the years:
- Hindi Web corpus 2021 (hiTenTen21) – 790 million words
- Hindi Web corpus 2017 (hiTenTen17) – 1.2 billion words
- Hindi Web corpus 2013 (hiTenTen13) – 350 million words
Tools to work with the Hindi corpus
A complete set of Sketch Engine tools is available to work with this Hindi corpus to generate:
- word sketch – Hindi collocations categorized by grammatical relations
- thesaurus – synonyms and similar words for every word
- keywords – terminology extraction of one-word units
- word lists – lists of Hindi words organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
hiTenTen21 (April 2021)
- initial version – 1.907 billion tokens (901 million tokens after deduplication)
- part-of-speech tagged and lemmatized by Siva Reddy’s tagger using the IIIT tagset
- basic word sketches available
- crawled by SpiderLing in March and April 2021 & Hindi Wikipedia 2020
hiTenTen17 (March 2019)
- 1.375 billion tokens
- part-of-speech tagged and lemmatized by Siva Reddy’s tagger using the IIIT tagset
- basic word sketches available
- crawled by SpiderLing between June and September 2017
hiTenTen13 (4 December 2013)
- initial version – 405 million tokens
- no tagging and lemmatization
- crawled by SpiderLing in November 2013
Bibliography
TenTen corpora
Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).
Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).
Search the Hindi corpus
Sketch Engine offers a range of tools to work with this Hindi corpus.
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.
