bnTenTen: Corpus of the Bengali Web
The Bengali Web Corpus (bnTenTen) is a Bengali corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.
For detailed information about TenTen corpora, see Common TenTen corpora attributes.
The most recent version of the bnTenTen corpus in the language of Bangla consists of 470 million words. The texts were downloaded between March and April 2021. The sample texts of the biggest web domains which account for 88% of all corpus texts were checked manually and content with poor quality text and spam was removed.
Part-of-speech tagset and lemmatization
The Bengali Web corpora are part-of-speech tagged with the following Bengali part-of-speech tagset indicating the part of speech and grammatical category. The corpus texts also contains lemmatization when each word form from the corpus is assigned to its base form (lemma).
Search the Bengali corpus bnTenTen
Sketch Engine offers a range of tools to work with this Bengali corpus.
Overview of Bengali TenTen corpora
These web corpora were crawled and processed repeatedly during the years:
- Bengali Web corpus 2021 (bnTenTen21) – 470 million words
Further information about texts in the bnTenTen21 corpus
Basic information
Frequency | |
Tokens | 541,618,092 |
Words | 470,732,738 |
Sentences | 40,009,892 |
Web pages | 1,736,893 |
Preloaded subcorpora
A list of preloaded subcorpora (based on the top-level domains or genre annotation and topic classification) that are available in the Bengali Web 2021 corpus.
Subcorpus | Tokens | % of the whole corpus |
Bengali domain .bd | 44,175,497 | 8.2 |
Bengali Wikipedia | 9,509,708 | 1.756 |
Indian domain .in | 2,694,674 | 0.498 |
Tools to work with the Bengali corpora from the web
A complete set of Sketch Engine tools is available to work with these Bengali corpora to generate:
- word sketch – Bengali collocations categorized by grammatical relations
- thesaurus – synonyms and similar words for every word
- keywords – terminology extraction of one-word units
- word lists – lists of Bengali nouns, verbs, adjectives etc. organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
Bengali Web 2021 (bnTenTen21)
version bntenten21_ba (July 2023)
- 470 million words
- further cleaning and spam removing
Bibliography
TenTen corpora
SUCHOMEL, Vít. Better Web Corpora For Corpus Linguistics And NLP. 2020. Available also from: https://is.muni.cz/th/u4rmz/. Doctoral thesis. Masaryk University, Faculty of Informatics, Brno. Supervised by Pavel RYCHLÝ.
Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).
Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.