msTenTen: Corpus of the Malay Web
The Malay Web Corpus (msTenTen) is a Malay corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.
For detailed information about TenTen corpora, see Common TenTen corpora attributes.
The msTenTen Malay corpus consists of 296 million words. The texts were downloaded between June to August 2020 and the Wikipedia part in October 2020. The sample texts of the biggest web domains which account for 82% of all corpus texts were checked manually and content with poor quality text and spam was removed.
Part-of-speech tagset
The msTenTen Malay corpus has not been tagged and lemmatized yet.
Basic information about the msTenTen corpus
| Frequency | |
| Tokens | 352,076,417 |
| Words | 296,419,465 |
| Sentences | 19,608,079 |
| Web pages | 1,391,566 |
Search the Malay corpus msTenTen
Sketch Engine offers a range of tools to work with this Malay corpus.
Tools to work with the Malay corpus from the web
A complete set of Sketch Engine tools is available to work with this Malay corpus to generate:
- keywords – terminology extraction of one-word
- word lists – lists of words organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
version 1
- initial size of 1+ billion tokens
- manual checking of the samples from the biggest web domains, content with poor linguistic quality was removed
- final size of 352 million words
Bibliography
TenTen corpora
SUCHOMEL, Vít. Better Web Corpora For Corpus Linguistics And NLP. 2020. Available also from: https://is.muni.cz/th/u4rmz/. Doctoral thesis. Masaryk University, Faculty of Informatics, Brno. Supervised by Pavel RYCHLÝ.
Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).
Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.
