idTenTen: Corpus of the Indonesian Web

The Indonesian Web Corpus (idTenTen) is an Indonesian corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.

For detailed information about TenTen corpora, see Common TenTen corpora attributes.

The most recent version of the idTenTen corpus consists of 7.1 billion words. The texts were downloaded in June–July 2024, October–December 2023 and in June–August 2020. The sample texts of the largest web domains were checked manually and content with poor quality text and spam was removed.

Part-of-speech tagset and lemmatization

The Indonesian Web corpora are part-of-speech tagged with the following Indonesian TreeTagger tagset indicating the part of speech and grammatical category. The corpus texts also contain lemmatization when each word form from the corpus is assigned to its base form (lemma).

The Indonesian Web corpus 2024 in statistics

Number of words: 7+ billion
Number of tokens: 8+ billion
Number of sentences: 495+ million
Number of documents: 28+ million

Search the Indonesian corpus idTenTen

Sketch Engine offers a range of tools to work with this Indonesian corpus.

open in Sketch Engine
about Sketch Engine

Overview of Indonesian TenTen corpora

This is a list of Indonesian Web corpora available in Sketch Engine:

  • Indonesian Web corpus 2024 (idTenTen24) – 7.1 billion words, genre annotation and topic classification
  • Indonesian Web corpus 2020 (idTenTen20) – 3.6 billion words

Genre annotation and topic classification

A part of the Indonesian Web 2024 corpus contains genre annotation and topic classification. These can be displayed as corpus structures in Concordance or in the Text type Analysis tool. Genres refer to writing styles and are divided into four groups (blog, discussion, fiction, legal, news, reference/encyclopedia) whereas topic classification is inspired by categories used by https://curlie.org/ (formerly dmoz.org).

  • genres cover 20.8% of the corpus, i.e. 1.8 billion tokens
  • topic classification covers 5.8% of the corpus, i.e. 500 million tokens

Please refer to our article on topic and genre classification for more information: https://www.sketchengine.eu/blog/topics-and-genres-in-corpora/

Hover over the chart to display a number of tokens of the particular topic.

Tools to work with the Indonesian corpora from the web

A complete set of Sketch Engine tools is available to work with these Indonesian corpora to generate:

  • word sketch – Indonesian collocations categorized by grammatical relations
  • thesaurus – synonyms and similar words for every word
  • keywords – terminology extraction of one-word and multi-word units
  • word lists – lists of Indonesian nouns, verbs, adjectives etc. organized by frequency
  • n-grams – frequency list of multi-word units
  • concordance – examples in context
  • text type analysis – statistics of metadata in the corpus

Indonesian Web 2024 (idTenTen24)

version idtenten24_tt2 (July 2024)

  • 7.1 billion words
  • tagged by TreeTagger
  • language identification was performed using CLD2
  • cleaning: Boilerplate removal, near paragraph de-duplication, manual inspection of large websites

Indonesian Web 2020 (idTenTen20)

version idtenten20 (October 2023)

  • 4.4 billion words
  • TreeTagger pipeline with part-of-speech tagging and lemmatization
  • samples from the biggest web domains were manually checked and content with poor linguistic quality was removed

TenTen corpora

SUCHOMEL, Vít. Better Web Corpora For Corpus Linguistics And NLP. 2020. Available also from: https://is.muni.cz/th/u4rmz/. Doctoral thesis. Masaryk University, Faculty of Informatics, Brno. Supervised by Pavel RYCHLÝ.

Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).

Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).

Other text corpora

Sketch Engine provides access to 800+ language corpora.

available corpora

Use Sketch Engine in minutes

Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.

Quick Start Guide