ptTenTen: Corpus of the Portuguese Web
The Portuguese Web Corpus (ptTenTen) is a Portuguese language corpus made up of texts collected from the Internet. It belongs to the TenTen corpus family which is a set of web corpora built using the same method with a target size 10+ billion words. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.
The Portuguese Web 2023 corpus contains almost 17 billion words (downloaded in August 2020, December 2020 – February 2021, August 2023 – November 2023) and includes both main Portuguese language varieties – European and Brazilian Portuguese.
Detailed information about TenTen corpora is on the separate page Common TenTen corpora attributes.
Part-of-speech tagset
The Portuguese Web corpora are annotated by the FreeLing tool using the following part-of-speech tagset schema.
Overview of Portuguese corpora from the web
Sketch Engine offers several Portuguese TenTen corpora:
- Portuguese Web 2023 (ptTenTen23) – tagged using FreeLing pipeline version 6, genre annotation and topic classification
- Portuguese Web 2020 (ptTenTen20) – tagged using FreeLing pipeline version 5, new term grammar, genre annotation and topic classification
- Portuguese Web 2018 (ptTenTen18) – tagged using FreeLing pipeline version 4, genre annotation and topic classification
- Portuguese Web 2011 (ptTenTen11) – tagged using FreeLing pipeline version 4 with orthography normalization + new word sketches by Tanara Zingano Kuhn
- automatical detection of words in Brazilian Portuguese
- attribute morpheme containing clitics, e.g. apresentar, se, nos for the word apresenta-se-nos
- attribute ao containing orthographic normalization, e.g. ação for acção (acc. the Portuguese Language Orthographic Agreement)
- lemma always corresponds to the new Portuguese orthography (2016)
- Portuguese Web 2011 (ptTenTen11, Freeling v3) – version processed with the FreeLing tagger using the following POS tags
- Portuguese Web 2011 (ptTenTen11, Palavras parsed) – version processed with Eckhard Bick’s PALAVRAS parser, post-processed to optimise word sketch output by Pete Whitelock and tagged with the VISL part-of-speech tagset
Genre annotation and topic classification
A part of the Portuguese Web 2023 corpus contains genre annotation and topic classification. These can be displayed as corpus structures in Concordance or in the Text type Analysis tool. Genres refer to writing styles and are divided into four groups (blog, discussion, fiction, legal, news, reference/encyclopedia) whereas topic classification is inspired by categories used by https://curlie.org/ (formerly dmoz.org).
- genres cover 26.75% of the corpus, i.e. 5.3 billion tokens
- topic classification covers 13% of the corpus, i.e. 2.6 billion tokens
Please refer to our article on topic and genre classification for more information: https://www.sketchengine.eu/blog/topics-and-genres-in-corpora/
Hover over the chart to display a number of tokens of the particular topic.
Basic frequency statistics of the Portuguese Web 2023 corpus
| Tokens | 20,044,773,021 |
| Words | 16,976,742,883 |
| Sentences | 876,729,377 |
| Web pages | 52,090,369 |
Tools to work with the Portuguese corpus
A complete set of Sketch Engine tools is available to work with this Portuguese corpus to generate:
- word sketch – Portuguese collocations categorized by grammatical relations
- thesaurus – synonyms and similar words for every word
- keywords – terminology extraction of one-word and multi-word units
- word lists – lists of Portuguese nouns, verbs, adjectives etc. organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
September 2024 – ptTenTen23
- FreeLing pipeline version 6
- crawled in in August 2020, December 2020 – February 2021, August 2023 – November 2023
- spam removal
- topic & genre classification
May 2023 – ptTenTen20
- FreeLing pipeline version 5
- 14.8 billion tokens
- crawled by Spiderling from June to July 2020
November 2022 – ptTenTen18
- FreeLing pipeline version 4
- 8.7 billion tokens
- crawled by Spiderling from February to April 2018
February 2018
- tagged with FreeLing pipeline v4 with orthography normalization
- improved tokenization 3,896,392,719 words
- new word sketch grammar by Tanara Zingano Kuhn
19th December 2013
- tagged by Freeling
- attribute lempos added
August 2012
- finished version – 3.2 billion tokens processed with Palavras parsing system
- crawled by the SpiderLing web spider in March 2011 and August 2012 which yielded almost 4 billion words
24th March 2011
- initial version – 0.9 billion tokens
Bibliography
ptTenTen corpus
Kilgarriff, A., Jakubíček, M., Pomikalek, J., Sardinha, T. B., & Whitelock, P. (2014). PtTenTen: a corpus for Portuguese lexicography. Working with Portuguese Corpora, 111-30.
Word sketches by Tanara Zingano Kuhn
Kuhn, Tanara Zingano, and Iztok Kosem. “Devising a Sketch Grammar for Academic Portuguese.” Slovenščina 2.0: empirical, applied and interdisciplinary research 4.1 (2017): 124-161.
Kuhn, Tanara Zingano (2017). A Design proposal of an on-line corpus-driven dictionary of Portuguese for university students (doctoral dissertation). University of Lisbon, Lisbon, Portugal.
Portuguese word sketches of the PALAVRAS parser
Kilgarriff, A., Pomikalek, J., Jakubíček, M., & Whitelock, P. (2012). Setting up for corpus lexicography. Skin, 1, 1-38.
TenTen corpora
SUCHOMEL, Vít. Better Web Corpora For Corpus Linguistics And NLP. 2020. Available also from: https://is.muni.cz/th/u4rmz/. Doctoral thesis. Masaryk University, Faculty of Informatics, Brno. Supervised by Pavel RYCHLÝ.
Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).
Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).
Genre annotation
SUCHOMEL, Vít. Genre Annotation of Web Corpora: Scheme and Issues. In Kohei Arai, Supriya Kapoor, Rahul Bhatia. Proceedings of the Future Technologies Conference (FTC) 2020, Volume 1. Vancouver, Canada: Springer Nature Switzerland AG, 2021. s. 738-754. ISBN 978-3-030-63127-7. doi:10.1007/978-3-030-63128-4_55.
Search the Portuguese corpus
Sketch Engine offers a range of tools to work with this Portuguese corpus.
Positional attributes in ptTenTen
(1) lemma – the basic form of a word, typically the form found in dictionaries.
(2) ao – the orthographic normalization based on the Portuguese Language Orthographic Agreement, e.g. ação for acção
(3) morphemes – clitics, e.g. apresentar,se,nos for apresenta-se-nos
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.
