viTenTen: Vietnamese corpus from the Web
The Vietnamese Web Corpus (viTenTen) is a Vietnamese corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family which is a set of web corpora built using the same method with a target size of 10+ billion words. Sketch Engine currently provides access to TenTen corpora in more than 40 languages.
For detailed information about TenTen corpora, see Common TenTen corpora attributes.
The most recent version of the viTenTen corpus consists of 6 billion words. The texts were downloaded in May–July and November 2017 and January 2018. The sample texts of the biggest web domains which account for 50% of all corpus texts were checked semi-manually and content with poor quality text and spam was removed.
Part-of-speech tagset
This Vietnamese corpus is part-of-speech tagged with the following Vietnamese part-of-speech tagset summary indicating the part of speech and grammatical category.
Basic frequency statistics of the Vietnamese Web 2017 corpus
Tokens | 7+ billion |
Words | 6+ billion |
Sentences | 138+ million |
Web pages | 24+million |
Genre annotation and topic classification
A part of the Vietnamese Web 2017 corpus contains genre annotation and topic classification. These can be displayed as corpus structures in Concordance or in the Text type Analysis tool. Genres refer to writing styles and are divided into four groups (blog, discussion, fiction, legal, news, reference/encyclopedia) whereas topic classification is inspired by categories used by https://curlie.org/ (formerly dmoz.org).
- genres cover 14.6% of the corpus, i.e. 1 billion tokens
- topic classification covers 2.8% of the corpus, i.e. 200 million tokens
Please refer to our article on topic and genre classification for more information: https://www.sketchengine.eu/blog/topics-and-genres-in-corpora/
Hover over the chart to display a number of tokens of the particular topic.
Tools to work with the Vietnamese corpus
A complete set of tools is available to work with this Vietnamese corpus to generate:
- word sketch – Vietnamese collocations categorized by grammatical relations
- thesaurus – synonyms and similar words for every word
- keywords – terminology extraction of one-word and multi-word units
- word lists – lists of Vietnamese nouns, verbs, adjectives etc. organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
Vietnamese Web 2017 (August 2024)
- POS tagging
- Word Sketch grammar for collocations, Term grammar for terminology extraction
- topic and genre classification
- spam and bad content removal
Vietnamese Web 2017 (November 2018)
- crawled by SpiderLing in November and December 2017 and in January 2018
- not tagged yet
Bibliography
TenTen corpora
Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).
Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).
Genre annotation
SUCHOMEL, Vít. Genre Annotation of Web Corpora: Scheme and Issues. In Kohei Arai, Supriya Kapoor, Rahul Bhatia. Proceedings of the Future Technologies Conference (FTC) 2020, Volume 1. Vancouver, Canada: Springer Nature Switzerland AG, 2021. s. 738-754. ISBN 978-3-030-63127-7. doi:10.1007/978-3-030-63128-4_55.
Search the Vietnamese corpus
Sketch Engine offers a range of tools to work with this Vietnamese corpus from the web.
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.