arTenTen: Corpus of the Arabic Web
The Arabic Web Corpus (arTenTen) is a language corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family which is a set of the web corpora built using the same method with a target size 10+ billion words. Sketch Engine currently provides access to TenTen corpora in more than 40 languages.
Detailed information about TenTen corpora is on the separate page Common TenTen corpora attributes.
The most recent version of the arTenTen corpus consists of 6.5 billion words. The texts were downloaded December 2020 to March 2021 and from November 2023 to January 2024. The sample texts of the largest web domains which account for 51% of all corpus texts were checked semi-manually and content with poor quality text and spam was removed.
Part-of-speech tagset
The Arabic Web corpus 2024 is part-of-speech tagged with the POS CAMeL tagset indicating the part of speech and grammatical category. The corpus texts also contain lemmatization when each word form from the corpus is assigned to its base form (lemma). Both level of annotation is created by the CAMeL tools.
The Arabic Web 2012 corpus is only part-of-speech tagged by the Standford Arabic parser with the following POS tagset summary.
Arabic Web 2024 corpus sizes
The chart shows the distribution of the parts of speech in the Arabic Web corpus 2024.
| Tokens | 7.5+ billion |
| Words | 6.5+ billion |
| Sentences | 300+ million |
| Web pages | 21+ million |
Search the Arabic corpus
Sketch Engine offers a range of tools to work with this Arabic corpus.
Arabic Web 2024 – Genre annotation and topic classification
A part of the Arabic Web 2024 corpus contains genre annotation and topic classification. These can be displayed as corpus structures in Concordance or in the Text type Analysis tool. Genres refer to writing styles and are divided into four groups (blog, discussion, legal, news) whereas topic classification is inspired by categories used by https://curlie.org/ (formerly dmoz.org) and includes the following topics: religion, beauty & fashion, travel & tourism, economy & finance & business, sports, cars & bikes, education, culture & entertainment, history.
- genres cover 30.59% of the corpus, i.e. 2.3 billion tokens
- topic classification covers 5.29% of the corpus, i.e. 400 million tokens
Hover over the chart to display a number of tokens of the particular genre.
Overview of Arabic TenTen corpora
This is a list of Arabic Web corpora available in Sketch Engine:
- Arabic Web corpus 2024 (arTenTen24) – 6.5 billion words, PoS tagging and lemmatization, genre annotation and topic classification, processed by the set of CAMeL tools
- Arabic Web corpus 2018 (arTenTen18) – 4.6 billion words, PoS tagging and lemmatization, genre annotation and topic classification, processed by the set of CAMeL tools
- Arabic Web corpus 2012 (arTenTen12) – 7.4 billion words, PoS tagging only, processed by Stanford tagger
- Arabic Web 2012 sample (arTenTen12) – 115-million-word corpus processed by Madamira tagger
Tools to work with the Arabic corpora from the web
A complete set of Sketch Engine tools is available to work with these Arabic corpora to generate:
- word sketch – Arabic collocations categorized by grammatical relations
- thesaurus – synonyms and similar words for every word
- keywords – terminology extraction of one-word and multi-word units
- word lists – lists of Arabic nouns, verbs, adjectives etc. organized by frequency
- n-grams – frequency list of multi-word units
- concordance – examples in context
- text type analysis – statistics of metadata in the corpus
Changelog
Arabic Web 2024 (April 2024)
- initial size – 6.5 billion words
- processed by CAMeL tools (part-of-speech tagging and lemmatization)
- advanced cleaning and spam removing
- genre annotation and topic classification
Arabic Web 2018 (February 2018)
- initial size – 4.6 billion words
- processed by CAMeL tools (part-of-speech tagging and lemmatization)
- advanced cleaning and spam removing
- genre annotation and topic classification
Stanford version (August 2015)
- tokenized and tagged with Stanford NLP Tools
word sketch version (fall 2014)
- created and implemented word sketches for Arabic
initial version (April 2012)
- initial size – 5.8 billion words
- part-of-speech (POS) tagged and lemmatized with the MADA tool
We have also created ‘word sketches’: one-page, automatic, corpus-derived summaries of a
word’s grammatical and collocational behavior. We use examples to demonstrate what the corpus can
show us regarding Arabic words and phrases and how this can support lexicography and inform
linguistic research.
Bibliography
TenTen corpora
SUCHOMEL, Vít. Better Web Corpora For Corpus Linguistics And NLP. 2020. Available also from: https://is.muni.cz/th/u4rmz/. Doctoral thesis. Masaryk University, Faculty of Informatics, Brno. Supervised by Pavel RYCHLÝ.
Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).
Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).
Arts, T., Belinkov, Y., Habash, N., Kilgarriff, A., & Suchomel, V. (2014). arTenTen: Arabic Corpus and Word Sketches. Journal of King Saud University-Computer and Information Sciences, 26(4), 357-371.
Belinkov, Y., Habash, N., Kilgarriff, A., Ordan, N., Roth, R., & Suchomel, V. (2013). arTenTen: a new, vast corpus for Arabic. Proceedings of WACL.
Genre annotation
SUCHOMEL, Vít. Genre Annotation of Web Corpora: Scheme and Issues. In Kohei Arai, Supriya Kapoor, Rahul Bhatia. Proceedings of the Future Technologies Conference (FTC) 2020, Volume 1. Vancouver, Canada: Springer Nature Switzerland AG, 2021. s. 738-754. ISBN 978-3-030-63127-7. doi:10.1007/978-3-030-63128-4_55.
Use Sketch Engine in minutes
Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.
