enTenTen: Corpus of the English Web

The English Web Corpus (enTenTen) is an English corpus made up of texts collected from the Internet. The corpus belongs to the TenTen corpus family. Sketch Engine currently provides access to TenTen corpora in more than 40 languages. The corpora are built using technology specialized in collecting only linguistically valuable web content.

For detailed information about TenTen corpora, see Common TenTen corpora attributes.

The most recent version of the enTenTen corpus consists of 21.9 billion words. The texts were downloaded between 2016 and 2018 (80% in 2018). The biggest web domains which account for 70% of all texts were checked manually and content with poor linguistic quality was removed.

Part-of-speech tagset

The enTenTen English corpora were tagged by the TreeTagger tool with the following versions of part-of-speech tagsets:

Overview of English TenTen corpora

These web corpora were crawled and processed repeatedly during the years:

  • English Web corpus 2018 (enTenTen15) – 21.9 billion words
  • English Web corpus 2015 (enTenTen15) – 13 billion words (topic classification)
  • English Web corpus 2013 (enTenTen13) – 19 billion words
  • English Web corpus 2012 (enTenTen12) – 11 billion words
  • English Web corpus 2008 (enTenTen08) – 2.7 billion words

enTenTen18 corpus in detail

The chart shows the distribution of the most frequent top-level domains and selected domains of the states where English is the official language.

Basic information

Frequency
Tokens 25,825,512,637
Words 21,926,740,748
Sentences 1,186,201,144
Web pages 36,785,613

Preloaded subcorpora

A list of preloaded subcorpora (based on the top-level domains) available in the English Web 2018 corpus.

Australian domain .au 699,931,081 tokens 2.71% of the whole corpus
Canadian domain .ca 514,362,319 tokens 1.992% of the whole corpus
EU domain .eu 79,375,718 tokens 0.307% of the whole corpus
Indian domain .in 5,489,170 tokens 0.021% of the whole corpus
Irish domain .ie 76,969,678 tokens 0.298% of the whole corpus
New Zealand domain .nz 138,028,382 tokens 0.534% of the whole corpus
UK domain .uk 1,731,454,108 tokens 6.704% of the whole corpus
US domain .us 240,119,290 tokens 0.93% of the whole corpus

Tools to work with the English Web corpora

A complete set of Sketch Engine tools is available to work with these English corpora to generate:

  • word sketch – English collocations categorized by grammatical relations
  • thesaurus – synonyms and similar words for every word
  • keywords – terminology extraction of one-word and multi-word units
  • word lists – lists of English nouns, verbs, adjectives etc. organized by frequency
  • n-grams – frequency list of multi-word units
  • concordance – examples in context

English Web 2018 (enTenTen18)

version enTenTen18_tt31 (February 2021)

  • 21.9 billion words (Oct & Nov 2018; Jan, Nov & Dec 2017; Nov & Dec 2016; mainly from 2018)
  • TreeTagger pipeline version 3.1
  • manually checking of biggest web domains (account for 70% of all texts) and content with poor linguistic quality was removed.

English Web 2015 (enTenTen15)

  • initial size 28 billion words

version 2 (spring 2017)

  • 15 billion words
  • TreeTagger pipeline version 2

version enTenTen15_tt31 (March 2020)

  • 13 billion words
  • TreeTagger pipeline version 3.1
  • topic classification (according to dmoz.org)
  • depth analysis of spam and its removal including too short documents

English Web 2013 (enTenTen13)

version ententen13_tt2 (2014)

  • 19 billion words
  • TreeTagger pipeline version 2

version ententen13_tt2_1 (fall 2016)

  • new version of word sketch grammar
  • dynamic attribute doc.website instead of doc.t2ld

English Web 2012 (enTenTen12)

version ententen12_sample40M (14 June 2012)

  • sample of corpus – 3.7 billion words
  • crawled by SpiderLing in May 2012
  • encoded in UTF-8

version ententen12_1 (2012)

  • full corpus – 11 billion words

English Web 2008 (enTenTen08)

version 1 (15 November 2010)

  • initial version – 3.3 billion tokens
  • crawled by Heritrix in 2008
  • encoded in Latin1

TenTen corpora

Jakubíček, M., Kilgarriff, A., Kovář, V., Rychlý, P., & Suchomel, V. (2013, July). The TenTen corpus family. In 7th International Corpus Linguistics Conference CL (pp. 125-127).

Suchomel, V., & Pomikálek, J. (2012). Efficient web crawling for large text corpora. In Proceedings of the seventh Web as Corpus Workshop (WAC7) (pp. 39-43).

Search the English enTenTen corpus

Sketch Engine offers a range of tools to work with this English corpus.

or

Other English corpora

Explore our largest Timestamped English corpus with 50+ billion words.

Use Sketch Engine in minutes

Generate collocations, frequency lists, examples in contexts, n-grams or extract terms. Use our Quick Start Guide to learn it in minutes.