Simple maths is the keyness score used in Sketch Engine to identify keywords, terms, key n-grams and key word sketch collocations. Simple maths compares the frequencies in the focus corpus with the frequencies in the reference corpus. Alternatively, two subcorpora in the same corpus or in different corpora can be used.

The N value makes the score prefer more frequent or less frequent items.

A higher N value shifts to focus on higher-frequency words (more common words), whereas a lower N value focusses on low-frequency (rarer words). The value should be changed in orders of magnitude, i.e. 0.1, 1, 10, 100, 1000, 10000 etc. Smaller changes rarely produce any noticeable effect.

The statistics is a variation on “word W is so-and-so times more frequent in corpus X than corpus Y”. The formula is:

where

is the normalized (per million) frequency of the word in the focus corpus,

is the normalized (per million) frequency of the word in the reference corpus,
is the smoothing parameter ( is the default value).

Example

Your focus corpus (BNC): 112,289,776 tokens
Frequency of the lemma (shard) in the corpus: 35

Relative frequency

Selected reference corpus (ukWaC): 1,559,716,979 tokens
Frequency of the lemma (shard) in the corpus: 263

Keyness score


For more details see:

Adam Kilgarriff. Simple maths for keywords. In Proceedings of Corpus Linguistics Conference CL2009, Mahlberg, M., González-Díaz, V. & Smith, C. (eds.), University of Liverpool, UK, July 2009.

Statistic used in Sketch Engine (Chapter 5). Lexical Computing Ltd., 8 July 2015.

Explore Sketch Engine

Find out whether Sketch Engine is an appropriate tool for your tasks.