deduplicationDeduplication is a process of removing duplicated content from a corpus. Only the first instance of the text is preserved, any subsequent (duplicated) occurrences are removed.
Deduplication is especially important with corpora built by crawling the web. This is because lots of web content [...] Read More
disambiguationis the process of identifying meanings of words (lemma, part of speech) when a word has multiple meanings. The result of this process is one word with one meaning.
distributional thesaurusis an automatically produced thesaurus which identifies words that occur in similar contexts as the target word. It draws on the theory of distributional semantics. It is available for every word in the corpus.
more about automatic thesaurus
The distributional thesaurus in Sketch Engine [...] Read More
documentA document (called a file in old corpora) in Sketch Engine refers to any file, document or webpage that makes up the corpus.
If a user uploads a file (such as .doc, .pdf, .txt), each of the files becomes a corpus document. If the user downloads content from the web, each web page becomes a [...] Read More
document frequency (docf)Document frequency is the number of documents in which a token or phrase appears.
If the corpus has 100 documents and 2 documents contain the word city:
document number 7 contains 17 instances of city,
document number 31 contains 6 instances of city,
the document frequency of city is [...] Read More
