GDEX stands for “Good Dictionary EXamples”. It is a tool for the evaluation of sentences to find suitable dictionary examples or good examples for teaching purposes. Sentences are evaluated with respect to their length, the use of vocabulary, presence of controversial topics (politics, religion…) and other criteria.
GDEX stands for “Good Dictionary EXamples”. It is a tool for evaluating sentences with respect to their suitability to serve as dictionary examples. or examples for teaching purpoes. It identifies such sentences automatically, making the process of finding suitable sentences much more efficient. It can effectively exclude poor sentences and offers the user a pre-sellected set of sentences with a much higher chance of containing good sentences for the purpose of dictionary examples.
The GDEX configuration is fully customizable. It can take into account linguistic and cultural restrictions. To request a bespoke GDEX configuration, please contact inquiries@sketchengine.eu
Alternative GDEX configurations can be selected, if available. (The configuration selected here is also used in TickBox Lexicography (TBL).)
The sentences will be sorted by the best GDEX score from best to worst. The sentences often require some postediting before using them as dictionary examples or as sentences for teaching. Despite this, the use of GDEX makes the process of developing example sentences much more effective.
Users can develop their own GDEX configuration to make the filtering more restrictive or permissive. For example, sentences to use with young learners may require stricter criteria. See manual for GDEX .
You may like to contact us first before starting a major GDEX project.
Normal concordance for the word historically. GDEX not applied.
The same concordance but with the first 3,000 lines evaluated by GDEX and sorted by the GDEX score.
A concordance has to be generated to use GDEX. The GDEX tool then evaluates the sentences and sorts them from the best to worst. The number of sentences to be evaluated has to be specified. A random sample of the same size is generated from the sentences in the concordance and evaluated.
To activate GDEX, click the button in the toolbar above the concordance results.
If more than one GDEX configuration is available, use the dropdown list to select it. The configuration selected here will also be used in TickBox Lexicography.
Users can create their own configuration to apply more restrictive or more permissive criteria on the sentences depending on their intended use. For example, teaching materials for children may require a stricter configuration to exclude unsuitable sentences.
User configurations can be uploaded to Sketch Engine by going to My account – My Sketch Engine. (Local installations need to register gdex configurations manually.) User configurations can also be shared with other users or user groups.
GDEX configuration files are written in YAML (Wikipedia.org) which is a human-readable (and human-editable) format also suitable for effective machine processing. The actual formula for calculating sentence scores is an expression in the Python programming language. Its syntax is limited to basic mathematical and logical operations, as well as function calls to pre-defined GDEX classifiers. Several variables such as the values of positional attributes are available. Named variables defined in the configuration file, typically regular expressions, can also be referenced in the formula.
The file contains two top-level keys – the mandatory formula and optional variables. Unless the formula fits on a single line, it must be preceded by the > YAML symbol for multi-line values. YAML does not allow the tab character, so you must use spaces for indentation. This is what a simple configuration file may look like:
formula: >
(50 * is_whole_sentence() * blacklist(words, illegal_chars) * blacklist(lemmas, parsnips)
+ 50 * optimal_interval(length, 10, 14)
* greylist(words, rare_chars, 0.1)
* greylist(tags, pronouns, 0.1)
) / 100
variables:
illegal_chars: ([<|][>/\^@])
rare_chars: ([A-Z0-9'.,!?)(;:-])
pronouns: PRON.*
parsnips: ^(tory|whisky|cowgirl|meth|commie|bacon)$
The formula must produce a number between 0 (worst) and 1 (best). Values outside this range will be changed to its nearest limit.
Apart from the variables (actually constants) defined in the configuration, these important ones are available:
This is a general example of how variables in the GDEX configuration can work and can be named. You can name them differently or add more variables for checking further sentence features.
The attribute lists can be used as a whole (for example as a parameter to a classifier) or you can even access individual tokens using standard Python syntax. For example, words[0] is the first word in the sentence and tags[-1] is the tag of the last token.
blacklist(tokens, pattern) returns either 1 if none of the tokens (e.g. words, lemmas etc.) matched pattern (regular expression), 0 otherwise
greylist(tokens, pattern, penalty) is similar to blacklist, but you can specify a penalty that will be subtracted from 1 for each token matching pattern down to 0. With a penalty of 1, it behaves as a blacklist.
optimal_interval(value, low, high) returns 1 if value is between low and high. Outside this range, the score linearly rises from 0 at low/2 to 1 at low and falls from 1 at high to 0 at 2*high. For value lower than low/2 or higher than 2*high, the score is 0. Usually used with length.
is_whole_sentence() (mind the parentheses) returns 1 if the sentence starts with a capitalized word and ends with a full stop, question mark or exclamation mark. Otherwise, it returns 0.
word_frequency(word) returns the absolute frequency of the given word in the corpus. word_frequency(word, normalize) returns the relative frequency per normalize tokens. For example: word_frequency(words[1], 1000)
keyword_position() returns a number between 0 and 1, starting at zero for a keyword at the beginning of the sentence, and rising in equal increments (depending on the length of the sentence) to 1 for a keyword at the end of the sentence.
keyword_repetition() returns the number of occurrences of the keyword in the sentence.
This page gives information for local installations only. Nothing needs installing when using GDEX in Sketch Engine on www.sketchengine.eu
GDEX is distributed as Python distutils packages that is installed in a standard way:
python setup.py install
Sketch Engine automatically detects gdex (must be in PYTHONPATH) and unlocks respective functionality.
The online interface provides a special page for uploading user configurations to Sketche Engine. Local installations need to modify gdexpath variable in run.cgi in section # ConcCGI options.
The value should contain a list of tuples (‘configuration_name’, ‘full_path_to_the_configuration). E.g.:
class BonitoCGI (WSketchCGI, UserCGI, WSEval):
...
# ConcCGI options
gdexpath = [('Shorter', '/corpora/gdex/configurations/shorter')]
...
Alternatively a path to default configuration of a corpus can be set in its registry file using GDEXDEFAULTCONF.
Adam Kilgarriff, Miloš Husák, Katy McAdam, Michael Rundell and Pavel Rychlý (2008). GDEX: Automatically finding good dictionary examples in a corpus. In Proceedings of the 13th EURALEX International Congress. Spain, July 2008, pp. 425–432.
