Configuring experiments

With the goal of explicitness, every field in the configuration files is mandatory, except for a few.

Dataset

Name

The name field defines the name of a dataset. It is used mainly for presentation purposes.

Groupings

DWUG datasets have a grouping column in each lemma's uses.csv file. Most of the time there are only two values, in which case each grouping corresponds to a historical periods. However, other datasets have more than two grouping IDs, which might indicate a regional shift, instead of a historical one. The groupings field therefore lets the user choose which uses to load, based on their corresponding grouping.

It is represented as a tuple of two strings.

Cleaning

cleaning can be used to filter out lemmas depending on certain values. The structure of this field is defined by the following two classes:

class CleaningParam(BaseModel):
    threshold: float
    keep: Literal["above", "below"]

class Cleaning(BaseModel):
    stats: dict[str, CleaningParam]
    match: Literal["all", "any"]

In the Cleaning class, the keys of Cleaning.stats refer to column headers in the stats_agreement.csv file of the selected dataset. For example, the following configuration keeps only lemmas that have a Krippendorf alpha value higher than 0.7:

stats:
 kri_full:
  threshold: 0.7
  keep: above
match: all

Here, the value of match doesn't have a big impact. If we had defined two criteria (two columns), all would filter out lemmas satisfying all the specified criteria. If match is set to any, it will filter out lemmas matching any of the criteria (i.e., at least one)

Preprocessing

This field (preprocessing) refers to how a given use should be preprocessed. The possible options are:

• toklem: tokenize the text, and lemmatize the target word
• tokenize: tokenize the text
• lemmatize: lemmatize all words
• normalize: some languages contained characters that are no longer in use, which language models have never seen during training. Some of the DWUG datasets contain text whose spelling has been pre-normalized

Spelling Normalization

The spelling_normalization field of preprocessing can be used to replace certain substrings with other substrings. It can be used to apply custom spelling normalization.

Model

Evaluation