Working Examples#

You can create an combination of configurations based on your needs.

Step 1#

Start your basic configurations with as the following example command line:

python main.py -m \
    dataset=dwug_de_210 \
    task=wic \
    evaluation=none \

There are options for dataset, task, and evaluation.

  • diawug_110

  • discowug_110

  • durel_300

  • dwug_de_210

  • dwug_de_sense

  • dwug_en_200

  • dwug_es_400

  • dwug_la_1

  • dwug_sv_200

  • nordiachange_1

  • nordiachange_2

  • refwug_110

  • rusemshift_1

  • rusemshift_2

  • rushifteval_1

  • rushifteval_2

  • rushifteval_3

  • surel_300

Step 2#

The bash will return the feedback message to ask you specify more detail infromation. For example, if you choose to use LSCD_binary model, you will be ask to specify task/lscd_binary@task.model with the options of apd_compare_all and cos. If you use LSCD_graded model, you will be ask to specify task/lscd_graded@task.model with the options of apd_compare_all, apd_compare_annotated, cluster_jsd, permutation, etc.

Task Examples#

We provide command line examples with different tasks for reference. They use the German dataset dwug_de_210, the German BERT and do not have evaluation setting.

WiC#

Here is the command lines for applying the WiC task. You can find more detail about WiC task in the page Tasks/WiC.

  1. WiC task work with model contextual_embedder would be the following:

    python main.py -m \
    evaluation=none \
    task=wic \
    task/wic@task.model=contextual_embedder \
    task/wic/metric@task.model.similarity_metric=cosine \
    dataset=dwug_de_210 \
    dataset/split=dev \
    dataset/spelling_normalization=german \
    dataset/preprocessing=raw \
    task.model.ckpt=bert-base-german-cased

  2. WiC task work with model deepmistake would be the following:

    python main.py -m \
  dataset=dwug_de_210 \
  dataset/preprocessing=toklem \
  dataset/spelling_normalization=german \
  dataset/split=dev \
  'dataset.test_on=[abbauen,abdecken,"abgebrüht"]' \
  task=wic \
  evaluation=wic \
  evaluation/metric=spearman \
  task/wic@task.model=deepmistake \
  task/wic/dm_ckpt@task.model.ckpt=WIC_DWUG+XLWSD

WSI#

Here is the command lines for applying the WiC task. You can find more detail about WiC task in the page Tasks/WSI.

python main.py -m \
    evaluation=none \
    task=wsi \
    task/wsi@task.model=cluster_correlation \
    task/wic@task.model.wic=contextual_embedder \
    task/wic/metric@task.model.wic.similarity_metric=cosine \
    dataset=dwug_de_210 \
    dataset/split=dev \
    dataset/spelling_normalization=english \
    dataset/preprocessing=raw \
    task.model.wic.ckpt=bert-base-german-cased \
    task.model.wic.gpu=1

LSCD#

Here is the command lines for applying the WiC task. You can find more detail about WiC task in the page Tasks/LSCD. There are three different version for LSCD tasks, i.e. lscd_binary, lscd_compare, and lscd_graded. You can find the comman line examples for each of them in the following list.

  1. lscd_binary

    python main.py \
    dataset=dwug_de_210 \
    dataset/split=dev \
    dataset/spelling_normalization=german \
    dataset/preprocessing=raw \
    evaluation=none \
    task=lscd_binary \
    task/lscd_binary@task.model=apd_compare_all \
    task/lscd_binary/threshold_fn@task.model.threshold_fn=mean_std \
    task/wic@task.model.graded_model.wic=contextual_embedder \
    task/wic/metric@task.model.graded_model.wic.similarity_metric=cosine \
    task.model.graded_model.wic.ckpt=bert-base-german-cased \
    task.model.graded_model.wic.gpu=1

  2. lscd_compare

    python main.py \
    dataset=dwug_de_210 \
    dataset/split=dev \
    dataset/spelling_normalization=german \
    dataset/preprocessing=raw \
    evaluation=none \
    task=lscd_compare \
    task/lscd_compare@task.model=cos \
    task/wic@task.model.wic=contextual_embedder \
    task/wic/metric@task.model.wic.similarity_metric=cosine \
    task.model.wic.ckpt=bert-base-german-cased \
    task.model.wic.gpu=1

    previous

    Usage

    next

    Developer’s Guide