Neural Machine Translation (NMT) model to translate from German to English
Model Overview
This model can be used for translating text in source language (De) to a text in target language (En).
Model Architecture
The model is based on Transformer "Big" architecture originally presented in "Attention Is All You Need" paper [1]. In this particular instance, the model has 12 layers in the encoder and 2 layers in the decoder. It is using YouTokenToMe tokenizer [2].
Training
These models were trained on a collection of many publicly available datasets comprising of millions of parallel sentences. The NeMo toolkit [5] was used for training this model over roughly 300k steps.
Datasets
While training this model, we used the following datasets:
- Europarl de-en set from: http://www.statmt.org/europarl/v10/training/europarl-v10.de-en.tsv.gz
- De-En version of parallel common crawl from: http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz
- De-En version of paracrawl from: https://s3.amazonaws.com/web-language-models/paracrawl/release7.1/en-de.txt.gz
- De-En News commentary version from: http://data.statmt.org/news-commentary/v15/training/news-commentary-v15.de-en.tsv.gz
- De-En Wikipedia Parallel Titles from http://data.statmt.org/wikititles/v2/wikititles-v2.de-en.tsv.gz
- A subset of Tilde Corpus from: https://tilde-model.s3-eu-west-1.amazonaws.com/EESC2017.de-en.tmx.zip
- A subset of Tilde Corpus from: https://tilde-model.s3-eu-west-1.amazonaws.com/rapid2019.de-en.tmx.zip
- A subset of Tilde Corpus from: https://tilde-model.s3-eu-west-1.amazonaws.com/ecb2017.de-en.tmx.zip
- A subset of Tilde Corpus from: https://tilde-model.s3-eu-west-1.amazonaws.com/EMA2016.de-en.tmx.zip
- De-En portion of WikiMatrix v1 de-en data from: http://data.statmt.org/wmt20/translation-task/WikiMatrix/WikiMatrix.v1.de-en.langid.tsv.gz
Tokenizer Construction
We used the YouTokenToMe tokenizer [2] with shared encoder and decoder BPE tokenizers.
Performance
The accuracy of translation models are often measured using BLEU scores [3]. On WMT20 Test set this model achieves 36.4 BLEU score measured using SacreBLEU package [4]. BLEU+case.mixed+lang.de-en+numrefs.1+smooth.exp+test.wmt20+tok.13a+version.1.5.0 = 36.4 71.7/46.7/32.6/23.3 (BP = 0.912 ratio = 0.915 hyp_len = 34980 ref_len = 38220)
How to Use this Model
The model is available for use in the NeMo toolkit [5], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
Automatically load the model from NGC
import nemo
import nemo.collections.nlp as nemo_nlp
nmt_model = nemo_nlp.models.machine_translation.MTEncDecModel.from_pretrained(model_name="nmt_de_en_transformer12x2")
Translating text with this model
python [NEMO_GIT_FOLDER]/examples/nlp/machine_translation/nmt_transformer_infer.py --model=nmt_de_en_transformer12x2.nemo --srctext=[TEXT_IN_SRC_LANGUAGE] --tgtout=[WHERE_TO_SAVE_TRANSLATION] --target_lang en --source_lang de
Input
This translate method of the NMT model accepts a list of de-tokenized strings.
Output
The translate method outputs a list of de-tokenized strings in the target language.
Limitations
No known limitations at this time.
References
[1] Vaswani, Ashish, et al. "Attention is all you need." arXiv preprint arXiv:1706.03762 (2017).
[2] https://github.com/VKCOM/YouTokenToMe
[3] https://en.wikipedia.org/wiki/BLEU
[4] https://github.com/mjpost/sacreBLEU
Licence
License to use this model is covered by the NGC TERMS OF USE unless another License/Terms Of Use/EULA is clearly specified. By downloading the public and release version of the model, you accept the terms and conditions of the NGC TERMS OF USE.