Master's Thesis

Glass-Box Quality Estimation for Neural Machine Translation

João Pinto Correia de Moura — 2021

Key information

Authors:

João Pinto Correia de Moura (João Pinto Correia de Moura)

Supervisors:

André Filipe Torres Martins (André Filipe Torres Martins); João Miguel Da Costa Sousa (João Miguel Da Costa Sousa)

Published in

October 1, 2021

Abstract

Quality Estimation has become increasingly relevant in the last few years for practical and confidence-aware Machine Translation applications, with recent advancements in the field of Natural Language Processing having enabled new approaches to the task. Despite the great improvements that state-of-the-art Quality Estimation systems boast, most overlook a promising source of information: the translation system under evaluation is treated as a black box, with only its input and output being regarded. In this thesis, we introduce a method which allows for the integration of information extracted from the internal mechanisms of Machine Translation models, into the training process of Quality Estimation models, which we call Glass-Box Quality Estimation. First, in order to extract this internal information, we leverage existing model uncertainty quantification methods based on Monte Carlo dropout, which recent work has shown to yield features highly relevant to estimating the quality of machine translated text. We then propose a novel model architecture based on the Predictor-Estimator framework, and an accompanying method to integrate the extracted features into the model's training procedure. Finally, we provide an empirical evaluation based on six language pairs in the context of the WMT Quality Estimation Shared Task, with encouraging results. Our analysis of the proposed model suggests various directions for future improvements.

Publication details

Authors in the community:

Supervisors of this institution:

Fields of Science and Technology (FOS)

mechanical-engineering - Mechanical engineering

Publication language (ISO code)

por - Portuguese

Rights type:

Embargo lifted

Date available:

August 19, 2022

Institution name

Instituto Superior Técnico