The researchers show examples of four major types of errors discovered by their tests/methods on three SOTA NMT systems. Test results (PR) on the capabilities for numerical translation, with low-averaged scores in bold. The behaviour tests were run on popular commercial translation systems, with the team using pass rate (PR) as their evaluation metric. High-resource set included English-German and English-Chinese pairs, and the low-resource set contained English-Tamil and English-Nepali pairs. In addition to this, the researchers conducted their assessments in both high-resource and low-resource scenarios. However, the researchers noted that all the numbers created for a format were seen as a set of ‘adversarial’ examples. It allowed them to generate diverse numbers at scale, akin to fuzzing a programme with random inputs to uncover bugs. The team examined numerical formats of various lengths and decimal-point positions and tested them on 25 different formats across four capabilities. Checking capabilities of NMT systems in translating common types of numerical text (Source: arXiv)
“Systems that fail to manifest one or more of these capabilities may produce wrong numbers that can be inconspicuous to users and become a ready, exploitable source of misinformation,” revealed the researchers. On the other hand, numerals were tested by numbers written in words and separators with numbers containing periods or commas. The tested NMT systems were more likely to malfunction when translating larger integers and decimals with longer fractional parts. The integers were tested on sequences of digits with ‘variable lengths’ and the decimals on ‘floating-point numbers’ with different levels of precision. The researchers drew inspiration from the CheckList behavioural testing framework in designing their evaluation suite for NMT systems, alongside exploring four capabilities that demonstrate the expected translation abilities of systems on common numerical text: integers, decimals, numerals and separators. Finally, the abuse of these errors constitutes vectors of attack: error-prone numerical tokens injected into monolingual data may corrupt back-translation-based training, as the resulting back-translated sentences are likely to contain the desired errors. The researchers said these error types greatly extend the number category (NUM) of the catastrophic errors of NMT systems with richer error types. Numerical errors were discovered when behavioural testing two popular commercial translation systems using their public APIs. Their tests on SOTA NMT systems expose novel error types that have evaded close examination, as shown in the table. The researchers believe their method extends the CheckList behavioural testing framework by designing automatic test cases to assess a suite of fundamental capabilities a system should exhibit in translating numbers.
“Surprisingly, numerical mistranslation is a general issue faced by SOTA NMT systems, including commercial and research systems, with evidence present across contexts: for both high and low resource languages, and for both close and distant languages,” wrote the researchers, in ‘As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation’ paper, co-authored by Jun Wang, Chang Xu, Francisco Guzman, Ahmed El-Kishky, Benjamin I. Numerical errors made in financial document translation, for example, an extra or omitted digit or decimal point, could lead to significant monetary losses. For instance, in healthcare, mistranslating the number of confirmed cases of COVID-19 may exacerbate public health misinformation. Mistranslation constitutes a critical but under-explored category with serious implications. The new approach revealed novel errors common across multiple SOTA translation systems. Researchers from the University of Melbourne, Facebook AI, and Twitter Cortex recently proposed a black-box test method for assessing and debugging the numerical translation of neural machine translation ( NMT) systems.
0 kommentar(er)
