This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.
A few years ago, humans were clearly superior to machine learning algorithms in tasks such as language translation. But now, these once distinct lines dividing capabilities are becoming blurred.
In a recent study, researchers compared the translation abilities of several large language models (LLMs) to professional human translators, finding that only certified experts with 10 years of experience or more were able to clearly outperform the models. In certain aspects of language translation, the models actually outperformed humans. The results were published on 15 December in IEEE Transactions on Big Data.
Yue Zhang, the associate dean of the school of engineering at Westlake University in Hangzhou, China, notes that over the past two decades, there has been “a major paradigm shift” in the capabilities of machine learning algorithms, and points to a particularly large leap in performance seen in the newest generation of LLMs. But just how well do these models perform in terms of language translation?
“While there have been claims of ‘human parity’ in the past, these have been debated,” he says. “We wanted to move beyond vague comparisons and scientifically calibrate LLM performance against specific tiers of professional human expertise—ranging from junior to senior translators.”
How Do LLMs Compare to Human Translators?
In the study, junior translators were defined as having one to two years of experience in the translation industry; medium-level translators having three to five years of experience, or were a native speaker of the target language; and senior-level translators having a minimum of ten years of translation experience, plus holding a distinguished China Accreditation Test for Translators and Interpreters translation certification, China’s national standard for translators.
The human translators and LLMs models—including GPT-4, ALMA-R, and Deepseek-R1—were tasked with translating the same text samples. Six professional annotators were hired to evaluate the quality of the translations, without knowing which ones were produced by humans or by LLMs.
The researchers tasked both groups with translating text between common pairings, such as English and Chinese, as well as less common pairings such as Chinese and Hindi.
The results show that GPT-4 has translation abilities comparable to junior and medium-level human translators, which Zhang notes is very likely the first time in history that an algorithm has reached human-level translation quality.
In translating chunks of text containing roughly 200 sentences each, across eight pairs of languages, GPT-4 produced an average of 3.71 major translation errors, whereas junior-level and medium-level translators produced an average of 3.27 and 3.30 major errors, respectively. Senior-level translators offered the highest quality translations, with an average of just 1.83 major errors. Mistakes were more commonly made by the humans and models alike when they were tasked with translating less common language combinations, such as Chinese to Hindi.
However, kinds of errors made by humans and by models differed. LLMs tended to be overly literal in their translations at times, but humans were found to do the opposite, by being too imaginative when trying to “fill in the gaps” of vague or ambiguous wording. For example, one human translator in the study incorrectly understood the phrase “entering his second year,” as referring to a two-year-old baby, when the sentence in fact was describing a second-year sports player. “This is both the advantage of human translators and the disadvantage,” Zhang says.
In the study, human translators made more errors related to overinterpretation than the LLMs. But this same ability to look deeper into the context of language may have also contributed to the success of the senior translators in accurately translating the more nuanced text excerpts.
Zhang says that, for tasks requiring high precision, cultural adaptation, or complex creative interpretation, like literature, senior human translators are still necessary. However, he points to some initial evidence that this could be changing. He notes that, in the study, “DeepSeek R1, a deep reasoning model, was particularly good at avoiding major translation errors, suggesting that models capable of reasoning—such as OpenAI o1, GPT-5, DeepSeek v 3.2—might be the key to closing the gap [with senior-level human translators].”
From Your Site Articles
Related Articles Around the Web
