Beyond Words: Evaluating the Performance of ChatGPT and Google Translate for Accurate Translations

Key Takeaways

• ChatGPT has the potential to challenge Google Translate’s dominance in machine translation due to its ability to provide interpretations of idioms, preserving meaning and intent.
• Both Google Translate and ChatGPT have their strengths and weaknesses in general translation accuracy, with ChatGPT getting closer to the nuance of the message in some cases.
• While Google Translate covers more languages, the translations provided by ChatGPT for unsupported languages may be inconsistent in terms of accuracy. Both tools are free for users to experiment with.

From content writing and programming to product design and data analysis, ChatGPT is immediately impacting almost every digital field imaginable.

However, one area in which ChatGPT could be particularly impactful is machine translation. Currently, Google Translate is the top dog, and almost everyone else is playing catch-up.

But with the rise of ChatGPT, could Google Translate’s dominance be challenged? We’ll pit ChatGPT against Google Translate to see which tool can provide better translations.

What Does a Good Translation Look Like?

The primary purpose of translation is to convey the meaning of a speech written or spoken in one language using another language. Consequently, a good translation must not just swap the meaning of words between two languages but must convey the meaning of the entire body of text as a whole.

It must also not just be a literal meaning of a speech but must communicate the meaning intended by the user while preserving the tone, cultural connotation, and context.

Unfortunately, a good translation is hard, even from big translation services like Google Translate. Machine translation is tough because languages aren’t the same in their approach to constructing sentences. For example, in a language like French, pronouns have a gender, while in Japanese, pronouns are almost completely omitted. Similarly, in Chinese, there are barely any differences between singular or plural nouns. In English, the reverse is the case.

This all adds up to present unique challenges when translating between pairs of languages.

And then there’s the issue of context and colloquialism. Machine translation tools find it hard to get the context of a statement right. One statement may mean one thing in one context and a different thing in another context.

Similarly, colloquialisms that involve proverbs, idioms, and wordplay can be problematic to translate.

Since text translation is one of thethings ChatGPT can do impressively , we decided to compare how it stacks up against Google Translate. We selected some hard-to-translate text that featured common translation problems to compare the two tools.

Google Translate vs. ChatGPT: Translating Colloquialisms

When translating colloquialisms, the translation may not preserve the meaning and intent in a similar tone as the original language.

We asked Google Translate and ChatGPT to translate the simple English idiom “Juan kicked the bucket” to Spanish. Both translation services produced “Juan pateó el balde,” a literal translation of the idiom. While this works, the meaning or intent would be completely lost for someone who doesn’t understand the context.

Here, Google Translate’s ability to help ends. However, ChatGPT offers more help. Rather than simply prompting it to translate, you can ask ChatGPT to provide a “meaning in Spanish” or “provide a meaning in English,” depending on which language you are translating. In this case, ChatGPT will provide a literal translation and interpretation of the idiom.

We tried several other colloquialisms, and both services almost always provided a literal translation. While this works, it could provide misleading information in some situations. ChatGPT’s ability to provide an “interpretation” rather than just a literal translation of colloquialism is an advantage.

But one task is insufficient to draw conclusions, so we increased the heat, this time using a Filipino idiom.

“Sa gitna ng kagutumang buto’t balat at butas na bulsang kahirapan, mataba ang lupa para sa pagtatagumpay ng anakpawis”

The closest native translation of the source text should be: “Amid emaciating hunger and extreme poverty, the grounds are fertile for the victory of the toiling masses.”

ChatGPT had a go at it:

And so did Google:

This clearly gave both tools a hard time, but Google Translate seemed to have an edge.

Of course, things were made even harder, this time switching to the Malayalam language. The task was for both translation tools to interpret an excerpt from a fairly popular novel in Malayalam.

ChatGPT had a go at it, but evidently, complex Malayalam text is not one of ChatGPT’s strong points.

Google, on the other hand, did fairly well.

EmEditor Professional (Lifetime License, non-store app)

None of the tools could get it perfectly, but Google Translate was as close as you can get. This is a close contest. While Google Translate has an edge, ChatGPT can provide the meaning of idioms instead of just translations. This could be very useful when translating a larger body of text that contains an idiom. In such cases, translating an idiom literally could be a source of confusion when read together with the surrounding text.

Jutoh Plus - Jutoh is an ebook creator for Epub, Kindle and more. It’s fast, runs on Windows, Mac, and Linux, comes with a cover design editor, and allows book variations to be created with alternate text, style sheets and cover designs. Jutoh Plus adds scripting so you can automate ebook import and creation operations. It also allows customisation of ebook HTML via templates and source code documents; and you can create Windows CHM and wxWidgets HTB help files.

Google Translate vs. ChatGPT: General Accuracy

Subtle differences between a source text and its translation can completely alter the meaning of a text. Therefore, we tested both Google Translate and ChatGPT to see their general translation accuracy.

We started simple, a Chinese philosophy text:

在这个虚伪的社会中，真正的人格是一种罕见的珍贵财富。

The closest translation of this is, “In this hypocritical society, true personality is a rare and precious asset.” Again, both ChatGPT and Google Translate were spot on. No difference in translation.

Up next is a translation task in Filipino.

Ang gamot sa topaking babae ay ang lambing ng isang maunawaing lalaki

The source text is a bit of a complex wordplay: “The cure for a girl who acts up is the sweetness of an understanding man.”

Google turned up: “The medicine for a woman is the tenderness of an understanding man.” So it wasn’t a complete disaster, but it didn’t get the nuance of the message right.

Jutoh is an ebook creator for Epub, Kindle and more. It’s fast, runs on Windows, Mac, and Linux, comes with a cover design editor, and allows book variations to be created with alternate text, style sheets and cover designs.

ChatGPT, on the other hand, turned up: “The cure for a heartbroken woman is the affection of an understanding man.”

None of the translations are particularly perfect, but ChatGPT got as close as possible.

Google Translate vs. ChatGPT: Translating Creoles

Creoles, especially those that borrow many words from other languages, are notoriously hard to translate. We tasked both ChatGPT and Google Translate to interpret a creole called “Pidgin English,” spoken in West Africa.

Below is the source text:

“Wetin come happen na. Since you say you go come help me with that thing, I no come see your brake light. Wetin sup?”

In the source text, the speaker is complaining to the reader that they promised to help with a task but haven’t turned up since then. Then, it concludes with “What’s happening?”

While Google Translate understands “Pidgin English ‘’ to some extent, it failed woefully at translating the source text. Part of the problem is due to Google Translate’s limited ability to distinguish Creole from English because of the presence of English words.

Although not flawlessly, ChatGPT’s interpretation was able to clearly convey the meaning and intent of the speaker. Considering the highly contextual nature of Pidgin English, the results are very impressive.

Google Translate vs. ChatGPT: Language Coverage

Comparing the language coverage between ChatGPT and Google Translate is a bit tricky. Google Translate covers more than 140 languages, and that number keeps growing. This is because Google regularly adds new languages to the Google Translate tool. For ChatGPT, on the other hand, there are no official numbers regarding how many languages the tool can cover. However, based on how ChatGPT was trained, there’s a logical basis to suspect that the coverage is significantly more than ChatGPT.

Of course, we tested ChatGPT on several languages not supported by Google Translate. It was able to provide translations for these additional languages, which Google Translate does not offer. However, the quality of ChatGPT’s translations for these unsupported languages was a bit inconsistent. While some were considerably accurate, some were quite poor and inaccurate. While ChatGPT covers more languages, the translations it provides for many of them leave much to be desired in terms of accuracy.

Google Translate vs. ChatGPT: Which Translation Tool Is Better?

It’s hard to say with certainty. Both translation tools have their strong points. Google has been making massive investments inNatural Language Processing (NLP) . As a result, it outperforms ChatGPT in languages to which it has paid special attention. However, consideringhow ChatGPT works and its training , it represents a unique and exciting approach to translation. Which of them should you use? Both tools are free, so feel free to experiment and settle for which one works for you.

ChatGPT: An Emerging Translation Tool

Google Translate has made a name for itself in the machine translation space. However, ChatGPT, although relatively new, is not a pushover. One of ChatGPT’s greatest advantages is its ability to adjust its translation based on the context or extra provided by the user. Google Translate can not currently do this.

While we can’t confidently say whether ChatGPT will challenge Google Translate’s dominance as the preferred translation tool, the AI chatbot clearly has the potential to give Google a fight.