The Smartest AI (Fable 5) Model Almost Cost My Friend the Job
Recently, a friend in Germany asked me to help him improve his CV using AI. He's excellent in his profession, but lacks AI skills, so I offered to help. I started drafting the CV and chose one of the smartest models currently on the market for the task: Fable 5.
It's a super simple task, one might think. No tools, agents or code were used, just pure text processing and optimisation. The kind of thing LLMs are supposedly best at.
The result was shocking.
I knew there were significant differences in model capabilities when it came to languages. But I did not expect an extremely intelligent model, one that is supposedly capable of contributing to the largest open-source projects, to produce phrasing that even a 10-year-old would find awkward in German. I did not expect that.
Here is what happened:
One word that gives everything away
Fable 5 produced the following opening line for the resume (job title changed for illustration):
"Ich bin Senior IT-Projektleiter mit 15+ Jahren Hintergrund in komplexen Systemmigrationen..."
If you don't speak German, this looks fine. If you do, one word jumps out immediately: "Hintergrund".
It's a crude 1:1 translation of the English "15 years of background". No native German speaker would ever phrase it that way; in German you have "Erfahrung" (experience) or "Expertise", never "Hintergrund". And this wasn't buried in paragraph three. It was the very first sentence of a professional CV.
That's a knockout criterion. A recruiter who reads dozens of resumes a day senses it instantly: a machine wrote this. Or worse: this candidate can't write professional German.
The experiment: three models, three completely different outcomes
I wanted to know whether other models would catch the mistake. So I gave the same sentence to different models for review, each in a fresh chat.
The prompt was simple: what do you think about this phrasing?
Model 1: Fable 5 (the culprit). Produced the sentence and saw nothing wrong with it.
Model 2: Opus 5 (one of the most popular models out there). It got tangled up in strange grammar feedback, but completely missed the actual problem: the unnatural vocabulary.
Model 3: Gemini 3.1 Pro (AI Studio), the model currently leading many of the global language leaderboards, nailed it on the first try:
"What should be slightly adjusted: 'Hintergrund'. In German, 'mit 15+ Jahren Erfahrung' or 'Expertise' sounds far more professional and active than 'Hintergrund'."
One model made the mistake. One model overlooked it. One model found it instantly. Three top-tier models: three different worlds the moment German language nuance is involved.
Why does this happen?
Large language models are trained predominantly on English data. German is well represented, but the "thought patterns" of many models remain English-shaped. The result is a kind of high-gloss Denglisch: grammatically correct, idiomatically off.
Typical patterns to watch for in AI-generated German text:
- "Hintergrund" instead of "Erfahrung": from "background in..."
- "Ich bin passioniert über...": from "passionate about" (natural: "Meine Leidenschaft gilt..." / "Ich begeistere mich für...")
- "Das macht Sinn": from "makes sense" (formal German prefers "Das ergibt Sinn" / "Das ist sinnvoll")
- "In 2024 habe ich...": from "in 2024" (correct: "2024 habe ich..." or "Im Jahr 2024...")
- "Ich freue mich, von Ihnen zu hören": a literal "I look forward to hearing from you" (more natural: "Ich freue mich auf Ihre Rückmeldung")
Each of these is a small red flag. In a job application they add up to one overall impression: not authentic.
As you might expect, German is no exception here; it's a general problem when it comes to non-English texts.
How is this even measured? The Global-MMLU-Lite benchmark
If you're wondering whether my experience is just an anecdote: it isn't. Exactly this capability, real language understanding versus mere translation, is measured systematically.
The origin: Global MMLU
The classic MMLU benchmark (Massive Multitask Language Understanding) tests knowledge and reasoning across dozens of subject areas. The problem: it's English, and heavily Western-centric. Researchers (including Cohere Labs) demonstrated in the Global-MMLU study just how skewed it is:
- 28% of all questions require culturally sensitive knowledge, predominantly from a Western perspective.
- Among geography questions, 84.9% refer to North America or Europe.
- Earlier multilingual versions relied on machine translation, with exactly the translation artifacts that ruined my resume sentence.
Global MMLU fixes this with professional, compensated translators plus community verification across 42 languages. Questions are additionally split into "culturally sensitive" and "culturally agnostic" subsets. One striking finding: model rankings shift significantly depending on which subset you test. A model that shines in English can stumble badly in other languages.
The measurement: Artificial Analysis
The independent evaluation platform Artificial Analysis uses the Lite variant of this dataset to continuously test over 100 current models, across 16 languages (including German), with roughly 400 questions per language.
The methodology is what makes the results comparable:
- Standardized conditions: every model gets identical prompts, temperature settings, and evaluation criteria.
- Zero-shot: no examples, no hand-holding: the model must understand and follow the instruction directly.
- Pass@1: only the first answer counts. No retries.
- Full transparency: prompt templates, criteria, and limitations are publicly disclosed.
At the time of writing, Gemini 3.1 Pro leads the Global-MMLU-Lite ranking at 93.2%, ahead of Gemini 3 Pro and Claude Opus 4.6 (92.2% each). Suspiciously consistent with my resume experience, isn't it?
The crucial point: the benchmark doesn't test whether a model can translate. It tests whether a model can retrieve knowledge and reason directly in German (or Arabic, Japanese, Swahili...). A model that only "thinks" in English and translates internally gets exposed, in the benchmark just as much as in a cover letter.
What actually helped
After the Fable 5 disaster, here's the workflow that produced a text my friend was genuinely happy with:
- Pick the model by language ranking, not by general hype. For German output, I checked the multilingual leaderboard first instead of assuming "smartest model = best German".
- Prompt for idiomatic output explicitly. Not just "write my resume in German", but: "Write idiomatic, natural German as a native speaker would phrase it. Avoid anglicisms and literal translations of English idioms. Afterwards, critically review the text for unnatural phrasing."
- Cross-check with a second, provably strong model. Have a top-ranked model in the target language review the draft in a fresh chat. It caught what the author model couldn't see.
- Human in the loop, always. Read it. Read it out loud. Every word that feels off probably is. For high-stakes text (applications, contracts, client communication) a native speaker's final pass beats any leaderboard.
My three takeaways
1. Benchmarks matter, even if you're not a benchmark nerd. Metrics like Global-MMLU-Lite reflect, very concretely, how confidently a model navigates the nuances of a local language. If you regularly produce German (or any non-English) text with AI, check the language leaderboards before choosing a model.
2. English ≠ German. A model can be an absolute genius in English and still walk straight into embarrassing translation traps in German. Rankings differ per language, sometimes dramatically.
3. Never blind copy-paste. "Smartest model + maximum thinking" is not a quality guarantee for language. Try-and-hope is not a strategy; a review step is.
Happy writing:)