Was This Text written by a Human? We asked Real Humans (Part I)

AiRabbit

Nov 27, 2024 • 9 min read

The emergence of Generative AI (GenAI) in our lives has significantly changed many things, one of which is the blurring of texts made by AI and those written by humans. As AI improves, these worlds will blur even more.

In fact, AI derives its writing skills from humans, and as it becomes better at imitating human beings, it will become increasingly difficult to differentiate between the two.

But why does human-authored text matter?

There are many reasons. For example, to detect "cheating" or plagiarism, and because many of us believe that texts with a human touch are more appealing to readers—though for Google, the situation might be different, as we have learned from recent radical changes in the search engine and Google AI.

However, if you depend on AI and want to give your text a human touch—which is not a bad thing per se as long as it is still you who originated the idea—there are tools that can help you do that by rephrasing or paraphrasing your text.

But which of these tools can we trust to make AI-generated text truly more human? If you perceive it as human-like, someone else may perceive it differently.

There are many benchmarks that analyse texts from different LLMs, I have not been able to find many that do the test with the perception of real people and not just for LLMs but for paraphrasing services that claim to be able to humanise text.

Therefore, I decided to conduct an experiment with the help of real humans to see if we could come to a consensus regarding whether a text is made by a human or not.

---

The Experiment

We chose 16 of the most popular tools, including pure-play foundation models like Gemini, OpenAI, and Claude for the experiment.
Originally designed to be conducted with 9 participants.
For the foundation models, we explicitly asked them to make the text human-like.
The order of the tools is random and does not have any significance.
We asked every participant not only to tell us whether they believed the text was written by an AI but also to provide an explanation for their reasoning. Although it can sometimes be quite difficult to articulate—often stemming from a gut feeling—we wanted to capture their insights.
For the rest of the tools, we used the standard rephrasing functionality.
The users are all English speakers based in the US and UK.
They have diverse backgrounds and demographics.
The scores are absolute numbers out of 9; that is, the score you see is the number of users who said this text was not made by humans. Hence, a score of 9/9 means the text was perceived by all participants as human-generated.
The text chosen was itself generated by OpenAI's GPT and was fed to every single tool. The outputs were recorded in a sheet and replaced by numbers to avoid any indication of which tool was used, thus avoiding bias.
The results were then mapped back to the original tools, as you can see below.

Limitations of This Experiment

The experiment is certainly not scientific by any means; it is based on pure curiosity and keen interest in seeing real differences based on human feedback rather than computer-based benchmarks.
We did not test all plans—this certainly goes beyond our capabilities. Feel free to try the other plans yourself.
We do not judge whatsoever which tool is good or bad; again, it is your personal decision. It is a simple experiment to demonstrate how a group of humans perceives text as artificial or human.
The participants by no means represent any specific group (the sample is just too small).
This experiment has certainly many more caveats. I will be happy to listen to your feedback and improvement suggestions.

Having said that, let's jump into it.

---

The Original Text

The Power of Self-Motivation

Self-motivation is the fuel that drives personal and professional growth. It's the inner push that keeps you moving forward, even when challenges arise. Cultivating self-motivation starts with setting clear goals. Visualize your success and break your objectives into manageable steps to avoid feeling overwhelmed.

Celebrate small wins—they build momentum and boost confidence. Surround yourself with positivity: inspiring people, uplifting books, or even motivational podcasts. Remember, setbacks are part of the journey; treat them as learning experiences, not failures.

Practice self-discipline and stay consistent, even when motivation wanes. Ultimately, self-motivation isn't about waiting for the perfect moment—it's about taking action today. Trust in your ability to succeed, and let your determination guide you toward your dreams.

---

Part 1: Tests with LLMs Only

1 — Google Gemini

The Result

Unleash Your Inner Drive: The Power of Self-Motivation

We've all felt that spark, that inner fire that pushes us to do more, be more. That's self-motivation in action! It's the secret ingredient to personal and professional growth.

So, how do we tap into this powerful force? It starts with setting clear, exciting goals. Picture yourself achieving them—that vision is your fuel. Break those big goals into smaller, more manageable steps. This way, you'll avoid feeling overwhelmed and stay motivated.

Celebrate every little win, no matter how small. Each victory, no matter how tiny, builds momentum and boosts your confidence. Surround yourself with positive vibes—inspiring friends, uplifting books, or even those energizing podcasts. Remember, setbacks are just speed bumps on the road to success. Learn from them, grow stronger, and keep moving forward.

Self-discipline and consistency are key. Even when motivation dips, keep going. The best time to start is now, not when you feel "inspired." Trust in yourself, let your determination be your guide, and watch your dreams take flight.

Score: 2 out of 9

What humans say

The format used to write some sentences just seems too robotic and not human touch is felt.
The sentence flow doesn't feel quite natural.
The phrasing seems contrived and a little disjointed.
There seems to be an internal voice with the writing, so this is something that a human was involved with.
The writing style is personal and casual, very human-like.
Rhetorical devices such as the second paragraph asking then answering a question is a human trait.
I think AI wrote this information because a person would not write this much about finding their inner drive.
I don't think people when responding to this question respond that way.
This has a bit more human emotion in it. It's not as clean or sterile.

2 — OpenAI / ChatGPT

The Result

The Power of Self-Motivation

Self-motivation is like the engine that drives both your personal and professional growth. It's that inner spark that keeps you moving forward, even when things get tough. Building self-motivation starts with setting clear goals. Picture your success and break your goals down into small, manageable steps so you don't get overwhelmed.

Celebrate the little victories—they help build momentum and boost your confidence. Surround yourself with positive influences, whether it's inspiring people, uplifting books, or motivational podcasts. Remember, setbacks are just part of the journey. Instead of seeing them as failures, view them as opportunities to learn and grow.

Stay disciplined and keep a consistent routine, even on days when you don't feel motivated. Ultimately, self-motivation isn't about waiting for the perfect moment; it's about taking action right now. Believe in your ability to succeed and let your determination lead you toward your dreams.

Score: 8 out of 9

What humans say

There is some feeling of human to this. Asking the questions and using key words to connect with you can be felt.
This paragraph has emotion to it which makes it feel human.
Use of language and punctuation to accent certain phrases and sentences can only be human.
There does seem to be a glimmer of an internal voice, so this is possibly human created.
An AI would not know what "inner fire" would feel like. The writing style is personal and casual, very human-like.
Rhetorical devices such as the second paragraph asking then answering a question is a human trait.
I feel like someone wrote this information about unleashing their inner drive. It sounds like it came from a person.
My gut tells me a human did this. It seems a bit less reliant on broad general statements that are so sort of polished they lose meaning.
This generates a lot of feeling and emotion. It sounds very human.

3 — Claude

The Result

Finding Your Inner Drive

You know that feeling when something just clicks and you can't wait to get started? That's the magic of self-motivation—it's like having your own personal cheerleader inside your head. When you're truly motivated, nothing feels impossible. Even the biggest mountains look climbable.

The trick is to start small. Think about what you really want, but don't just leave it as a fuzzy dream. Break it down into bite-sized pieces that don't make you want to run and hide. Maybe you want to write a book—great! Start with a single page. Want to run a marathon? Begin with a walk around the block.

Here's something I've learned: those tiny victories matter. Did you write for 10 minutes today? That's worth celebrating. Made it to the gym? Give yourself a high five. These small wins add up, building your confidence day by day. And hey, surround yourself with things that lift you up—whether that's friends who believe in you, books that light a fire in your belly, or podcasts that get you pumped.

And when things go sideways (because they will), remember that every stumble teaches you something valuable. The most successful people aren't the ones who never fail—they're the ones who get back up, dust themselves off, and keep going.

Don't wait for motivation to strike like lightning. Sometimes you just need to start, even when you don't feel like it. Trust me, action creates motivation more often than motivation creates action. You've got this—now go show the world what you're made of.

Score: 7 out of 9

What humans say

It was in depth and tied in personal touch and human experience.
The sentence structure flows well, and shows empathy to the reader.
For me, it's the questions that are asked in the paragraph—AI tends not to do that.
There is too much fluff and creativity about that I imagine an artificial intelligence was involved.
The writing style is personal and human. It is not professional.
The personal "I" is convincing. There is a strong attitude ("when things go sideways (because they will)") that is difficult to manufacture.
I think AI wrote this information because a person would not write this much about finding their inner drive.
I believed (right or wrong) that this was composed by a human because it relates to how it feels to be a person facing these issues.
This passage contains a lot of feeling and emotion, from a personal perspective, which makes it feel very human.

---

Result Summary

Conclusion for Part I: Tests with LLMs Only

In Part I of our experiment, we evaluated the ability of Large Language Models (LLMs) such as OpenAI's GPT, Claude, and Google Gemini to produce text that humans perceive as written by a human. The findings were intriguing:

OpenAI's GPT achieved the highest score with 8 out of 9 participants believing the text was human-authored. The text's emotional resonance and conversational tone contributed to this perception.
Claude also performed well, scoring 7 out of 9. Participants noted the personal touch and relatable style in its output.
Google Gemini scored lower, with only 2 out of 9 participants perceiving its text as human-written. Feedback suggested that while the content was informative, it lacked the natural flow and warmth of human writing.

These results highlight that advanced LLMs like GPT and Claude are becoming increasingly adept at mimicking human writing styles, making it challenging to distinguish between AI-generated and human-written text.

In Part II, we explore how rephrasing services can further transform AI-generated text to enhance its human-like qualities. We'll examine whether these tools can bridge the gap and make AI text indistinguishable from that written by humans.