On the stupid AI vs human writer quiz

I'm coming for you, New York Times

When I was in year five or six, a Pepsi promo team was allowed into my school. I have no idea how this got the okay — at the time we weren’t even allowed vending machines that sold fizzy drinks. But for some reason, there was one recess when we came out of class to discover that Pepsi had set up a stall near the canteen.

They were doing the “let your taste decide” blind test. This involves pouring some Coca-Cola into one cup, Pepsi into another, and then having the participant nominate which drink they think tastes better. As an added bonus, the booth they’d set up at my school gave kids who chose Pepsi a free hat.

I was thrilled. Finally, a test I was guaranteed to ace.

As it happened, I was a Coca-Cola sommelier; a child with a sophisticated palate, honed through the consumption of untold litres of soft drink provided by over-indulgent grandparents. Yes, this talent came at a cost. I was a fat child and, were it not for modern dentistry, a few of my teeth would be so hollow they’d whistle in the wind like a flute.

But I didn’t care about those things back then – and why would I? There was a free hat on the line.

So I waited my turn and eventually got my chance to have a go at the taste test. I, a learned expert, was immediately able to identify the inferior, insipid sweetness of Pepsi. The hat was mine.

The Coke versus Pepsi taste test stunt first started in 1975 and has since been well studied as both a marketing phenomenon and an insight into human flavour preferences. People are far more likely to choose Pepsi in a blind test. The trick, it’s been proposed, is that people prefer the sweetest of the two beverages in small samples.

This is where the premise of the experiment is flawed. A small shot of fizzy drink isn’t the same experience as consuming a whole glass.

Which brings me to AI.

Yes, I’m sorry. It’s another article about the AI slop mudslide that’s coming to suffocate us all.

This time I’m hacked off at The New York Times, which last week published the quiz: Who’s a Better Writer: A.I. or Humans?

It’s a smug little article. In it, the author pits contextless human and AI-written paragraphs against each other, and then asks the reader to choose their preferred passage. The ‘gotcha’ moment is that, when averaged out, 54% of people choose AI. More than 86,000 people have taken the quiz.

This is, of course, a flawed test in a billion ways. It’s like asking who the better architect is after showing someone bricks from two different buildings. Or which car is faster from the shape of the door handle.

As one person in the article’s comments observed, this test has many parallels to the Pepsi challenge – AI writing is fine in small doses, but it falls apart over many paragraphs, pages, and (heaven forbid) chapters.

But even with the cards stacked in favour of AI, I could tell in almost every instance which phrases came from human writers.

AI has a number of tells. They’re not foolproof, of course. Perhaps the most annoying part of this new technological world is being forced to feel like Abigail Williams in The Crucible, whipped into a frenzy, shouting hysterical accusations.

Only these days people are openly cavorting with the devil. It leaves its mark, sure enough. For whatever reason, AI currently has an enormous boner for stating what something isn’t in order to frame what it is.

Here are the examples from the New York Times, with the relevant sentences highlighted:

Who knows why this pleases the computer’s programming, but it does. You’ll see it everywhere now that I’ve pointed it out. It’s in Instagram videos, news articles, ad scripts and essays. It comes out all the time, like an involuntary tic.

But having said that, it doesn’t surprise me that people prefer AI in short doses. Above all else, it prioritises clarity. Those short, punchy sentences get you to your chosen destination in the most efficient way possible. It’s a march. Predictable, and boring.

When writing is done right, it’s a dance. But like differentiating between Coke and its inferior cousin Pepsi, you need good taste to tell the difference.