Lots of companies are building Digital Twins. Almost none are marking them.

In August, a consortium led by Harvard and MIT published a study built on 8.3 billion simulated people. Buried inside it is a finding our industry should read twice.

They took identical groups of simulated people and asked them an identical question about a price rise. They changed one thing: which AI model was interpreting the data and speaking on their behalf. Under the first model, 27% said they would think twice about buying. Under the second: 98%. Same simulated people, same question, seventy-one points apart.

Human ground truth data remains necessary

The authors are refreshingly blunt about what follows. Human studies, they say, remain necessary before anyone applies this to real populations.

I should declare an interest before going further. My company builds Digital Twins for media decisioning, so read the rest with that in mind.

The challenge is, advertising has adopted synthetic personas very fast, and there is no industry-agreed standard for defining what separates a good approach from a bad one.

Most large agency groups are already using them in their planning stacks. Every major research company has launched its own. But too few publish evidence showing that their simulated people predict real ones.

The appropriate level of scrutiny

A synthetic persona or Digital Twin that is only used to develop a campaign’s direction in the planning phase (general interests, channel mix etc) is unlikely to face too much scrutiny. The audience might shape a strategy deck, but once the campaign has run, it’s very unusual (at least currently) for anyone to go back and verify whether the synthetic audience correctly predicted human behaviour during planning.

Conversely, a synthetic persona that gets deployed into activation gets marked, because the campaign performance itself becomes the exam. That is the discipline we hold ourselves to: our campaigns end with a brand lift study on real people, so the twins are marked after the buy as well as before it.

We recently ran a study with more than 1,800 people. Six different ads, across a range of video content, and a question to each viewer: how well, or how badly, did the ad fit the content it ran in? Then we put the same ads and the same content in front of our Digital Twins, asked them the same question, and compared which content came out best and worst for each brand. The result: the twins’ choices were about 87% as good as a full human study’s, and took seconds.

That is a strong number, but it comes with limits. The twins were sharper on some brands than others. And if we ask them to choose only between two similar pieces of content, we’re pushing them past what they can do (today at least!). Ask them which ten thousand pieces of content out of fifty thousand deserve your budget and they hand you a solid list that real people would also back. Thankfully, that happens to be the job in media decisioning and activation. We are publishing the number with its limits attached because, in a market where almost nobody publishes anything, that is worth something.

Three questions worth asking

So if you are being sold a synthetic persona, Digital Twin or anything that’s not purely human, three questions are worth asking, and none of them requires a PhD in statistics:

Firstly, what did you compare your simulated people against, and were they real humans answering the same question?

Secondly, what is the accuracy number, and what is the best score anyone could have achieved against your yardstick? A score means nothing without knowing the best score possible.

Finally, where does it break down? Any vendor who cannot tell you where their simulation is weak has not looked hard enough, or would rather you did not ask.

Synthetic data has some amazing applications and is improving quickly. But it is about to be handed decisions worth a great deal of money, and right now the space has a very low bar for entry and many are flooding into it quickly. So my advice: be inquisitive, ask the right questions and make sure you know that what you’re buying is grounded in reality.