The 99.8% Problem in Black Mirror’s Hang the DJ | Physics vs Film (and TV)

OK I suppose I better check the dating
apps, let’s do some swiping. No, no, yes, yes, yes. I wonder how they work out these match
percentages. It’s probably based on all those questions. But what if they ran a
simulation to see how compatible we’d actually be? No that’s a stupid idea….
isn’t it? In Black Mirror’s fourth season episode
Hang the DJ, the premise is everyone’s dating lives are controlled by an app
called Coach which sets you up with people and even tells you how long
you’re going to be with them… if you like it or not. From all the data Coach
collects, it sets you up with your perfect match eventually. “They’ll always
find your perfect match” “In 99.8% of cases” But as the two characters defy Coach and try to run
away together, what we find out is that they were
within a simulation the whole time. One of an ensemble of 1000 total different
simulations of that same couple, 998 of which ended up together which forms the
basis of a match percentage for a dating app in the real world – with that 99.8%
being a callback to earlier in the episode. And you might’ve thought ah yeah
that makes sense nice right? No it absolutely does not make any sense at
all… as I was recently explaining to somebody
on a date because that’s what you talk about when you’re on dates right? Right?
The whole 99.8% thing cropping up everywhere is just a mistake by the
writers and I’m going to tell you why. The most obvious way to see this is to
understand that it’s a circular argument. If the 99.8% is supposed to be the
percentage of all of the simulations that were successful run by the app in
the real world, then how on earth does the one simulation that we’re following
know that outcome of all 1000 simulations before it’s even finished
running? What’s the point of running a simulation if you already know the
answer? It doesn’t make sense! Then again you might say oh but we were
in simulation number 998. We actually saw that at the end of the episode, so maybe
they just used the number of the simulation that was actually running?
Which is fine if you’re talking about simulation number 998 but what if you’re
in simulation number 1. Are you really going to trust an app called coach who
claims its success rate is 0.1%? Now there is one way that this
could work. Say for instance that each simulation bases the confidence level of
Coach on the results of previous simulations. So for instance in
simulation number 998 it looks at the previous 997 simulations and then takes
the success rate out of all of those. That would mean 995 of those have been successful to end up with a 99.8%
probability, to that level of precision anyway. But if you have any
experience with running simulations you’ll know really you don’t want to be
doing that – running them sequentially one, two, three, four, five. That’s incredibly
inefficient! Simulationists will always try and run
independent calculations in parallel, essentially using different processors
so that they can be done exactly at the same time rather than taking up a huge
amount of time on just one processor. And there is evidence in the episode that
that’s what’s going on. When we see all of the simulation couples right at the
end it seems like they’ve all finished their simulation runs near
simultaneously. Another thing to think about is what about everybody else
that’s using that app in the real world? Where are their simulations? Are they all
running a separate 1000 simulations and every other person in each simulation
apart from the couple are just procedurally generated non player
characters? I would argue no, because again that would be hugely inefficient.
It would be much better resource wise for everyone to be running around in
same 1,000 sandbox simulations to work out those match percentages. But of
course each simulation needs to be self-consistent,
so if one couple are told that Coach has a 99.8% success rate then every couple has to be told that. But of course not every couple
is going to actually end up with a 99.8% match rate in the real-world app, so another reason why those two numbers shouldn’t
be the same. And finally you might just say, oh well it was just a coincidence.
Well that’s a one in 1,000 coincidence which I am not buying! Look I really did
enjoy this episode of Black Mirror, I enjoyed the whole series in fact, but
that 99.8% things slightly bugged me, because it makes a mistake
that people often fall into when you’re talking about statistics. The original
ninety-nine point eight percent said by Coach is something of a level of
confidence about an outcome of an experiment, whereas what we see at the
end is an experimental probability – a ratio of number of successes divided by
number of trials. They’re two very different sorts of percentages and you
shouldn’t mix them up. Let me give you an example, for instance say we were running
a thousand simulations and we did get 998 successes we could ask the question
well what truly was the match rate? Whilst on average it would be ninety
million point eight percent it could be ever so slightly different and with
ninety nine point eight percent confidence we could say that their match
percentage was actually between 98.8823 and 99.9955%. A 99.8% confidence interval on a 99.8% probability outcome.
If they said it like that I might have been a bit happier, because as a
physicist we often have to rely on statistics to draw conclusions and you
really need to get this stuff right! Thanks so much for checking out this
video I really do appreciate it I would appreciate it even more if you could
maybe write down there and what you thought of the episode Hang the DJ and
if you think I’m being far too pedantic. Otherwise please do give it a like and
subscribe to my channel to get the latest stuff and otherwise there’s a
couple of more videos over there.

Leave a Reply

Your email address will not be published. Required fields are marked *