Most classification problems have a comfortable property that nobody bothers to name, because it’s easy to miss while it’s true. Both kinds of error land on the same person.

If an account-takeover model locks out a legitimate session, your customer is annoyed and calls support. If it misses an actual account takeover, that same customer’s account gets emptied. The costs are wildly different in size, but they land on the same person. That’s what makes the threshold feel like a tuning exercise. You’re trading one harm against another, and both belong to somebody you have a relationship with.

Card fraud looks different. When a stolen card gets used on your platform, the person defrauded is the genuine cardholder, who may have no account with you and never agreed to anything. The harm lands on a stranger. But chargebacks, scheme rules and representment route that harm back to the company as cost, and if your ratios cross a network threshold the fines arrive on top. That plumbing is precisely why fraud losses become your problem, and why fraud teams get funded and staffed.

Products that generate images or video depicting real people have the same externality with none of that plumbing. If a likeness check fires when it shouldn’t, your user is inconvenienced. They can see what happened, complain, and ask for a refund. If it lets through something it should have blocked, the harm can land on somebody who has never used your product, never agreed to anything, and may never know it happened. Nobody charges you back for a fabricated likeness. No scheme rule connects that person’s harm to your balance sheet.

Legal routes do exist, and they matter: right of publicity, defamation, the newer non-consensual imagery statutes, an app store deciding it has seen enough. But compare how they behave. A chargeback is automatic, it is fast, and it costs the cardholder nothing to set off. A publicity claim needs the person to discover the harm, work out who to blame, find a lawyer, and live somewhere that recognises the claim at all. One mechanism finds you on its own. The other has to be found, by someone who may never know there was anything to look for.

So the only thing reliably standing between that person and the harm is a decision somebody makes on purpose.

That difference is not a tuning parameter #

A precision and recall curve invites you to think there’s one dial and one person on the other end of it. Slide it left, more of this error. Slide it right, more of that one. Pick the point where the total cost is lowest.

The arithmetic quietly assumes the costs are commensurable, which requires them to be borne by parties with roughly equal standing. When one error costs a paying customer 20 minutes and the other costs a stranger their face in something they never said, adding them up on one axis is a category mistake dressed as an optimisation.

So for products shaped like this one, my default is to block, and to absorb the false positives that come with it.

What should set the default is the shape of the harm, not a general taste for caution. Where the output is a recognisable person, easy to share, and effectively permanent once it exists, the third party is carrying something they can’t undo, and I’d rather absorb the complaints than hand them that. Where the output stays private, or is trivially reversible, or nobody real is depicted, a lower bar is perfectly defensible and I’ve argued for one.

What doesn’t vary is that somebody has to choose. A default nobody decided is still a default, and it ends up being set by whoever is fielding complaints that week.

There’s a cost to that default which took me too long to take seriously. Blocking on uncertainty isn’t neutral across users. No detector is equally accurate on everyone it gets pointed at, and those gaps are well documented, so a strict threshold doesn’t scatter its false positives evenly. It concentrates them on whoever the detector is worst at recognising. Saying “I’ll absorb the complaints” stays comfortable right up until you notice the same people keep making them. That’s an argument for reading false positives by group rather than in aggregate. It isn’t an argument for a looser default, and it is a real cost that belongs in the decision.

That’s not free in the ordinary sense either, and I don’t want to present it as though it were. Blocking on uncertainty means real users hit a wall for no reason they can see. Some of them don’t come back. The support load is real, and it arrives in the same week as everything else. I’ve never found a way to make that decision cost nothing. I’d still make it the same way, because the two errors aren’t the same kind of thing, and the one that’s cheap for me to absorb is the one my own users experience.

Thresholds should move on evidence #

A default is a starting position, not a permanent one. Thresholds are supposed to move as you learn how a detector behaves in the world rather than on a test set.

The question worth being strict about is what’s allowed to move them. Evidence about the detector should: it fires on a category of input it shouldn’t, the pattern is understood, the fix is scoped. That’s a good reason.

A quarter where signups are soft is not evidence about a detector. If the reason on the table is conversion rather than accuracy, say so out loud, because that’s a different decision and it deserves to be made as one. A threshold that only ever moves toward growth has stopped being a control. It’s a number somebody nudges when the number above it looks bad.

One detector is a single point of failure #

Any detector has a failure profile, meaning a kind of input it reliably gets wrong. A model tuned on clean, well-lit, straight-on examples fails differently from one built for messy real-world material, and neither fails the way a hand-written rule does. Layering detectors whose weaknesses don’t overlap buys coverage that tuning any one of them never will, and coverage is a different objective from accuracy.

Accuracy is a claim about a test set. Coverage is a claim about the world, and the world will find whatever your single detector is bad at, faster than your evaluation will.

This is also where the temptation lives to keep improving the detector you have, because that work is measurable and produces a chart. Adding a second one with a different failure profile is less satisfying, harder to justify in a sprint review, and considerably more useful.

A flag has to arrive somewhere #

Detection without a route to a person is not a control. It’s telemetry.

Who looks at it, and how fast? That question sounds administrative and it’s the whole thing. A detector with nowhere to send its output tells you, after the fact, that you could have known.

The part that gets skipped is what happens to the account while review is pending. What can it still do, and what can’t it? That belongs in the design as its own dimension rather than as an afterthought, because “under review” is not the same as active, and it’s not the same as blocked either. If your model of an account can’t represent it, somebody will end up representing it by hand, at speed, under pressure.

Build the controls before launch, or you won’t build them #

Controls of this kind get built before a launch or they don’t get built.

That isn’t cynicism about anybody’s priorities. It’s the shape of how roadmaps fill. Before you launch, the work is cheap, nothing depends on it, and the argument for doing it is abstract. Afterwards the product is working and the roadmap has real commitments on it, so safety work with no revenue attached goes to the back of the queue.

And nothing bad has happened yet. That’s the strongest argument in the world against spending a quarter on something.

The window where it’s cheap closes on launch day. That’s the entire reason to treat it as a gate rather than as a milestone.

The part with no feedback loop #

You never hear from the person you protected.

They don’t know. There’s no notification, no thank you, no metric, no chart in the review deck showing the harm that didn’t occur. The only feedback loop that exists runs through the users you inconvenienced, and it’s loud, and it’s specific, and it arrives every week.

So the pressure is asymmetric in exactly the wrong direction, and it will stay that way for as long as you run the product. Deciding the default early, in writing, before anyone is under pressure, isn’t administrative tidiness. It’s the only moment when the decision gets made by the argument rather than by whoever is most tired of the complaints.