Base Rate Fallacy

Detection methods (e.g., those used in medical screening) are often described by their true positive rate and false positive rate. This is also the case for tools that detect AI writing. But what we usually want to know is something different: after the detector flags a piece of writing, how likely is it to actually be AI-generated?

That answer depends heavily on the base rate: the overall probability that a piece of writing is AI-generated in the first place. When the base rate is low, even a detector with a small false positive rate can produce many false positives among flagged texts.