A female patient, 55 years of age, has taken a mammography and the test comes back positive for breast cancer. You understand that the mammography catches about 90% of breast cancers. Given this test result, what is the probability that the patient in fact has cancer?
A similar question was posed to 160 gynecologists and most (60%) estimated a probability of cancer of at least 80%. Is that close to your own estimate? The correct answer is much lower than 80%.
Remember that the result of a test is not all you know about the risk of someone contracting a particular condition. At the very least, you know the risk of breast cancer in the population among women of similar age in the population, which is about 2%. That is, without any further information you would estimate the probability of a given 55 year old patient contracting breast cancer at 2%. This probability is the patient’s baseline risk.
Of course, you may know even more information such as the patient’s family history and this additional information would, of course, shift the baseline risk. But for this example, let’s say that before the test, our best estimate would be 2%.
Now we come to the mammography. Extensive studies have shown that this test catches 90% of all cancer cases but also produces false positives (indicating the presence of cancer in cancer free breasts) in about 10% of cases. So we need a way to incorporate existing knowledge (in this case, breast cancer rates among women in their fifties), with the test results.
So the information we have is:
P(+ | Cancer) = 90%
P(+ | No Cancer) = 10%
The key to accurately answering the question we want to know is to split people into the correct subgroups.
Here, we’ve split the 1000 women into two groups: The breast cancer group has 20 (2%) and the cancer-free group has 980 (98%). We want the test to correctly identify the 20 women in the cancer group without incorrectly flagging anyone in the cancer-free group.
So the test correctly identified 18 of the 20 breast cancer patients. But these 18 are not the only ones given positive test results. Ninety eight of the cancer free patients were also given positive results. Thus, given a positive test result (of whom there are 116 patients), 18 will be correct. So 18/116 = 16%. This means that given a positive test result, the probability that a patient in fact has breast cancer is 16%. Nowhere near 90%!
It is important to note that the mammogram provides very little additional information above what the overall prevalence tells us. But it will cause unnecessary alarm in 10% of the women who take the test, resulting in anxiety and the discomfort of biopsy.
Whenever a test of any kind produces a binary outcome (i.e., positive/negative; yes/no) the result can fall into one of 4 buckets:
The test will put people into one of these 4 buckets. The likelihood of being placed in any one of these buckets depends on characteristics of the test.
For example, say a test always produces a positive result. As you can see in the table below, given a sample of 1000, all of them will be positive and 0 will be negative.
But of these 1000, some will be correctly positive and others will be false positives. The distribution of this 1000 among true and false positives will depend on the baseline information. If 90% have a condition, and the test always returns positive, then we would expect 900 true positives and 100 false positives.
Because this test identifies all positive cases, it has excellent sensitivity (90%), but terrible specificity (0%). It picks out every single positive case. But it does so at a cost. It falsely flags 100 of 1000 cases as positive. That means 100 people will suffer needless anxiety and worry and potentially undergo needless discomfort during follow-up tests.
Thus, all tests must walk a line between sensitivity and specificity; that is, between correctly identifying the true positives and true negatives. Mammograms could be made more sensitive, but that might be attended by increased false positives. They could also be made more lenient, but that would also come at the cost of missing some cancers.
Information Intelligence 