Suppose a patient P is informed by a medical doctor that she has a certain disease and that she is expected to survive for two further years. What consequences for her life should P draw from this information? It seems that without further knowledge, she can draw no precise consequences at all. P has to know how the doctor came to this decision. This is so because of the reference-class problem.
In most situations P is compared with people that are similar with respect to the disease in question- a reference class R. For instance, R may contain individuals with similar symptoms as P. The expected survival time of P is then equated with the average survival time of individuals in the reference class R.
However, R is not uniquely determined. There may be many factors that are relevant to P’s survival rate. As already remarked by John Venn “[…] that every individual thing or event has an indefinite number of properties or attributes observable in it, and might therefore be considered as belonging to an indefinite number of different classes of things.” (Venn, J.: The logic of chance, 3rd edition, 1988, p.213). The worst situation is the case of conflicting reference classes.
“Let us assume, for example, that nine out of ten Englishmen are injured by residence in Madeira, but that nine out of ten consumptive persons are benefited by such a residence. These statistics, though fanciful, are conceivable and perfectly compatible. John Smith is consumptive Englishmen; are we to recommend a visit to Madeira in his case or not? In other words, what inferences are we to draw about the probability of his death? Both of the statistical tables apply to his case, but they would lead us directly contradictory conclusions. […] Without further data, therefore we can come to no decision. (Venn, p.222-223)”
However, when a medical doctor predicts, for instance, P’s survival time, she does this often on basis of a reference class; thereby implicitly assuming a solution for the reference class problem. One may suppose that in most situations there is a single adequate reference class. That this is not the case is shown by the court case of Shonubi. The task was to estimate the amount of drugs smuggled by the drug mule Shonubi on the JFK airport during seven trips. The sentencing was changed several times, because there was no consensus on what reference class to employ. (For more details see http://blogs.kent.ac.uk/ebmplus/2014/10/24/fun-with-reference-classes/ or Colyvan, M., Regan, H. M. and Ferson, S. (2001), Is it a Crime to Belong to a Reference Class. Journal of Political Philosophy, 9: 168–181) .
What should P know about the decision of the doctor? A first shot is: Were there other reference classes that may be relevant? If so:
- The average values in these classes
- Relevance: (Within) variance of reference classes
- Unanimity: Variance between reference classes
- Precision: Standard error of the reference classes
- The procedure employed to obtain the predicted value: Is one reference class chosen? Is a weighted average of reference class values used?
The variance of reference class describes how similar members of the class are with respect to the outcome (in our case the survival time). The more similar the members, the more relevant for the patient is the reference class. The between variance of reference classes states how different reference class values are. If the between variance is high, then there was evidence that pointed in a different direction. Ignoring this fact, would violate the criterion of total evidence. If P were to know that there is also a reference class she belongs to where people survive on average 5 years, that may lead to different decisions regarding P’s life. The standard error measures the precision of the estimate of the reference class value. Especially, in the case of smaller sample sizes, this tends to be higher and therefore the estimate for the average value in the reference class is more imprecise.
Concluding, in order to make decisions, a patient has to know more about the procedure the doctor employed to arrive at her prediction. The five points listed in this small note may be considered as a starting point to the question: What should be reported in diagnosis and prediction? To come up with a more detailed answer requires more research on the reference class problem. Research that I am very eager to do.